How can I refinance my home, where the loan was modified in 2007?

We have good credit and have not had any late or missed payments.  At the time we originated the original mortgage we planned to do a modification the next day because the house appraised for significantly more than we paid for it but the original loan terms had to be based on the purchase price of the house and the lender told us we could do a modification right after we closed the loan and use the appraised value of the house which reduced our monthly PMI.  This modification was planned before we closed the original loan in no way was the modification done because we could not make our payments.  We have never been late on a payment and have 700-750 credit scores and the value of our home has not significantly declined. We would like to be able to refinance our mortgage to a lower rate but have been told we cannot refinance beacuse there is a modification and no lenders will touch a loan that has been modified.