when is it too late to switch lenders?

I'm, not a big fan of jumping to another lender for price unless you are really being taken advantage of.  If you are financing a conforming 30yr fixed rate with 720 or better fico on a conventional or FHA loan, the interest of rate of 5.500% with 0.500 fee seems fair.    I don't know how much better the other offers are, but some of the quotes on Zillow are from lenders that sell to Provident and that investor is notoriously difficult to obtain an approval with as they cherry pick applicants and properties they will finance.  If this is a  refinance and if you are pulling cash out, or have a loan amount under $100,000 or FICO is under 720, there are pricing adjustments that may not be factored into the rate quotes you received on Zillow.  Pricing is a moving target and since rates are a little worse already this morning (compared to Friday), you may have to pay a little more.    If you do decide to lock with this lender ask about a floatdown to a lower rate if the market substantially improves before your closing docs.  Almost all lenders offer this these days.