With the launch of Zillow Prize, we thought it would be helpful to put some context around our published Zestimate accuracy numbers since it can often be difficult to understand error rates in isolation. For example, is our current 5.4% median absolute percent error good or bad? One potentially useful context is to consider how the accuracy of the Zestimate home valuation compares to the accuracy of the initial list price of a home. One expects the list price to be fairly accurate since it’s typically created by a real estate professional familiar with both the home and the local market (although sometimes the seller can influence that price to be higher than an agent might prefer). Moreover, list prices are expected to have higher accuracy because they are not merely estimates of value, but rather the actual “ask price” itself in economic terms, meaning that a seller would presumably immediately accept any offer to buy the home made at that price.
In order to understand how Zestimates and list prices compare in terms of accuracy, we observed all full-value, arms-length sale transactions that occurred within Zillow’s national data footprint between January 1, 2016 and December 31, 2016. For each sale transaction, we tied the sale back to both the initial list price and the Zestimate at the time of initial listing. There were 1,951,999 home sales included in the analysis. The initial list price was above the ultimate sale price for 68% of the sales whereas the Zestimate was above the sale price for 42% of the transactions (see Table 1). The error was calculated as the difference between the initial list price (or Zestimate) and the ultimate sale price. The percent error was computed as this difference divided by the sale price. Absolute percent error was also computed as the absolute value of the percent error.
The full distribution of percent error for both the initial list price and the Zestimate is shown in Figure 1. As we’ve noted elsewhere and can be seen clearly in Figure 1, the error of the Zestimate home valuation has a nice symmetrical distribution around zero whereas the error of the initial list price skews noticeably above zero (indicating initial prices at or above the final sale price).
Why are so few initial list prices below the sale price whereas the Zestimate distribution is evenly distributed on either side of zero? Part of the reason, as alluded to above, is that the initial list price is not just an estimate of a home’s value, but rather the price at which you can expect the seller to accept a contract. Underpricing errors will only be corrected for homes that generate an auction-like competitive bidding situation where the sale price is bid up by multiple prospective buyers. While we see such multiple-offer situations in many hot seller’s markets today, it doesn’t happen everywhere and, when it doesn’t happen, an underpriced home may often sell for the lower list price without the seller knowing that they might have gotten more for the home. In contrast, a home with a list price that is too high will not sell until the price declines to a level at which somebody is willing to purchase it.
Looking at all sales, the accuracies for both the list price and the Zestimate are shown in Table 2. This table shows that the initial list price is within 5% of the final sale price for 55% of the observations whereas the Zestimate achieved this accuracy for 35% of the observations. The initial list price is within 20% of the final sale price for 92% of the observations whereas the Zestimate achieved this accuracy for 84% of the observations. The overall median absolute percent error for the initial list price is 4.2% versus 7.7% for the Zestimate.
Table 2 also presents the accuracy of the final list price (the list price immediately prior to the sale, possibly after several price cuts) and final Zestimate (the Zestimate on the home just prior to the sale). Both the list price and Zestimate are much more accurate when measured closer to the time of sale, with a median absolute percent error of 2.7% and 3.6% for the final list pricing and the final Zestimate, respectively.
Given the asymmetry of errors for initial list prices and the fact that a chief contributor to this pattern could be that the listing is underpriced, it’s also interesting to consider just those sales for which the initial list price is greater than the ultimate sale price (68% of the sales or 1,327,131 observations). With this approach, we ensure that we don’t include listings that may have been underpriced and for which we can’t observe the pricing error because the home is, in fact, purchased at the lower price (thereby preventing us from knowing that the home could have sold for more).
Looking at this set of sales, the accuracy for both the list price and the Zestimate is shown in Table 3. This table shows that the initial list price is within 5% of the final sale price for 45% of the observations whereas the Zestimate achieved this accuracy for 35% of the observations. The initial list price is within 20% of the final sale price for 89% of the observations whereas the Zestimate achieved this accuracy for 83% of the observations. The overall median absolute percent error for the initial list price is 5.8% versus 7.8% for the initial Zestimate.
As was the case when looking at the full set of data, accuracy for the final list price and final Zestimate is considerably better, with median errors of 3.3% and 3.7% respectively. Overall, the difference between the accuracy of the list price and the Zestimate is smaller in this subset of data for which we can be certain that a pricing error will be detected (because the price is too high).
These data clearly demonstrate two things. First, as we expected, real estate professionals who know the local market and intimate details of the home can price a home better than an automated valuation model. But the Zestimate does provide a very good starting point for a conversation about home values, a starting point that compares quite well to list price, particularly when considering only listings for which we can be certain that a pricing error will be detected. But, as we’ve always noted, the pricing conversation that may be started with the review of the Zestimate should ultimately be augmented with the input of opinions from local real estate professionals (agents, brokers and appraisers).
