Putting Accuracy in Context

Posted by: Stan Humphries    Tags:      Posted date:  September 21, 2012  



With our quarterly release several weeks ago of our latest benchmarks on Zestimate accuracy, we thought it would be helpful to put some context around the numbers since it can often be difficult to understand error rates in isolation. For example, is our current 8.8% median absolute percent error good or bad? One potentially useful context is to consider how the Zestimate accuracy compares to the accuracy of the initial list price placed on a home. One expects the list price to be fairly accurate since it’s typically created by a real estate professional familiar with both the home and the local market (although sometimes the seller can influence that price to be higher than an agent might prefer). Moreover, list prices are expected to have higher accuracy because they are not merely estimates of value, but rather the actual “ask price” itself in economic terms, meaning that the home can actually be transacted at that price if a matching bid can be found.

In order to understand how Zestimates and list prices compare in terms of accuracy, we selected transactions that occurred within Zillow’s data footprint between March 1, 2012 and June 1, 2012 and tied these sales back to both the initial list price and the Zestimate at the time of initial listing. There were 296,473 home sales included in the analysis. The initial list price was above the ultimate sale price for 74% of the sales whereas the Zestimate was above the sale price for 51% of the transactions (see Table 1). The error was calculated as the difference between the initial list price (or Zestimate) and the ultimate sale price. The percent error was computed as this difference divided by the sale price. Absolute percent error was also computed as the absolute value of the percent error.

The full distribution of percent error for both the initial list price and the Zestimate is shown in Figure 1. As we’ve noted elsewhere and can be seen clearly in Figure 1, the Zestimate error is symmetrically distributed meaning that it is just as likely that the Zestimate will be higher than the sale price versus lower.

Why are there so few initial list prices that are below the sale price whereas the Zestimate distribution is evenly distributed on either side of zero? Part of the reason, as alluded to above, is that the initial list price is not just an estimate of a home’s value; it’s the price at which you can expect the seller to accept a contract (in the absence of competitive bidding). For this reason, we’re less likely to detect a list price that is too low than we are to detect a list price that is too high.  A home with a list price that is too low will quite possibly sell at the lower price (and we won’t actually know that it may have sold for a higher price), whereas a home with a list price that is too high will not sell until the price declines to a level at which somebody is willing to purchase it. Indeed, we’ll only observe that a list price is too low if the price gets bid up, but this does not always happen, particularly in many markets that are still distinctly more of a buyer’s than seller’s market. Consider the fact that more than 8% of listings sell at precisely the same dollar amount as the initial list price (versus only 0.15% of listings that sell at exactly the same dollar amount as the Zestimate). The high percentage of listings sold at precisely the initial list price could indicate that some of these were underpriced.

Looking at all sales, the accuracies for both the list price and the Zestimate are shown in Table 2. This table shows that the initial list price is within 5% of the final sale price for 48% of the observations whereas the Zestimate achieved this accuracy for 31% of the observations. The initial list price is within 20% of the final sale price for 90% of the observations whereas the Zestimate achieved this accuracy for 81% of the observations. The overall median absolute percent error for the initial list price is 5.6% versus 9.2% for the Zestimate.

Table 2 also presents the accuracy for the final list price (the list price immediately prior to the sale, possibly after several price cuts) and final Zestimate (the Zestimate on the home just prior to the sale). Both the list price and Zestimate are much more accurate when measured closer to the time of sale, with a median absolute percent error of 4.5% and 6.3% for the final list pricing and the final Zestimate, respectively.

Given the asymmetry of errors for initial list prices and the fact that a chief contributor to this pattern could be that the listing is underpriced, it’s also interesting to consider just those sales for which the initial list price is greater than the ultimate sale price (74% of the sales or 220,719 observations). With this approach, we ensure that we don’t include listings that may have been underpriced and for which we can’t observe the pricing error because the home is, in fact, purchased at the lower price (thereby preventing us from knowing that the home could have sold for more).

Looking at this set of sales, the accuracy for both the list price and the Zestimate is shown in Table 3. This table shows that the initial list price is within 5% of the final sale price for 38% of the observations whereas the Zestimate achieved this accuracy for 32% of the observations. The initial list price is within 20% of the final sale price for 88% of the observations whereas the Zestimate achieved this accuracy for 81% of the observations. The overall median absolute percent error for the initial list price is 6.9% versus 9.1% for the initial Zestimate. As was the case when looking at the full set of data, accuracy for the final list price and final Zestimate was considerably better, with median errors of 5.3% and 6.1% respectively. Overall, the difference between the accuracy of the list price and the Zestimate is smaller in this set of data than for the full set.

These data clearly demonstrate two things. First, as we expected, real estate professionals who know the local market and intimate details of the home can price a home better than an automated valuation model. But the Zestimate does provide a good starting point for a conversation about home values, a starting point that compares fairly well to list price, particularly when considering only listings that can be definitively ruled out as being underpriced. But, as we’ve always noted, that pricing conversation that may be started with the review of the Zestimate should ultimately be augmented with the input of opinions from local real estate professionals (agents, brokers and appraisers).

 


About the author
Stan Humphries
Stan is Zillow's Chief Economist. To learn more about Stan, click here