The private listings premium that wasn’t

Another week, another misleading discussion about the statistics of private listings networks, a scheme where brokers try to sell your home to their own clients before the rest of the market has a chance to see it and bid on it. 

While I previously called out a claim about the potential for inventory growth coming from PLNs as being derived from self-serving assumptions (rather than from observed empirical analysis), this week I’d like to broach a more nuanced subject. Specifically, how empirical econometric work about the effect of private listings on sales price took on a life of its own when self-interested parties overstated what the science is actually showing. 

Specifically, I’m going to discuss a study from the University of Georgia that was widely circulated by the proponents of PLNs. The University of Georgia is home to top scholars and a world class economics department, so I want to be very clear that I’m distinguishing econometrics conducted by a researcher, from the much less honest discussion that has stemmed from the research. The research paper itself is part of the long-term process of scientific discovery, the promotion of it has been misleading.

First, as noted recently by Darryl Davis in Inman, most readers who forwarded the claims did not apparently read past the synopsis. The higher price effect the author identified vanished after the Clear Cooperation Policy was implemented. So the touted paper specifically states that the observed price effect was a function of a pre Clear Cooperation Policy world and that world is gone now, so there is no longer an observed price premium for private listings. 

But beyond reinforcing that point, I also want to raise a second broader criticism about science and discourse. 

An industry commentator noted that “there are Poisson arrival rates and coarsened exact matching procedures and simultaneous quantile regressions [...] and I am not going to pretend I know what those mean”. That’s fair, and I expect it applies to most people in the industry. However, as a trained econometrician and active university instructor in econometrics, I thought it would be worth diving a little into what those terms mean, and how those with a pecuniary interest in private listings are overstating what the econometrics involved are actually demonstrating. 

Forgive the technical jargon in advance, but it's important to be honest about good science and what it does and doesn’t say. So engaging with some of the underlying statistical concepts, even if they’re inaccessible to the layperson, is important to get at the truth. The details matter so it’s important we disentangle what the words themselves mean. Statistical tools are useful and can solve specific problems, but they aren’t magic. 

First, the study uses what we call in econometrics ‘fixed effects’, meaning that the study controls for the month and year of the listing comparisons it is analyzing. Over the roughly 10 year observation window that contains the bulk of the private listings, that means there are approximately 120 month fixed effects.  

In addition to the roughly 120 month/year pairs, the paper places its observed transactions into like-for-like comparisons by creating 3601 microcells across the DFW metro. To make a valid like-for-like comparison of private vs public listings, the two types should be compared to one another in the same microcell during the same month, which would be roughly 120 months in 3601 locations, or, around 432,000 possible cell-month combinations where matches could occur. With only around 5,000 private listings to distribute across that space, the comparisons are necessarily concentrated in a small subset of those combinations.

This sparsity creates a classic statistical problem. In typical off-the-shelf statistics, if you have more controls than you have records in your sample, most of your controls aren’t actually doing anything, what we call in econometrics “underdetermined.” 

Econometrics has developed a range of statistical tools that researchers can employ to address this kind of problem. In this study, coarsened exact matching is used. What that means in simple terms is a researcher can create buckets of attributes on which to match properties and join them together. However, this means making a crucial tradeoff of what we call “internal vs. external validity.” 

Put another way, this matching and pruning exercise results in the statistical model becoming powered to do what is being asked of it, but the outcome of the model is no longer making a statement about the entire DFW housing market. Rather, it’s making a statement about the subset of neighborhoods where this matching is available, which is quite narrow. Specifically, the geographic cells and market months where private listings have local on-MLS comparison properties. 

The paper’s Table A1 reports what’s called an L1 imbalance score of 0.637 after the matching I described above (statisticians use the L1 score to measure how differently two groups are distributed across multiple characteristics, with 0 meaning the groups are identical and 1 meaning they have nothing in common). And while the author does address why they’re unconcerned about the 0.637 score, the following Table (A2) contains some revealing numbers about the inherent problems with the analysis, specifically attributes that still have unbalanced scores (meaning the on and off mls listings aren’t very similar). Specifically what they capture is that observable proxies for the decision to not make a listing public, still don’t match. What this means in plain English, is that the decision to use a private listing is still based on unique characteristics of that home, so the 1.7% lift is biased by those characteristics. 

This was probably more econometrics than you wanted to read, so I’ll summarize by saying what the study actually found was a 1.7% lift in sales prices within narrowly defined cell-month strata in DFW during the 2010s, before clear cooperation was implemented. And on top of that, the properties that do transact off the MLS are not random, but rather are doing so strategically. 

So even when the effect was observed briefly, it was still confounded by selection bias of which homes were chosen to be private listings by the agent involved. Again, the author of the paper is upfront about this, noting in the case of the luxury tier “this implies a strong selection mechanism: the minority of high-end properties that do transact off-market are likely doing so strategically” and that for the other tiers the model documents systematic, non-random sorting into the private channel on multiple dimensions, which is precisely the kind of selection that no observational matching procedure can fully neutralize.

I want to reiterate that there’s nothing wrong with what the researchers have done, this is the scientific process at work: use the data you have, use the statistical methods available, and report what you find. Then let other researchers comment on and critique your analysis and results. The study author is upfront about their methods, and directly noted that the effect went away after CCP. That’s good science. But the conversation within the industry that stemmed from the study has not followed that same rigor, and has not been honest about the effect or upfront about the limitations. 

So what is the real effect of private listings on sales price?

We have our own analysis showing that sellers have lost out on billions, but it will take the independent research community some time to reach an independent consensus on that. In any field of science, theory is what we rely on before we have empirical consensus, and on that note economic theory offers some pretty strong clues. 

As the study authors note: “The fundamental premise of a competitive auction is that maximizing the pool of bidders maximizes the clearing price…  any restriction on exposure should result in a liquidity discount; sellers who voluntarily hide their assets from the open market limit competition and, by extension, should realize lower sale prices.” So anyone using the study to claim PLNs are price maximizing has either misread the paper or is selectively quoting it and lacking integrity in their claims.