StreetEasy® is committed to making every experience of buying, renting and selling real estate in NYC more seamless. Artificial intelligence (AI) has long been a cornerstone of Zillow’s data products, powering features like our Home Value Estimate, Listing Recommendations, and Buyer Propensity Scoring. In 2024, we began incorporating large language models (LLMs) into our research and development processes, with the goal of giving people a more seamless experience of renting, buying, or selling their home. This involved prototyping innovative applications and developing new data product features that create a more personalized and intuitive user experience.

This blog post details StreetEasy Applied Science Engineering’s approach to AI, and showcases our recent deployment of two LLM-based features. These features are designed to enhance the consumer experience on our website. The first uses AI to provide quick, accurate answers to frequently asked property-related questions, improving efficiency and user satisfaction. The second feature creates a personalized introduction experience by connecting shoppers with agents, helping to foster trust and confidence during the often complex real estate purchasing process. This personalized introduction builds rapport and sets a positive tone for the crucial agent-client relationship. We’ll delve into the technical details and design considerations of both features, sharing insights and lessons learned along the way.

Instant Answers: Revolutionizing Real Estate Support with LLMs

Our first LLM-powered feature provides quick and accurate answers to frequently asked questions (FAQs) that are specifically related to properties. Based on our data insights and research, shoppers spent, on average, less than 61 seconds on our core property and listing pages. Their attention to detail is often challenged by the wealth of information our listing pages can present. This was apparent when we dove into the details of the questions that shoppers submitted to our agents. We took all the property-for-sale-related questions that were submitted between December 2021 and April 2023 and visualized the data as word clouds.

We realized that many commonly asked questions could be addressed with AI-generated FAQs. For various reasons, shoppers prefer to ask a question, rather than searching or reviewing all the details of a property. As such, the instant gratification experience that we created on our site is a desirable feature for many of our shoppers.

Technical Design of Instant Answers

The traditional, web-based knowledge base/FAQ depends on search mechanisms that are sometimes slow and inefficient. Users must often navigate through complex FAQ search engines and irrelevant search results, which can lead to frustration and a lengthier search for the answers they need. Our goal was to create a seamless, intuitive experience that provided instant, contextually-relevant answers. In addition, we wanted to design a system that was cost effective and scalable. Specifically, we wanted to build an AI system that would cost less than a few hundred dollars, while scaling to support StreetEasy/Zillow traffic.

By balancing between the requirements of our caching strategy and the scope of our answers we were able to build an AI-powered FAQ experience that would generate the answers requested by our shoppers. Working with our design team, we iterated through multiple options and narrowed it down to the following experience.

We decided to pre-generate answers to the most frequently asked questions using available data instead of responding to inquiries in real-time. This approach not only minimized computation cost and latency, but also ensured the reliability of the answers.

Generating the answers involved several key steps:

• Topic Modeling: We used BERTopic(1) to identify common themes from an anonymized collection of inquiries, which were submitted by past home buyers. Using this technique, we first created dense representations of these inquiries using BERT(2) (Bidirectional Encoder Representations from Transformers) embeddings. By applying clustering algorithms to these representations, BERTopic helped identify latent topics within the documents, while preserving the most important words for each topic. Given that our dataset consisted of short messages, each containing one distinct question, this approach allowed us to pinpoint the most frequently asked questions and their variations.

Example frequently asked topics and top terms for each topic

• Context Data Preparation: We aggregated and pre-processed property data from active listings, past listings and building pages to create a comprehensive dataset. This dataset served as the foundation for generating accurate answers.
• Prompt Engineering: Our primary goals were to identify which FAQs can be answered using available data, and to ensure that we only respond to answerable questions. To achieve these objectives, we experimented with various prompting techniques to generate answers to the FAQs. We then evaluated these answers using LLM-based metrics such as: Faithfulness, which measures the factual consistency of answers, given the provided information; and Response Relevancy, which measures how relevant the answer is to the given prompt(3). Additionally, we leveraged non-LLM-based-metrics to ensure compliance with design requirements. One of the main challenges we faced was preventing LLM from hallucinating and returning answers to unanswerable questions. To address this issue, we leveraged the chain-of-thought prompting technique(4). We instructed the LLM to first draft answers to all questions, and then validate these answers against the context data before returning the final response. This step-by-step approach helped us generate more accurate and reliable answers.

By following these steps, we confidently provided pre-generated answers to common questions, enhancing the user experience for home shoppers while maintaining efficiency and accuracy.

Easy as PIE (Personalized Introductory Introductions): AI-Powered Connections

Our second LLM-powered feature focused on enhancing the connection between home buyers and real estate agents.  Easy as PIE is a Zillow Hackweek 2024 winner of best of AI award, in which StreetEasy led an innovative project, based on the understanding that the real estate shopping process can be complex and intimidating, and that establishing trust between buyer and agent is crucial for a positive experience. Our goal was to create a personalized introduction that would preemptively address key concerns and establish a strong foundation for a productive relationship.

Trust building through transparency

One of the biggest challenges for home shoppers is finding a real estate agent that best fits their personal needs.  Based on our data and research, finding the right agent match and establishing a strong shopper/agent relationship were fundamental to a successful real estate transaction.  Our design research team conducted a thorough analysis of our existing connections experience based on the current connect experience design. The findings revealed insights into participant experience perceptions, and an opportunity to surface additional relevant information to home shoppers about the agent with whom they were connecting.

Technical Design of Easy as PIE

The new experience in Easy as PIE features a focused design that leverages data to create unique personal experiences that highlight an agent’s fit with the shopper. With this experience, we focused on providing the relevant information that shoppers care about most when they are looking for an agent.  Based on our buyer survey, the overwhelming majority of successful buyers (9 in 10) hire an agent to help with some part of their buying journey; 46% of shoppers care about agent expertise in the building or neighborhood area; 37% of shoppers like the agent personally and finally 32% shoppers care and are impressed by an agent’s past deal making experience.

Based on our findings, we highlighted key data points about the agent to demonstrate their suitability for your personal needs. These data points included:

• Ability to work within your price range
• Knowledge of the area or neighborhood
• Experience with the specific listing or building
• Expertise with specific property types
• Total years of real estate experience

We computed these attributes based on the agent’s deal history (based on their StreetEasy profile), the user’s home preferences (inferred from their search activity), and data about the specific property they are browsing.

Additionally, we display an AI-generated bio summary to highlight the agent’s professional experience and qualifications as a successful buyer’s agent. Different from the Instant Answers feature, the main goal of the summarization task was to create short and concise bio summaries—easily fitting into the carousel card shown in the design,while remaining informative and unique to each agent based on the longer biographical profiles written by themselves.

To achieve this goal, we worked with our UX Content Strategy partners in identifying specific requirements and drafting instructions for creating effective messages. When testing different prompting techniques, we used LLM-based evaluation to score how well the summaries comply with the requirement, and human-in-the-loop review to rate the level of informativeness and creativity. Based on the evaluation, we adopted a chain of prompts, iterating through drafting and revision tasks before returning the finalized summary.

Looking ahead, we see potential to further personalize these bios. By incorporating users’ home preferences, we can highlight the agent’s unique skills that are most relevant to a user’s interests and needs. Additionally, featuring the agent’s personal interests, and tailoring the tone to match their communication style, can make messages more engaging and help potential clients connect on a personal level.

Lessons Learned

The development and deployment of these two LLM-powered features provided invaluable learnings and insights into the challenges and exciting opportunities of integrating LLMs into StreetEasy’s applications.

• Data quality and trust are paramount: The performance of LLM applications depend on the accuracy and relevance of information provided. Our strategy was to understand the strengths and gaps of our data, and then focus on designing solutions that leveraged our strengths: comprehensive listing and property data. We also iteratively refined our prompting techniques and conducted rigorous testing to ensure our LLM provides reliable and contextually relevant responses.
• Ethical considerations are crucial: LLMs that are trained on vast datasets can inadvertently perpetuate and amplify existing societal biases, leading to unfair or discriminatory outcomes. Therefore, a commitment to responsible AI development necessitates careful attention to ethical implications, and the proactive implementation of bias mitigation techniques. To this end, StreetEasy collaborates closely with the Zillow Ethical AI team to ensure our practices are aligned and peer reviewed by our legal and larger AI teams. We also leverage the Fair Housing guardrails and create clear instructions and examples in our prompts, in order to set content boundaries and guide the LLM towards generating compliant responses.
• Iterative Product, Engineering, Marketing and Design (PEMD) team development is key to success: Building a successful AI application is an iterative, cross-functional process. Although LLM and AI implementations are highly technical, having a strong design and marketing partnership early on also enables a clear focus, and builds the development flywheel for continuous monitoring, evaluation, and refinement. Working together, the PEMD team can use its collective knowledge to build a holistic user’s journey and overall experience. The PEMD team must take full advantage of the respective areas of expertise in order to ensure that the AI application not only meets technical requirements, but also aligns with user needs and general business goals.
• Human oversight is essential: LLMs can automate many tasks, but human oversight is still vital, especially in sensitive areas like generating personalized communications. Automated evaluation and human feedback are both necessary to maintain quality and prevent errors. After deployment we conduct A/B tests and monitor key metrics and LLM responses, in order to ensure continuous improvement of the application. By integrating the generative capabilities of LLMs with human expertise, we can create a robust system that optimizes efficiency, while upholding the highest standards of quality and reliability.

Embracing the AI Revolution: Navigating New Opportunities and Challenges at StreetEasy

The rise of AI presents StreetEasy with unprecedented opportunities to enhance and personalize the shopper experience. This technological shift fundamentally alters our operational landscape, and the very rules of engagement within software development.

From an engineering perspective, the potential applications of AI are both exhilarating and daunting. Our excitement stems from the vast array of opportunities now within reach, such as scaling personalized experiences to an unprecedented level through an agentive framework, or empowering AI to autonomously handle numerous tasks such as searching a right home or right agent.

The challenges are also paramount, for which reason this transformation brings a degree of apprehension and uncertainty that beg questions such as: What is the right level of resource investment to build a new AI system versus traditional software systems? Can existing engineering systems become obsolete, some of our long-held software engineering skills could quickly diminish and be replaced by AI in relevance? How can we continue to lead in innovation?

We believe this necessitates a strategic leadership focus on two key areas:

Cultivating AI Expertise: Investing in AI talent acquisition and providing comprehensive training for our existing product, design and engineering teams is paramount. We must equip our PEMD team and software engineers with the skills and knowledge to effectively design, build, and deploy AI-powered solutions. This involves not only mastering new technical skills but also embracing a fundamental shift in mindset – one that fully leverages the power of AI as a collaborative partner. Whenever a new system or feature needs to be built at StreetEasy, we should ask ourselves how we can build it with AI?

Data-Driven Development: Enabling our organization to build data-driven solutions is crucial for unlocking the full potential of AI. To this end, we must foster a culture of data literacy, provide access to robust datasets, and equip our teams with the analytical tools necessary to extract valuable insights. At StreetEasy we continue to invest in building and collecting the best real estate data for our customers. By harnessing the power of this data, we can create more intelligent, personalized, and effective AI-powered features.

Our commitment to embracing AI is not simply about integrating new technologies; it’s about transforming our development processes and cultivating a culture of continuous learning. We’re dedicated to balancing our business needs and equipping our engineers with the skills they need to navigate this evolving landscape, and ensure that StreetEasy remains at the forefront of innovation in the real estate industry. This journey demands both bold vision and strategic adaptation, a challenge we enthusiastically embrace.

Acknowledgments

A special thanks to Adrian Tame, Shufei Ma and Stephan Guthrie as contributors to this blog and Nathan Johnson, Grant Fraley and Will Wallace for product and engineering leadership. Finally, thank you to the extended StreetEasy PED (Product, Design and Connect Engineering team) for their invaluable contributions, unwavering support and continuous efforts in building and enhancing these AI projects. Your dedication and expertise have been instrumental in the success of launching these projects.

StreetEasy is an assumed name of Zillow, Inc. which has a real estate brokerage license in all 50 states and D.C. See real estate licenses. All marks herein are owned by MFTB Holdco, Inc., a Zillow affiliate. © 2025 MFTB Holdco, Inc., a Zillow affiliate.

Citations

1. Grootendorst, Maarten. “BERTopic: Neural Topic Modeling with a Class-Based TF-IDF Procedure.” arXiv preprint arXiv:2203.05794, 2022.
2. Devlin, Jacob, et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” 2019. arXiv,  https://arxiv.org/abs/1810.04805.
3. Es, Shahul, et al. “RAGAS: Automated Evaluation of Retrieval Augmented Generation.” 2023. arXiv,  https://arxiv.org/abs/2309.15217.
4. Wei, Jason, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” 2023. arXiv,  https://arxiv.org/abs/2201.11903