I’ve always been passionate about open source, and Zillow Group’s involvement in open source made for an easy decision to start working here. Although Zillow Group (ZG) is open-source friendly, the motivations, methods, and future plans for open source participation haven’t always been clear.
In this article, I’ll share my perspective as a senior software engineer about:
I hope this clarifies ZG’s relationship with open source for others considering working here, or contributing to our open source projects!
Like most modern tech companies, Zillow Group is heavily involved in open source software (OSS) as part of developing our technology. Participating in open source allows us to efficiently develop our technology, grow and maintain our engineering teams, and support a healthy ecosystem of code and knowledge sharing that benefits everyone involved.
Participating in open source multiplies engineering impact, allows us to collaborate on mission-critical software, and results in higher code quality all around.
Perhaps the most common motivator for participating in open source is engineering efficiency. Open source can multiply development impact.
Consuming open source projects means you don’t have to reinvent the wheel. Creating open source projects means your wheel can be used by others, and benefit from the associated community contributions. Why create and maintain bespoke solutions with a small internal team when we could create popular open source projects collectively maintained by many individuals across the globe?
“…releasing work as open source and the corresponding contribution process eventually result in a higher return on the initial investment made versus the alternative closed source process.” – Why open source? by Google licensed under CC BY 4.0
By nature, open source encourages public collaboration. In the same vein, open source participation allows us to contribute improvements to projects critical to ZG’s success. For example, ZG employees contributed back to Grafana, Lerna, and OpenTelemetry, all of which are used heavily at ZG. (See Contributing back to the OSS we use below.) These contributions benefit ZG and other users of the projects at the same time.
Popular open source projects tend to be higher quality than software developed by small, internal teams. Popular open source projects benefit from many contributors with diverse perspectives, backgrounds, and experiences.
“Given enough eyeballs, all bugs are shallow” – “Linus’s law” from The Cathedral and the Bazaar by Eric Raymond
Regardless of a project’s popularity, open source projects can also benefit from the added positive pressures of public scrutiny and personal reputation.
“I think writing code as open source is an excellent way to encourage people to write better code. With an internal project, it’s easy to cut corners and write code that only supports your specific use case; with an external project, you need to consider how other people might consume it, and I think this usually produces a better, more reusable interface.” – Brian Stone, senior principal software engineer at ZG
Adopting popular OSS improves the software development experience by reducing engineering toil as they tend to be higher quality, more secure, and follow standard patterns to interface with other OSS. Participation in open source also helps engineers gain transferable skills, and build their professional reputations.
“Half of [5,500+ respondents] say that their open source work was somewhat or very important in getting their current role. Open source work helps people build their professional reputation.” – Open Source Survey by GitHub
Participating in OSS exposes software engineering candidates to ZG technology, connects them with ZG software engineers, and sends positive signals that ZG takes open source seriously. Additionally, adopting open source projects helps ZG new-hires be productive sooner if they’re already familiar with them.
“…onboarding, a long and expensive process, is that much easier when new hires are already familiar with some of the technology and the community building and supporting it.” – Why open source? by Google licenced under CC BY 4.0
Contributing to common open source solutions benefits everyone involved. Users of open source projects can contribute bug fixes and features while avoiding the overhead of project ownership. At the same time, owners of open source projects benefit from contributions from the OSS community.
“I believe we have a social duty to contribute back to projects that we extract great value from. We should also contribute our own projects, when sensible, to expand the ecosystem of quality software” – Daniel Stockman, former principal software engineer at ZG
Benevolence is a common personal motivator for open source participants, but it also aligns with several of ZG’s core values:
ZG participates in open source in many ways. We have three categories of participation:
Like most open source participants, ZG’s primary mode of participation is consuming and adopting OSS, which we use to build, document, test, deploy, operate, and monitor our software.
The vast majority of the ZG codebase depends heavily on open source languages, frameworks, libraries, and tools to build and test our software. It’s difficult to measure how many OSS projects we use, but with hundreds of services each with hundreds of direct and transitive open source dependencies, it easily numbers in the hundreds of thousands.
In addition to the dependencies we use to build and test our software, ZG uses many open source tools in nearly every phase of our software engineering process. Here are a few examples:
On and off the clock, ZG engineers contribute back to the open source community by submitting bug fixes, features, issues, documentation, and maintenance to non-ZG projects, a few examples of which are Lerna, OpenTelemetry, and Grafana.
Lerna is a tool for managing JavaScript projects with multiple npm packages. As a lead maintainer of Lerna, Daniel Stockman has made many contributions to the project on his own time.
“I was looking for a way to improve the sharing of code between teams via npm packages, but without an explosion of one-package repositories and preserving a fluent developer experience (linking between repositories is still painful, for the most part). At the time, Lerna struck a good balance toward these goals.”
Lerna has become an important component of ZG technology that powers many of ZG’s multi-package front-end projects.
OpenTelemetry is quickly becoming one of the most popular open source solutions for measuring, observing, and debugging software telemetry data.
Yusuke Tsutsumi is actively involved in the project, primarily contributing to the Python SDK, opentelemetry-python. Yusuke became involved with OpenTelemetry when his team was looking for existing open source solutions to replace internal libraries.
“We had a combination of maintenance and training overhead for our internal libraries. Coupled with the lack of capacity to add new features, my team started looking for any existing open source projects.”
We believe non-code contributions like documentation updates are important too. I like to update documentation as I’m learning about a new technology to improve my reading comprehension. For example, when learning about Grafana development, I submitted pull requests to fix and improve their documentation, which helped me learn, helped the project maintain quality documentation, and will help others learn as well.
Considering how much we benefit from consuming open source, we think it’s important to release our own open source projects in return. Examples of our more popular projects are react-slider, Luminaire, and cTDS.
React-slider is an accessible, CSS agnostic, slider component for React. Michał Powaga created react-slider, and Brian Stone has since become its primary maintainer.
“AFAIK, it was (and maybe still is), the only slider implementation that was CSS agnostic. This kept the component lightweight and allowed us to use whatever styling method we wanted. At the time though, the component did not meet accessibility guidelines, so we wanted to make those improvements.”
We use react-slider as a foundation for our slider component in ZG’s internal master design system.
Luminaire is a machine-learning library that provides automatic anomaly detection for low and high-frequency time-series data. Sayan Chakraborty is the author and primary maintainer of Luminaire.
“There is no other tool that can do a hands-off anomaly detection and therefore, it was a great opportunity for ZG to contribute back to the community (since we use several open source projects).”
Luminaire is a major part of a centralized data quality and anomaly detection service used across ZG.
cTDS (C Tabular Data Stream) is a full Python DB API-2.0-compliant Microsoft SQL Server database library for Linux, Windows, and Mac OS X. Joshua Lang is the creator and primary maintainer of cTDS.
“At the time (years ago now), the alternatives were either buggy or did not perform well in a multi-threaded application. Our team decided to write our own driver to suit our needs at the time and it evolved into the current state.”
cTDS is used across most python services at ZG that require communication with Microsoft SQL Server.
Consuming is ZG’s primary mode of open source participation. Our biggest participation opportunities involve contributing and creating, i.e., increasing our contributions to OSS, and releasing more OSS of our own.
A small group of passionate, self-motivated engineers is responsible for the majority of ZG’s open source participation, but we want to make it easier for everyone to participate. The most efficient way to facilitate participation is to improve and ratify our open source participation guidelines.
“I would suggest having a better documentation for all the developers who are new to open sourcing.” – Sayan Chakraborty’s feedback on the current open source experience at ZG
Open source participation was easier when ZG was young with fewer engineers, but we’ve grown tremendously in the last 5 years or so (nearly 10x!). Our next step is to create comprehensive guidelines to facilitate open source participation.
Some teams strive toward an “open source by default” model of development whereby the overhead of “open sourcing” internal projects can be avoided entirely. We’ve built integrations between GitHub and our internal delivery systems, but they could be improved, and include more of the software engineering process, including planning and issue tracking.
“Open source by default” is not an option for every team and every project, but since virtually all projects depend on OSS, upstream contributions should be prioritized like any internal development task.
“At a minimum developers need to be involved in open source simply to ensure the code they’re responsible for continues to function, is secure, etc.” – Joshua Lang, principal software engineer at ZG
The benefits of open source increase with participation, but so do the associated risks to the company.
To mitigate these risks, our legal and security teams review certain kinds of open source participation, e.g., when open sourcing an internal project. However, as open source participation grows, there will be increased demand for legal and security reviews. How can we maintain or reduce the review workload while continuing to mitigate the associated risks?
Zillow Group’s involvement in open source made it easy for me to choose to start working here, but the motivations, methods, and future plans for open source participation weren’t always clear. I hope this article was enlightening for anyone considering working at ZG, contributing to our open source projects, or who is simply interested in how open source is involved in software engineering at Zillow Group!
If you’d like to learn more, please feel free to check out ZG’s GitHub organization where we host our open source projects, or shoot us an email at opensource@zillowgroup.com.
Thanks for reading!
Rob McGuire, senior software engineer at ZG
People were interviewed and quoted in this article:
Sources researched and referenced in this article: