REST assured with web service contracts
Our Collaborative REST API Process at Zillow Premier Agent
So, your engineering team is building out a great new feature that is going to delight the users. Where do you start? Let’s see… you have iOS, Android, and Web clients… and you have some backend web service teams that are ready to provide the data… but how does everyone come together?
Without a well-defined process, your team will likely waste lots of time stumbling through inefficiencies that could be avoided. At Zillow’s Premier Agent, we learned that cross-team collaboration and solid documentation are the key to streamlined feature development. Here, we will cover the open-source tools we use, along with the process we developed in order to step up our game.
As we step through the process in this blog, we will build out a tutorial project using what we learned in our experience. In the tutorial, we will define some web service endpoints, and build an Android client to consume them. For the sake of brevity, we will not go into any detail about how to build an Android client, nor will we dive too deeply into the various OpenAPI tools used. But you can follow the links provided throughout the blog for further reading.
Now, before we jump into our process, let’s briefly visit our past to help illustrate why we had to change.
Before… The Dark Ages
In the past, our approach was less than ideal. Backend teams would get their work done first, in order for client teams to have endpoints ready to work with. These backend teams would create the specs (or sometimes not), and then build out their services. After that, the clients would get started on their work, hoping these backend services were ready to go.
Documentation?
Before getting started, clients would have to track down the documentation on the API specifications the backend teams created. But, depending on the engineer(s) who worked on the feature, these specs could be anywhere. Sometimes there were no specs; sometimes they were in a JIRA ticket; other times, one might have found them in a team’s wiki.
Roadblocks
There would often be issues with the specifications and/or the implementations, which would require amendments to be made by the backend teams. At this point, those backend teams had moved onto other things, so these changes would impede progress on their next projects; and, in turn, the client teams would be blocked until those changes were completed. This ended up in a vicious cycle of change requests, causing roadblocks for everyone involved.
Diagram of wasted time
The Contract Process… Our Renaissance
To address the issues mentioned above, we developed a formal contract process that we follow for every feature we create. This process requires key members from each team involved to get together, and design the API before development begins on either platform. The output of this process is a well-documented and well-defined set of REST API specifications that are referenced throughout the lifetime of the feature’s development.
Contract process
Prototype
Members of each team should spend some time together to prototype the feature work, and eventually end up with a REST API specification. This usually involves at least one client and one backend engineer. In this phase, engineers can develop a shared understanding of the work that needs to be done on either side. The backend engineer has the understanding of the data source and composition, while the mobile engineer has the understanding of the data presentation and user experience. Prototyping together should help reveal some “gotchas” up front, rather than later on when they’re more expensive to address.
So, let’s introduce our tutorial project. We will create a modified version of the OpenAPI petstore example contracts, as well as an Android client to consume them. Let’s assume we have finished the prototype phase, and decided on two endpoints:
- Store Info — This will return basic information about our pet store
- Pets — This will return a list of all of the pets available in the store
YAML Specification
Once a prototype is created, the engineers involved will draft a preliminary specification of the REST API. This is where OpenAPI and Swagger Editor come in. In short, OpenAPI is a standard used to describe REST APIs. And the Swagger Editor is a browser-based editor where these specifications are written. Using the Swagger Editor, we can easily build out YAML specifications that define our web service endpoints, and the operations that can be carried out. This specification is added to our contracts repository so it can be versioned and reviewed.
In our contracts repository, we will create two YAML files — one for each endpoint.
Our store info endpoint — Source is here
Our pet listing endpoint — Source is here
Using Swagger Editor, we can easily build out these schemas. Any syntax errors will be highlighted by the Editor, so we can be sure to create valid contracts before building out the feature.
Feedback & Finalized YAML
Now that a YAML specification has been created, a pull request is opened, and the rest of the engineers involved in the development of the feature provide feedback. This phase allows backend and client engineers to discover any issues with the specification, as well as arrive at a final agreement on the REST API definition. Once the pull request has been merged, we tag it and upload it to our Swagger UI. Swagger UI provides a very helpful browsing experience for our existing contracts, and is great for documentation.
For the sake of our tutorial, let’s assume the YAML files we created have been accepted in the feedback process.
Build
Now, the fun begins. With a well-defined and well-documented REST API specification, both client and backend teams can work in parallel to build out the feature. If any issues arise that require an amendment to the specification during development, a PR with the changes is opened, reviewed, and uploaded to the Swagger UI, like in the Feedback phase.
Now let’s create our Android client that will consume the APIs created by our pet store web services. This will be a very simple Android app, just to demonstrate how one might map a contract into code. Things like dependency injection, and other good practices, have been omitted for the sake of simplicity.
First, using Retrofit with Gson, we will create a network layer that parses the JSON response into network objects. The serialization fields will match those defined in our contract schemas.
For example, here is a retrofit interface for our pet store api. Click here to see our network layer
One of our network models. Notice that the serialized names match those in the YAML specs.
Next, we will create mappers that convert those objects into something our domain layer will understand. Our domain layer consists of the business logic that runs our app.
PetStoreMappers.kt maps our network models into domain layer models.
Now that we have fleshed out our network layer, we can build a simple UI in our presentation layer to display our data. And with that, we have completed our basic Android client.
Using Charles Proxy, we can generate static JSON data (through the Map Local tool) that matches the schema defined in our contracts. By enabling our device/emulator to proxy through Charles, we have created a client off of our YAML specs without the need of a backend service!
With this contract process (and these great open source tools), we have been able to cut down on a significant amount of the engineering time it takes to build out a feature. And, now, our documentation is standardized and put in one place, thanks to OpenAPI and Swagger.
Backend and client teams can now work in parallel on the same feature.
Further Improvements
Despite having a formal process between all teams involved, issues still arise that should be addressed. Namely, human error and code duplication.
Issue #1: Human Error
The OpenAPI spec is just documentation, really. The actual code that backend and client teams write is translated into objects (POJOs, for instance) from the contract. If typos make it into the final contract, which is not uncommon, they might be “corrected” when translating into POJOS. For instance, if a field in the contract is defined with “transcation” (meant to be “transaction”), one team might translate that into code with the corrected “transaction” without realizing the typo, while another team might have copy-pasted the incorrect spelling into their code. Typos like this often result in subtle bugs that can be difficult to track down.
Varying definitions of the same business entities turn into lots of redundant client dependencies
Solution
To address this, we recommend you use code generation. Swagger Codegen and OpenAPI Generator are some great open-source projects for this purpose. If backend and client teams all use code generation, even those nasty typos will be referenced correctly (i.e. everyone will have “transcation” in their network objects)!
Let’s update our contract repository and android project to use code generation. By adding this step, we will have a gradle task that will generate those POJOs for us. We will add the Swagger codegen gradle plugin to the contract project for this purpose:
Code generation build.gradle for our contract repository project
Now, we can generate code and assemble it with this command:
./gradlew clean :codegen:generateSwaggerCode assembleNext, we copy the generated jars into our android project. Having generated code in our project, we can remove the network models we created manually, and update the mappers and repository to use the generated code instead.
Our updated PetStoreMappers.kt
Rebuild, and you should see the same UI as before.
Issue #2: Duplication
As more services are introduced, you will likely run into object duplication. If your business has, say, a “User” object, that object will likely be used in many different contracts. Or worse, some services might have a slightly different definition of the same business object than other services, causing even more code bloat. The code created off of these specs can add up, causing unnecessarily large client dependencies.
Single (shared) definitions of business entities remove redundancies in client dependencies.
It’s time to refactor our repository project to move shared object definitions into their own YAMLs, and update $refs to them from our endpoint contracts.
Create a new ‘shared’ directory, and place reusable YAML definitions in there.
For example, our pet is defined in a separate YAML that can now be reused.
Now, our endpoints, like this pets.yaml spec, have updated $refs to pull in reusable definitions
In order for our code generators to find all of our reusable definitions, we need to create a dummy contract that includes all of them in one place. Then we update the code generation gradle configuration to account for these shared objects. Now, we can rerun code generation, and pull the latest jars into the android project
Now that we have our latest jars, let’s refactor the pet store project to account for these shared objects. We will just have to update our mappers, like so:
Updated PetStoreMappers.kt that accounts for shared objects.
And we’re done. Now our artifacts are smaller, and our mapping code has been reduced.
Takeaways
Hopefully, this post has been helpful in identifying ways to improve your organization’s cross-team development process. Basically, our experience boils down to these key takeaways:
- Plan up front with all teams involved. Get everyone on the same page and discover possible issues before you get to coding.
- Use a well-defined contract. This provides great documentation, as well as a way for all teams to work in parallel.
- Use code generation. This cuts down on human error and development time.
- Reuse objects. This reduces code bloat, and even more development time.
