Last year, Zillow began to use a technology we’ve developed to keep our mobile apps and the web site current with latest product innovations: embedded HTML that’s updated asynchronously by both Android and iOS applications.

Background

Zillow’s success is driven by its rapid innovation and much of that focuses on the real estate and rentals “home details page.” This is where the home shoppers (and voyeurs) go to see a home, its photos, the Zestimate and lots of other facts.

Home Details Page (to the right of the map) on iPad

Zillow’s early mobile applications were built with native controls and application navigation, but loaded a WebView with the home details page rendered by the server. This resulted in a significant delay for the user, so, as the native apps became more important, we moved some of the home details into native controls. These were laid in above the WebView (the portion between the red bars in the illustration), with the remainder still being rendered by the server. This got information to the user faster, but made it more difficult to implement product innovations across iOS, Android and web platforms.

What we set out to do

With the home details screen split between a native implementation of the UI for the most basic listing information and a WebView that has to wait for HTML to arrive from the server and be rendered, innovation was becoming more difficult. The server-rendered HTML has to be consistent with the native UI that sits on top of it, restricting changes that can be made to the structure and style of the page. At the same time, more users have iPads and tablets where the larger display means that the small portion of the page handled by the native UI doesn’t fill the screen.

We wanted a solution that would facilitate innovation and get information in front of users faster – native-app features while allowing web developers to use their skills to control the experience. We want developers to implement an innovation once, so we provide a single point of control for the structure, style and content of the home details page. We also wanted it to allow the native apps to display the home details more rapidly than they could when the HTML is rendered by the server – filling the screen from the upper red line in the illustration above to the bottom of the display and beyond.

Architecture

In order to deliver on these goals, we wanted the iOS and Android apps to be built with the HTML needed to render the home details page, so first-time users get a snappy response instead of waiting while an empty cache is populated from server requests. That’s in addition to being able to download the latest HTML whenever we enhance the home details page. We decided that the runtime architecture would encompass adding a manifest of resources to the response of the ClientConfiguration API. We built a resource manager for iOS and Android apps to receive the manifest and then request any resources that it doesn’t already have from a static server.

Runtime Deployment

This requires a build process that creates the manifest and packages the resources so they can be consumed in the native app builds and also be deployed to a web server for download. The build process produces encapsulated resources, described below.

Flow of resources from packaging to deployment.

The HTML starts at the top with resource packaging. The HTML is encapsulated – all the JavaScript, CSS and image (sprite) resources it needs are combined (‘inlined’) into a single HTML file so there’s no need to resolve those dependencies when the HTML is displayed. The packaging process also produces the manifest file that gives the location (URI) and digest (SHA256) of each encapsulated resource. The encapsulated resources are made available for download while the manifest is made available on the API server used by the mobile applications. As noted earlier, the application builds also consume the encapsulated resources and include them into the application bundles.

When the mobile app starts, it calls the API server to retrieve configuration updates. The response includes the manifest from the latest build that was deployed. The ResourceManager component keeps track of the resources that are present in the app, and if the resource specified by the manifest has a different digest from the one already present, it initiates download of the new resources and replaces the old ones in persistent storage.

In addition to the digest, each resource supplies an api version property that’s communicated to the mobile app through the manifest. This api version is used to request the appropriate data (JSON) to be supplied to the resource for rendering.

For more details, see below.

iOS & Android

As noted above, we created a resource manager for our iOS and Android apps. The resource manager has two functions:

• Update the resource when the manifest specifies a digest that differs from the resource the app has.
• Provide a method for native code to get access to the current resource given its name.

When a new resource is available, the resource manager initiates a download and allows the app to proceed to use the resource that it already has until the download is completed. Since the URI includes the digest value, these resources can be accessed through a CDN for quick downloads.

The mobile app initializes a WebView with the encapsulated resource (an HTML file generated by the resource packaging process described below) at startup. This WebView parses all the JavaScript and CSS before the user requests to see it, at which time the native code sends a JSON object to the JavaScript API defined by JavaScript in the WebView. The JavaScript API consists of a render function and a reset function. The render function renders a Mustache template with the supplied JSON and inserts the result into a div element of the document that’s already loaded into the WebView. The reset function removes the children of the div in the document and cleans up any outstanding events.

As noted above, the native app has to cache and reuse a WebView once the encapsulated resource is loaded into it. WebViews were not originally designed to be used in this fashion. For example, “Page Loaded” events are not normally emitted when the resource is done processing JSON data; we had to manufacture those. In Android, we need to register our javascript-to-native bridge interface with the WebView before we load the resource, but each time it displays new JSON data, we needed to bind a corresponding object to the javascript-to-native bridge. This was achieved by implementing a two-phase javascript-to-native interface, where a permanent handler delegates to an instance-specific handler. Finally, we had a lot of trials to get the transition from one set of JSON data to another appear smooth (using the “render” and “reset” functions discussed above). We built this into controller and view classes that encapsulate the native WebView.

Build and Tools

As noted above, the HTML resources are encapsulated so that there are no assets that need to be loaded in order to render the WebView. We did this to assure a consistent, rapid response even for first-time users, while also avoiding the difficulties of assuring the correct version of each of the JavaScript and CSS resources used by the page are loaded. Instead of referencing the dependencies at load time, we decided to ‘inline’ the dependencies into the resource at build time.

Since Zillow uses YUI as the JavaScript (and also CSS) module manager for the web front-end, the resource packaging process takes an HTML file that has script tags for the YUI seed and a few other bootstrapping scripts, tags for the div that will hold the result of expanding the Mustache and a YUI().use() block where it defines the render and reset APIs needed by the native code. The process then uses YUI’s Loader to resolve JavaScript and CSS dependencies. The dependecy-resolve tool produces an HTML file with tags to load each of the dependencies according to YUI.

The HTML file with added script tags is then put through an ‘inliner‘. The tool we use is essentially the same as the one developed by Remy Sharp of Left Logic. The version we started with required that the HTML (and other assets) be loaded through and HTTP server; since that added some complexity to the build, we made it work with on-disk files and also added an ability to search a list of folders when attempting to open files. The tool includes the ability to minify the JavaScript and CSS it processes, but the uglify minifier it uses causes problems with our YUI code, so we use the YUI compressor to minify the JavaScript before running the inliner.

In addition to the tools for encapsulating the resources for deployment, we found a need for a couple of tools to facilitate development. We built a preview tool that drives mobile-to-JavaScript calls in the developer’s browser and incorporates the build steps so the resource can be loaded dynamically and developers can do the familiar change-refresh-change-refresh cycle that keeps them productive. Since some interactions really need to be seen running in the native app, we developed a tool that works as a proxy for the API server and assures that a mobile app using the proxy for API requests will always try to download a new version of its resources, and when it does, the resources will be generated from source. Not quite as quick a turnaround as reloading with the preview tool, but way better than rebuilding the native app (and the only tooling it requires is nodeJS and NPM).

Testing

A main goal of the project is to assure a quick response for users, so in addition to performance testing in the lab, there is instrumentation that generates timing for each phase of processing the display both in the native code’s set up and the JavaScript rendering.

Home Details Display times – encapsulated vs. legacy.

With this technology, users receive the information they want in half the time. Moreover, the response time is the same for first-time users, while the server-rendered HTML is often significantly slower than the average when its resources aren’t cached. The timings of iOS and Android devices don’t account for the age/type of devices used, so this chart doesn’t necessarily reflect a performance difference for the platforms. There is no instrumentation for the legacy Android apps, so there are no figures to compare, though it’s easy to observe that Android performs similarly to iOS.

Since the JavaScript is built with YUI, the build process automatically executes YUI unit tests and integration tests. In addition to these tests, we generate screen shots of the HTML rendered for each of 27 different combinations of data that can be reviewed for accuracy.

All in All

We set out to accelerate users’ ability to view listing information and Zillow’s ability to deliver innovations in mobile apps. We’ve taken the next step in blending native components with WebViews without compromising performance.

Video:  Loading the old Home Details page compared with Encapsulated Resource Display