Migrating HotPads to an Isomorphic Stack with React
In December 2014, the HotPads team made the decision to do a complete rewrite of hotpads.com to an isomorphic stack using React. Despite some initial hesitation to veer from a tried and true stack, we jumped straight into it.
Motivation for the Rewrite to React
Our legacy stack consisted of JSPs, jQuery with AMD Require.js, and global CSS stylesheets. After 10 years of development and constant evolution, our front-end codebase became unwieldy and difficult to manage.
The front-end stack was inherently tied into our massive back-end infrastructure, and was a major limiting factor for development. It became cumbersome to ramp up new front-end developers, who had to learn the entire monolithic codebase and keep up with many unrelated, although critical, environment updates. Adding dynamic content via AJAX required separate code paths on both the front-end and back-end, which demanded lots of extra work to create a modern web experience.
The comprehensive body of knowledge became too much for any one developer to fully master in a reasonable timeframe. We finally decided to completely rewrite our web application infrastructure, with one of the major goals being to separate out the front-end codebase into its own smaller domain.
The front-end domain encompassed all the back-end infrastructure.
Client-Side Rendering versus Server-Side Rendering
One of the most important decisions was whether to have a client-side rendered app or a server-side rendered app.
Websites have traditionally been server-side rendered apps. The key point is that in server-side rendering, the server’s response (i.e. the HTML that is sent down to the client) contains the data rendered in the HTML.
In recent times (e.g. Angular.js, Ember.js), there has been a large push towards client-side apps, also known as single-page apps or single-page architecture. Client-side apps typically serve up a static ‘shell’ HTML page. The client-side app must wait for the browser to parse and execute a javascript file.
Loaded response from server-side rendered content versus empty response from client-side rendered content.
The benefit of client-side apps is the seamless interactions on the web app, once the app is fully loaded. Transitions to new pages are instant, thanks to the parsed javascript on the browser. New pages don’t require a HTTP request or the associated overhead.
Another benefit of client-side apps is in unifying the architecture across web and mobile apps. Similar to native mobile apps, client-side web apps act as separate entities from the server, and fetch data through AJAX/XHTTP requests. This helps us enforce a separation of domains for the web app, while also allowing a uniform architecture across the web app and native app platforms.
Server-side rendering excels where client-side rendering falls short: initial page load and SEO. A browser will immediately display any server-side rendered page content — there is no need to wait for javascript parsing and api calls to complete before populating the page.
This is crucial for SEO. Search engine bots typically* digest only the server-side rendered content. Bots do a simple fetch to a URL, analyze the content, and continue. In a client-side app, most of the content is rendered dynamically by the browser, and therefore there is little that can be indexed by a bot that analyzes static content.
* Search engines such as Google are experimenting with parsing javascript and consequently understanding DOM manipulations, but it is far from a sure thing.
Our Isomorphic Stack
We researched the variety of existing and up-and-coming Javascript tools and frameworks. Some were too bloated for our usage, others were too opinionated (“you have to do it the Angular.js way”), and others were too free-form (lack of structure around jQuery leads to spaghetti code). We finally settled on what many other developers have also settled on: React and flux.
After deciding what technology to use, it was time to work on the architecture. We wanted the initial speed and SEO benefits of server-side apps, but also the seamless experience of a client-side app. We also wanted the front-end to be able to communicate with our back-end services. This would allow for more uniformity and feature parity across the web and our native apps. We did not, however, want to duplicate all logic in independent server and client apps.
To achieve the goal in an efficient way, we needed a setup wherein we could share code between the client and the server. This is known as an isomorphic stack — one single codebase that runs on both platforms. In our solution, the combined Node.js server and client bundles operate as a single, separate entity that consumes our back-end services. Similar to a mobile app, the Node.js server and client bundles each request data from the main API. The result is that any given page can be both server-side rendered and client-side rendered.
The domain of the front-end stack is now separate from our back-end.
Lessons We Learned with React
As for the technology, our team has thoroughly enjoyed working with React and flux. React has quickly gained popularity in the tech community. We won’t cover the basics — there are plenty of guides out there on how to get started. However, we would like to share some of the lessons we learned throughout the project.
Shift of separations of concerns: Not by technology, but by functionality
The shift from “separation of concerns by technology“ to “separation of concerns by functionality“ is the first mental hurdle to overcome. In a typical MVC, controllers (the Javascript) and views (the HTML/template) exist in different files — the code is separated by technology. Most of the time, the view and the controller are tightly coupled.
React takes a different approach — using components to separate concerns by functionality. Components are blocks of combined UI code (JSX/HTML/CSS/etc.) that encapsulate both the controller and view in the same file.
Write declarative code
Writing declarative code means writing code that is predictable and driven by the component’s (or application’s) state. The content on the page does not depend on time or a series of events, but rather only on the state of the component. React is inherently declarative. However, the line between declarative and imperative code can be blurred when incorporating third-party packages.
Much of the code that has been developed relies on tracking data in the DOM. As an example, think of Bootstrap’s modal component. The programatic API to activate the modal dialog uses a reference to the modal object in the DOM, namely $(‘#myModal’).modal(‘show’). If you want to know the modal’s state, you must consult the DOM. When integrating imperative code into a React project, it is important to expose a declarative interface for them. By controlling the modal’s state in the application, we make the application more predictable — we know what will be rendered for the given state. The ability to efficiently re-render the page on any state change is one of the greatest benefits of React — take advantage of it!
Understanding the app’s data layer and structure: Optimizing client-side performance
React components re-render when they get any new state or properties (“props”). If a component’s state or props doesn’t change, no DOM manipulations will be made. In un-optimized React code, the virtual DOM would typically be calculated and compared. By doing a shallow comparison on the state and props in the shouldComponentUpdate() lifecycle method, we can stop the virtual DOM calculation. The virtual DOM is fast, but in a large application, you may have hundreds or thousands of components, many of which probably do not need to update on every change. Those milliseconds add up quickly. In building out our application, we found events to be excruciatingly slow on older phones and older Windows machines. By implementing this optimization, we were able to restore performance on our apps.
Incremental performance now becomes an optimization problem. The less objects you pass around, the less comparisons and less re-renders you have. But if you abstract the objects too high up, your component may not re-render when it should! By understanding and better segmentation of the data, we will able to incrementally improve it.
Checkout PureRenderMixin for React’s mixin solution. Also noteworthy: The official binding for React in Redux, react-redux, gives you this optimization by default when calling the connect() method on a component.
Server-side rendering with async data and server-side optimizations
In order to render the application on our server, we must fetch the appropriate data for the requested page before sending down the HTML response. Leveraging react-router, we were able to architect a solution that allowed us to define a static method on route components that returns a promise. On the server, we execute all the promises for a given route in parallel and wait for all of them to complete. Once completed, we send the response back to the client. All of our data fetching happens in parallel — the bottleneck is now limited to the longest API call. Being able to wrap requests for different data sets in their respective components also helps us keep the codebase light and uncluttered.
Another big win for us was optimizing static asset delivery. As soon as we generate the full HTTP response for the incoming request, we stream across the HTML headers first, then we load the data and render the rest of the app into markup. Doing this allows the client’s browser to download Javascript files, CSS files, and any other assets linked in the header before even receiving the full HTML response. By streaming the headers, we are able to achieve significant gains in TTFB (Time to First Byte), first-render, and page-load times.
Webpack & better organization through declarative dependencies
Using webpack we are able to explicitly import almost any type of file in a common Javascript module. We have a 1:1 relationship between components and their SCSS stylesheets — each component explicitly requires its own styles. All our CSS rules are scoped to that one particular component. This significantly reduces conflicts with class names and rule importance. This has helped improve the readability of our source code.
As you can imagine, Node.js does not support this bizarre convention. Node.js cannot import an SCSS file. The code that runs on the server must be pre-processed, similar to the client-side bundle. With webpack, we actually build not only the client-bundle.js file for the client, but also a server-entry.js file which runs on the server. The source code must be built for the each of the different environments; each environment has slightly different build configurations. This was a significant mental hurdle we had to overcome.
Our single code base gets built in a different way for each environment.
Aftermath
We were able to launch our first pages in Node.js just three months after we began the rewrite. It took another nine months to finally launch the full site, which included some rather large changes. Since the launch, we have seen benefits from our new isomorphic stack in both website performance and code organization.
Performance metrics have seen significant improvement. Start render times are faster by 8.3% (despite now executing an HTTPS handshake). Full page load times are faster by 18%. Page-views per user session are up 50%. These improvements are critical for user perception as well as SEO. As would follow, we have seen a meaningful uptick in SEO traffic in the short while we’ve had the new site.
Switching to a service-oriented architecture has made the front-end repository a more manageable domain. Everyone on the team is able to work across the project and understand it fully. Having this smaller domain has also simplified our development environment setup and workflow. Front-end developers only have to run front-end code locally on their development machines, and can connect to back-end API servers on a shared test environment. As for the codebase, using only declarative dependencies makes it easy to track down resources. We spend more time writing new code than fixing bugs. We send down less Javascript than our legacy site did (even though we now send down the entire app!)
Working with a declarative, isomorphic code base has made a big impact at HotPads. This maintainable, future-proof solution combines the best of both server and client-side rendering. Having Javascript as a common language across the server and client codebases eliminates the need to switch mental context between different languages. The uni-directional flow architecture of React and flux provides clarity and structure to the code, making it easy to track down bugs. Overall, we have been very satisfied with the new architecture, and the new isomorphic stack has worked very well for us.