To Gatsby and Beyond!

This is our fourth and final post on Gatsby. The first post¹ introduced Gatsby by explaining its product vision. In the second one², we discussed several architectural views and the third one³ explored technical debt. This essay will cover another architectural concept influencing Gatsby’s flexibility and maintainability: variability management. Due to Gatsby’s plugin architecture with over 1800⁴ plugins, it has an “infinite” number of different possible configurations. Still, everything works together surprisingly well!

First, let’s get an idea of the main variable features. That is, features that are not hardwired in the product, but that you as a user, can enable, disable and play around with. When you start making a website with Gatsby, you can get a head start using a theme⁵ or sub-theme. If that is not to your liking, you can also have a look at the hundreds of starters⁶ that are available. Filling your site with data is done with (again hundreds of) data sourcing⁷ and transformer⁸ plugins, which allow you to fetch data from different sources, to transform that data to different formats and to customize⁹ the exposed GraphQL schema. But Gatsby’s variability doesn’t stop there. Developers can adjust the workflow to their needs with custom linting rules¹⁰, and custom Webpack¹¹ and Babel¹² configurations . Furthermore, Gatsby supports all long term supported Node.js versions¹³ and allows developers to specify on which browsers the website should run using browserlist¹⁴. Finally, when you are done building your website and its time to ship it, you can choose any static web hosting service¹⁵. All in all, we would say that there are enough options for most people.

Feature model

Since Gatsby has so many variable features, putting all of them together into a feature model wouldn’t show you much. Therefore this section looks at the plugins feature only. Each plugin you write or use has access to some specific apis. We’ll look into how the features exposed to the plugins are composed. The Gatsby team wants you to be able to write as much custom functionality as you desire, without bothering you about stuff you don’t want to use¹⁶. This makes creating a new plugin the least cumbersome. Gatsby does this by checking whether a plugin implements some pre-specified functions. If such a function is exposed, it will be used in the process. Figure 2 shows that plugins can implement three API’s for Gatsby to call when running.

First, there is the Browser API¹⁷, which allows the plugin to interact with the browser. This enables plugins for instance to register a service worker, which handles page caching and lazy-loading, or to perform actions on route updates, such as tracking analytics. Secondly, there is the Node API ¹⁸, which is responsible for actions performed in or around the build stage. Cleanup of intermediate plugin outputs or confirming the build has correctly outputted required files can be done with post-processing in the onPostBuild. Finally, there is the Server Side Rendering API¹⁹, which allows plugins to change content served to the user. These three APIs together allow the user to implement any functionality in Gatsby.All the APIs are inherently optional, so users don’t have to write boilerplate.

An example of a commonly used configuration would use Node.js V12 (LTS) with Yarn for managing npm packages. It would support not dead[https://github.com/browserslist/browserslist] browsers, deploy directly to Netlify and use basic plugins such as:

• gatsby-source-filesystem for reading files from local filesystem
• gatsby-transformer-remark for transforming Markdown files into GraphQL format
• gatsby-plugin-sharp for transforming and optimizing images
• gatsby-transformer-sharp for exposing transformed images

A more exotic configuration could for example extend the Babel configuration with @babel/plugin-proposal-optional-chaining, which allows the new JavaScript maybeObject?.field syntax. Additionally it could use the source plugins gatsby-source-contentful and gatsby-source-airtable and stitch their data together with GraphQL schema customization using the createResolvers API in gatsby-node.js. Deployment could for example be done via Gatsby Cloud on Zeit Now.

That said, it is near impossible to guarantee that all possible combinations of configurations are bug-free. We analyzed all issues and pull requests labeled type: bug and status: confirmed in the Gatsby main repo. Surprisingly, this yielded only 5 issues concerning incompatible configurations. This issue²⁰ describes a failure in data fetching in case of a naming conflict between a theme name and a query name. An example of browser incompatibility is described here²¹, showing that images are not properly fading in Safari. The two plugins for KaTeX and MDX are not working together²² yet. It has also happened that a specific plugin configuration does not work in a specific environment²³ (CSS modules in development using Less.js), or that the combination²⁴ of two plugins and environment gave rise to problems (offline and style components plugins in development). It can be expected that the more variables Gatsby introduces, the more of these issues will arise. How does Gatsby plan on dealing with so many factors?

Variability management

How variability is managed depends on whom you ask. Different stakeholders might give different answers. For example, the end-users (web visitors) will not even know if a page was made with Gatsby or not, let alone in which configuration. To find information about different configurations, web vendors can consult the partnering walkthrough²⁵ or reach out by mail²⁶. Developers and web agencies can make use of Gatsby’s extensive documentation, or the documentation of configuration tools like Babel and Webpack. Additionally, each plugin has its own documentation (some even have their own webpage²⁷). Gatsby maintainers have their own team on GitHub. Finally, everyone can raise issues on GitHub²⁸ or ask questions on Discord²⁹.

However, the different ways to get information or report issues can hardly be called management. How does Gatsby actually prevent issues caused by variability? Right now, for every change, tests are executed using the pipeline discussed in our previous³ post. This includes tests run on a Windows machine and in the cloud. Although this will certainly catch some exceptions, it can’t possibly check all different configurations, due to the enormous amount of them.

However, Gatsby has some tricks up its sleeve! Currently over 179,000³⁰ open source websites are built with Gatsby, and each has its own configuration. In this issue³¹, Gatsby is working on an ambitious project to exploit this fact. The idea is that on each update, one thousand of those websites will be selected at random. Then, they will be built using a new Gatsby version. Since the websites are chosen at random, they will have widely varying configurations. If no error is spotted in building these sites, there is a high chance that the update is free of errors. If some fail, the Gatsby team immediately has a reproducible example that can guide them while bug hunting. Gatsby is working on another³², more active approach of handling variability. In this approach, they aim to validate options provided to plug-ins, so that if they fail, they fail early with a useful error message.

Variability implementation mechanism

We have seen that there are many ways to configure a Gatsby project. But when are the configurations resolved? And how can you actually use them? The answer to the first question is simple: since Gatsby is essentially a compiler, the binding time of all the variable features is at compile-time. To answer the second question, we need to have a look at the variability implementation mechanism that Gatsby uses. We’ll explore the implementation and scalability of Gatby’s plugins and of Gatsby Themes.

Plugins are at the core of Gatsby. Each Gatsby application defines the plugins that it uses in a configuration file called gatsby-config.js. Internally, Gatsby reads this list of plugins and initializes each one in order during build time. Adding a plugin to an application is as simple as adding a new line to the gatsby-config.js file. Updating a plugin is as simple as installing the new version of the npm package using yarn add <plugin>@latest. Removing a plugin from a project takes a bit more time. If there are any queries depending on this plugin. In this case, the developer needs to edit all queries that depend on the plugin.

We haven’t found any reason why the overall concept of plugins would not scale. One design choice that is currently in place, however will not scale infinitely. The order in which the plugins are specified, determines if the dependencies on each other are resolved correctly. The Gatsby team is working on a way to specify the dependencies plugins have on other plugins. This change would make it easier to add new plugins to a project already containing a lot of plugins. In major semantic versions of Gatsby, they might drop support for some of their exposed API’s. Since plugins make use of the API’s Gatsby exposes this means that every plugin that depended on the dropped API, needs to be adapted. Given the amount of plugins in existence, this can become bothersome.

Next up: Themes. Gatsby Themes are just Gatsby websites, but without the content. They contain a predefined configuration, which can be extended by users of the theme. Extending the configuration can be done in a straightforward way using Gatsby’s configuration files gatsby-(config|node|browser|ssr).js. Themes also provide layouts and page elements as React components. These React components can be ‘shadowed’³³ by creating a file with the same name in the project directory. For example, if the component is located at some-theme/src/components/FooComponent.jsx, it can be shadowed by creating a file my-gatsby-website/src/some-theme/components/FooComponent.jsx. This way, anything in a theme can be overridden. Since Gatsby Themes are just self-contained Gatsby configurations and anything in there can be overridden, makes them as scalable as Gatsby itself.

With that, we’ve come to the end of our fourth and final post of our series on Gatsby. We really hope you enjoyed reading them, and perhaps even more importantly, that they were useful to you. If you are interested in building websites with Gatsby, or in contributing to the Gatsby project, you should now know where to go. We’ll see you there.