Deciding to write a Static Site Generator

For a while I’ve grown increasingly frustrated with my current static site generator (Gatsby). It does the things I want well enough, but also adds way too much bloat to the site (which you can partially remove with a plugin…). My main frustration with it arose when I wanted to add support for a new image format, but its plugin architecture (while presumably “powerful”) made this way too complicated.

So at the start of September I gradually arrived at the only logical conclusion of having to write my own static site generator that would fit my needs. (Insert xkcd #927 but it’s less about standards and more about making something that works well for me.)

If you don’t care about my history with Gatsby feel free to skip to Trying out stuff.

How we got here

Some time in 2022 I started making a site with Carrd to showcase some of the art I had commissioned. I chose Carrd because I knew a lot of artists who use it to showcase their portfolio and it seemed like a simple option at that time to get things running quickly. And tbh if I’d written something from scratch I never would’ve made the site anyway (oh how the turntables have turned).

This worked well enough for a while, until I ran into the restriction of “max. 50 elements per page” of the free Carrd account. Since I’m a software developer I of course didn’t see a reason to pay for a pro account when I could just make a site myself and host it for free with Whatever Pages.

Making my own site would also give me more freedom to design things (not being tied to their WYSIWYG editor) and potentially ease the process of adding new images.

With Carrd, my current flow of adding an image looked like this:

1. Each image is wrapped in a container, together with its description
2. So I would select the Container and duplicate it
3. Then select the image and replace it by uploading a new one
4. Then update the description
5. And finally publish my changes

I was hoping to get to a flow like this:

1. Have a config file in the repository which describes for each image where it’s located and what its metadata (description etc.) should be
2. So I would add the new image to an “images” folder
3. Update the config file with the data for the image (filename, description, …)
4. Commit everything
5. A pipeline runs, the page rebuilds based on the config file, and everything gets automatically deployed!

So in May of 2023 I started looking for static site generators to hopefully replace the Carrd site – with a focus on it being able to read data from some kind of file and maybe even automatically resizing the images. I ended up with Gatsby since it seemed to tick all the boxes. Its static site generation was built on React and GraphQL, which I had both never worked with before, but I had some experience with Angular and how hard could it be.

Using Gatsby

So for the past 1.5 years I managed my site with Gatsby. And for the most part without any issues.

Here’s a short rundown of how it worked.

I have a data.yaml file which contains an array, each element containing a map of the following data:

• file – the relative path to the file in the repository
• artistName – the name of the artist who drew it
• artistUrl – an URL to the artists social media page
• rendering (added fairly recently) – sets the image-rendering CSS attribute, so pixel graphics are scaled properly

Gatsby then has a couple of plugins, which discover the resources and do image transformations. I was using the following plugins:

• gatsby-source-filesystem – this adds the data.yaml as well as the “images” folder to Gatsby’s GraphQL pipeline
• gatsby-transformer-yaml – this automatically creates GraphQL nodes for our data.yaml
• gatsby-transformer-sharp – this converts our images to different sizes and formats
• gatsby-transformer-ffmpeg – this converts video to different sizes and formats
• gatsby-plugin-image – support for displaying responsive images with gatsby-transformer-sharp
• gatsby-plugin-manifest – for generating favicions and a web manifest
• gatsby-plugin-no-javascript-utils – to remove all the unnecessary stuff Gatsby puts in our page, even though we don’t use any JavaScript
• and finally gatsby-plugin-sharp and gatsby-plugin-ffmpeg, which are called by the transformers to invoke the respective libraries

This is a lot of stuff! But the documentation was mostly fine and I didn’t have many issues writing the GraphQL that returns nodes for the transformed images and videos, and subsequently integrating them into the page.

The frustrations started when I wanted to add an animated image in form of a GIF. This in itself didn’t cause any issues because the transformers would just skip any files they didn’t support and I could add the unprocessed file to my site. But it still would be nice to resize them and have automatic conversion to other animated formats like APNG and WebP.

The easiest way (apart from not caring) would be to convert everything locally and add support for multiple image sizes to the data.yaml. But it felt weird to mix these approaches: Locally generated AND transformed with Gatsby plugins.

So I went the route of trying to write my own Gatsby plugin… I’m not gonna go into too much detail except that the whole thing is very complicated. When you create a plugin you add functions to gatsby-node.js and gatsby-worker.js which are called at different steps by Gatsby after registering it. And despite offering TypeScript support pretty much all plugins are written without it, which makes it even harder to follow how things work and how changing some parts could break the entire thing.

I spent a week or two trying things out by first using the ffmpeg and then the sharp plugin (+ transformer) as a base, but in the end gave up because adding such a simple thing as “call external program to transform the GIF” really wasn’t worth all the headaches.

What followed were three weeks of not really thinking about it trying anything. What I had now worked well enough and while I occasionally thought about writing my own thing – it always seemed like “too much” to warrant all the potential hassle involved with it.

Trying out stuff

Last Friday (Sep 13) I decided to just put “rust generate static html” into my search engine of choice and maybe find some inspiration that way. Evaluating some options and figure out what other people are doing to generate simple static sites.

It’s been about a year since I last tried out Rust and the recent discussions (drama?) about Rust in the Linux kernel lead to it being fresh on my mind. And it also seemed like a good choice, maybe, if I was looking for something performant.

The third result “Building a static site generator in 100 lines of Rust” immediately caught my intention. Only 100 lines? That could give me a good starting point for some ideas and it should be easy to develop it further into the direction I needed.

So I quickly created a new project named “Ocarrd” (since the repository for my current site was named “carrd” due to its origins, combining it with “OC” and maybe the “o” could also mean “open”) and initialized a new Rust project. After copy-pasting the 100 lines of code and making sure the example worked I went about changing things.

For the record here’s a short summary of what the example provides and my thoughts on it:

• Web Server – nice so I don’t have to manually run python3 -m http.server on the build directory
• File Watcher – while not super important it’s nice that it rebuilds automatically when a file changes
• Split between the layout as plain HTML in strings (templates.rs) and the content generated from Markdown files

I like the idea of having the HTML be part of the code. It makes it easier to conditionally apply things and you don’t have to prepare some kind of structure that gets passed to a template file.

So I tried out and rewrote the existing templates.rs using Maud, which seemed to do everything I wanted:

pub fn render_body(body: &str) -> Markup {
  html! {
    (DOCTYPE)
    html lang="en" {
      head {
        meta charset="utf-8";
        meta name="viewport" content="width=device-width, initial-scale=1";
      }
      body {
        nav {
          a href="/" { "Home" }
        }
        br;
        (PreEscaped(body))
      }
    }
  }
}

This looks nice!

It’s easy enough to read and you can insert Rust code wherever you want using (). There’s support for control structures with @if and @for – which introduces a new syntax but that would be a trade-off I’d be willing to make.

It also has a nice feature that allows you to put classes and IDs right after the tag name. Which might not add much, but would definitely reduce some boilerplate.

html! {
  input #my-id .class.another-class type="button" value="My Button";
}

Being optimistic that I could work with this I went about fixing some of the issues I had with it. For example if I ran cargo run it would always look for the content directory in the current directory and try try to create the public directory in the current directory as well. Small issue, should be easy to solve. Right?

After searching for a bit I found out about the environment variable CARGO_MANIFEST_DIR, which always points to the directory of the Cargo.toml – so the project root! I quickly updated the code to add new variables that join the Manifest directory with the existing constants but… As it turns out the constants were used in quite a lot of places, inside of nested closures, and while I did fight the borrow checker trying to solve this issue, my motivation to keep using Rust quickly evaporated.

So I went to bed, and while being unsatisfied with the outcome searched for other ways for HTML templating. This time in Elixir.

Trying it again, but with Elixir

I’ve learned about Elixir last year because of its usage in the Akkoma fediverse software, and while being interested in it I didn’t have enough of a reason to try it out. Until December when Advent of Code (AoC) came along.¹

I tried solving the problems with it until Day 20, when I finally lost motivation because I didn’t vibe with AoC’s approach of making you recognize patterns in the input file instead of asking for a purely generic solution… Since then I constantly had the urge to maybe get a bit deeper into the language, but was always blocked by my main problem of not having anything I want to work on in my free time.

Alright.

I’d still like to make my own static site generator.

I’m pretty frustrated with both Gatsby and also my failed attempt at using Rust.

So maybe I can find something cool for templating in Elixir? At least I wouldn’t have to fight the borrow checker.

The options for templating in the Elixir ecosystem (or I guess in general) boil down to two kinds:

1. Your template is only partially connected to the code around it and can only use data passed to it in a special struct
2. Your template is fully integrated with the code around it and has access to all variables and functions of the surrounding scope

The first option seems to be the most common one in all templating engines I’ve looked it. Elixir has it’s own string templating with EEx (Embedded Elixir), which can be stored in a separate “.eex” file. The Phoenix web framework (a popular Elixir project) provides an extension for it called HEEx (HTML+EEx), but regardless of how you use it the strings or files can’t integrate code directly and you have to pass all data to them. In HEEx specifically this is done with an assigns parameter that must contain all the data you want to display or perform conditional logic on. If you want to integrate existing templating engines like Mustache it works pretty much the same way.

And even my initially biggest hope – Temple – unfortunately also works the same way (due to it wanting to be compatible with HEEx). You may be able to write code that looks similar to what I liked about Maud, but it still requires everything to be put in an assigns variable. Here’s a short example:

def render() do
  title = "My Site"
  assigns = %{title: title}

  temple do
    "<!DOCTYPE html>"

    html do
      head do
        meta charset: "utf-8"
        title do: @title
      end
    end
  end
end

You can’t pass the title variable directly. You have to make it a key of the assigns map first.

The section option is what Maud does and what I was also used to from JSX – being able to integrate the surrounding scope directly with the template. This is what I liked about the most about Gatsby (or rather React) and if I couldn’t do it I might as well put all templates in separate files. Unfortunately the templating options for this are much rarer.

Being frustrated again I looked at the one project I had immediately brushed off because it hadn’t been updated since early 2018 – Taggart. It might be unmaintained, but it looked like it would offer the same things I liked about Maud – and apparently even more.

One problem with Maud was that it used just one Macro that you write your HTML in, which lead to requiring special syntax at a lot of places (Rust code wrapped in () and control structures prefixed with @). Instead Taggart has one macro for every, single, HTML tag. This allows mixing it (almost) seamlessly with Elixir code, because each macro returns the HTML it represents as a string.² And the inner body is just plain Elixir code.

So if you write for example:

def render(content) do
  html do
    head do
      meta charset: "utf-8"
      title "omg does this actually do everything I want"
    end
    body do
      content
    end
  end
end

It would insert content as is, because the result of the body macro is just a string containing <body>, the passed content and </body>. Likewise this string gets used as the text content for the html macro (along with the result of other macros and Elixir variables). So this function returns a string of all the HTML!³

And the best part is there is (almost) no need for special syntax because everything is Elixir code!

Write:

div class: "my-cool-class" do
  if some_boolean do
    "optional content"
  else "" end
end

And since if in Elixir returns its content last statement it’s really easy to render an empty <div> if some_boolean evaluates to false!

Having made sure everything worked I got really excited and proceeded to fork the unmaintained repository to update its dependencies (and fix a couple smaller issues I encountered later on).⁴

And then started to work on other things like an automatically started web server and file watcher to give me some of the comfort of gatsby develop back.

But more on this in the next blog post!