Skip to content

Updating a private Git submodule before builds in Netlify

Published on in Git and Netlify

Automatically updating a private Git submodule before builds is convenient but requires some setup.

Table of contents

Skip table of contents

Why a private Git submodule

My website is open source, but the blog posts and other content reside in a separate, private Git repo.

Tania Rascia explains her rationale for her similar setup in her blog post Using Git Submodules for Private Content:

My website has been open source for as long as it has existed. Originally, it was a WordPress site, but only the layout was out there for everyone to see since the data was saved in a database. Once I moved to Gatsby, I kept all the images and posts in a content directory. This was way better, as my content is all conveniently stored in one easy-to-save folder and the posts are all in beautiful markdown.

However, people often like my layout and want to use it, so they clone and deploy this site. Sometimes they will just leave up all the posts and images and update the name and image. Although I subscribe to the Zenhabits Uncopyright philosophy towards content – my content is out there for the world to see and do what they want with it, and it doesn't bother me – I don't think I should make it quite so easy to just clone everything I've written in a moment. If you're going to plagiarize, you should at least have to do a bit of work.

Another benefit: I can store my sloppy drafts and scheduled posts in my private Git repo so that no one can see them (except me of course).

Why automatic updating

From the summary of the aforementioned blog post by Tania Rascia:

My current process for updating the site looks like:

  1. Make changes to content repo.
  2. Commit changes to content and push to private submodule hosted on GitHub: git commit && git push.
  3. Pull new updates into local taniarascia.com repo: git submodule update --remote.
  4. Commit the new submodule changes and push to public GitHub repo: git commit && git push.
  5. Netlify deploys the new site.

That's cool, but I don't want to update my website repo after modifying my content repo. My process looks like this:

  1. Same as above. (Make changes to content repo.)
  2. Same as above. (Commit changes to content and push to private submodule hosted on GitHub: git commit && git push.)
  3. Netlify deploys the new site automatically on the next day because I have set up automatic daily Netlify builds using GitHub Actions. Alternatively I can manually trigger a build on the Netlify dashboard if I want the content changes to go live immediately (in a few minutes).

There are two benefits to this way of working:

  • I need to work with only one Git repo (the content repo) to write content.
  • I can avoid the frequent "update content submodule" commits in my website repo.

My website repo points to a commit from 1st Jan 2021 on the content repo, meaning that I haven't needed an "update content submodule" commit on the website repo in almost two years. This is possible because Netlify automatically updates the private Git submodule (i.e. pulls the latest changes) before it builds the site.

How-to

First, check out the aforementioned blog post by Tania Rascia. It's a good primer on the subject.

When you have a private Git submodule set up, here are pointers how to make Netlify automatically update it before builds. These pointers are based on a support guide on Netlify Support Forums, but I had to trial and error quite a bit before I got it all working.

Deploy key + SSH key

You need to add two keys on your private Git repo's settings page on GitHub:

  • Netlify deploy key: this is needed so that Netlify can initially access the private submodule. Check the blog post by Tania Rascia for a guide.
  • A new SSH public key: this is needed so that Netlify can update the private submodule (i.e. pull the latest changes) before it starts building the site.

Steps:

  1. Generate a new SSH key
    • Example on Mac: ssh-keygen -t ed25519 -C "yoursite.com content repo / Netlify"
    • The -C flag stands for "comment" – it's for you so you later know/remember what this SSH key is for
    • Save the file e.g. to /Users/you/.ssh/yoursite.com-content-netlify
    • Don't set a password or Netlify can't update the submodule
  2. Add the public key (the shorter one) as a read-only deploy key on your private repo's settings page on GitHub
    • Name it e.g. "SSH public key (Netlify)" so you can tell it apart from the other deploy key ("Netlify deploy key")

Note:

  • After changing the build image on Netlify (the previous one I was using was going to be deprecated), the Netlify deploy key changed too, so I had to add the new deploy key to the private Git repo's settings on GitHub. (I'm not 100% sure about this.)

Environment variable

Add the private SSH key (the longer one) to your Netlify site's environment variables. Before that though you need to modify the key because environment variables are single-line (not multi-line).

Your private key looks something like this:

-----BEGIN OPENSSH PRIVATE KEY-----
n4bQgYhMfWWaL+qgxVrQFaO/TxsrC4Is
...
-----END OPENSSH PRIVATE KEY-----

Replace the newlines with underscores (e.g. in VS Code or Notepad) so that it looks like this:

-----BEGIN OPENSSH PRIVATE KEY-----_n4bQgYhMfWWaL+qgxVrQFaO/TxsrC4Is_..._-----END OPENSSH PRIVATE KEY-----_

Note: it doesn't matter whether the private key has a trailing underscore or not.

Bash script

To let Netlify use the SSH key from the environment variables, create a file called e.g. netlify-setup.sh and save it to the website repo's root:

#!/usr/bin/env bash

# Credit goes to fool and hongaar at
# https://community.netlify.com/t/support-guide-using-an-ssh-key-via-environment-variable-during-build/2457

# Check if we're running in a Netlify environment
# See https://www.netlify.com/docs/continuous-deployment/#environment-variables
if [ ! -z "${DEPLOY_PRIME_URL}" ]; then
    # Init .ssh dir and expand $SSH_KEY
    mkdir -p ~/.ssh
    echo -e "${SSH_KEY//_/\\n}" > ~/.ssh/id_rsa
    chmod og-rwx ~/.ssh/id_rsa

    # Uncomment to debug
    # ls -la ~/.ssh
    # cat ~/.ssh/id_rsa

    # Add host keys
    ssh-keyscan -H github.com >> ~/.ssh/known_hosts
fi;

Build command

Last step! Modify your build command (e.g. npm run build) to first run the Bash script, then update the private Git submodule, and then build the site.

There might be several ways to do this. I did it by specifying this as my build command in the netlify.toml file:

[build]
  command = """
    bash ./netlify-setup.sh &&
    GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
      git submodule update --init --remote &&
    npm run build
  """