Since this article was written I’ve migrated to Hugo. The pipeline and deployment processes still apply, but I’m no longer using Gatsby.

I decided recently that it was about time I started a blog, if for nothing else other than document my own experiments so that when I forget what I did and how I did it I will at least have some semblence of a record.

This post is all about how to get started with Gatsby, host using Netlify and have a build and deployment pipeline using Azure DevOps. I use Azure DevOps extensively in my day job, but for my own purposes I want to try things out when there isn’t an urgent need to develop a solution immediately.

Development Environment

I am one of those annoying people that can’t decide between Windows and Mac, so I use both. I will try to highlight any platform differences throughout the post, but on the whole the experience is pretty much the same regardless of operating system.

Getting started

To begin with, you need to install the following:

Now that we’ve got the relevant packages and tools installed, we need to get going with Gatsby. I used a starter to site to get going. You can view and choose various starter sites online. I chose the awesome ‘Hello Starter’ kit by Panr.

Install Gatsby

Installing Gatsby is very straightforward. The following command installs the gatsby-cli as a global tool (using the -g flag):

npm install -g gatsby-cli

Now create a directory to hold your source code:

cd {base_dir}
mkdir web

For example, if my base directory is ~/dev/projects/my-site then the command would be:

cd ~/dev/projects/my-site
mkdir web

Tip: For Windows users, replace ~/dev/ with c:\Dev or equivalent

Assuming you’re using the ‘Hello’ starter kit, run the following:

gatsby new {directory} https://github.com/panr/gatsby-starter-hello-friend

For example, I want my directory structure to be:

- root/
   - pipeline/ (build pipeline config)
   - web/ (source code)
   - readme.md (readme for the repo)
   - .gitignore (exclude files/directories from my repo)
   - azure-pipelines.yml (Azure DevOps pipeline definition)
   - GitVersion.yml (more on this later)

As I want all the source code (i.e. the Gatsby site) to live in the web directory, I run:

gatsby new web https://github.com/panr/gatsby-starter-hello-friend

This will clone the repo into the web directory and the starter kit is ready to go.

Install site dependencies

Now we’re ready to install all the dependencies that come with the starter kit. From the web directory, run the following command:

yarn install

This will download all relevant dependencies.

Run the site locally

Now we’re ready to run the site locally. Once the dependencies have been installed, run the following from the web directory:

yarn run dev

This should complete successfully and your site will now be running locally at http://localhost:8000.

Tip: If you run into any problems at this stage, make sure you’re using the LTS version of node - I experienced issues with the latest node version. You can check the latest version with node --version.

At this point, your site should be running locally and we’re ready to start thinking about the build pipeline.

I’ll split this into several stages - the first thing we’ll look at is setting up a new pipeline, then we’ll discuss setting up a Netlify account, then we’ll add a new deployment stage to the pipeline to get the code deployed.

Azure DevOps

It seems to be a rare thing to not be a GitHub user these days, especially since free private repositories are now available.

However, I’m an Azure guy at heart and I use Azure DevOps extensively in my day job - I love the way it works, and I’m really into the fact that I can declare my pipeline in code and keep it with the source code. Of course this is possible in other platforms, but I like how it works in Azure DevOps. Other tools are available 😉

The pricing page for Azure DevOps provides the most up-to-date information, but you should be able to get a free account for a single private repository with up to 1,800 free build agent minutes per month using hosted build agents. You can add your own machine as a build agent if this isn’t enough, or if it isn’t suitable. I’ll cover this in a later post.

Once you’ve signed up and created your organisation, create a new repository to store your code.

Note: Your URL will be https://dev.azure.com/{username}/{repository_name}

Initialize git repo

Now that you’ve created an empty repository, it’s time to initialise the git repo locally and get your changes pushed in.

From the root directory of your project, e.g. ~/dev/my-project:

git init

This will create a new empty git repository. Before committing any of your changes to the repo, add a .gitignore file so that you don’t commit unnecessary files to the repository. Again from the root directory:

touch .gitignore

This will create a new .gitignore file.

If you’re on Windows, you can use PowerShell:

New-Item -Path .gitignore -ItemType File

Now we need to add some entries to the file to prevent us from checking in things like the node_modules directory:

node_modules

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# Typescript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# dotenv environment variables file
.env

# gatsby files
.cache/
public

# Mac files
.DS_Store

# Yarn
yarn-error.log
.pnp/
.pnp.js
# Yarn Integrity file
.yarn-integrity

Adding these entries will prevent items from being checked in.

Now make your initial commit to the repository:

git add .
git commit -m "Initial Commit"

These two lines will add all files to the repository (except those excluded by .gitignore) and will then make an initial commit with the message “Initial Commit” (I’m not Mr. Imagination).

Add Azure DevOps git remote

Now we’re going to add the remote git repository details so that push and pull operations interact with our Azure DevOps account.

Note: You’ll need a valid SSH key to run these commands. Details of this can be found here: https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops.

git remote add origin [email protected]:v3/{organisation}/{project}

Note: To confirm your remote URL, check your repo in Azure DevOps and click on the ‘Clone’ link in the top right-hand corner. Switch to the ssh tab and copy the URL.

Now you can push to the remote:

git push -u origin --all

If you now check in the Azure DevOps portal you should be able to see your files.

Add pipeline details

Now we’re going to start defining the build pipeline.

Note: I have my own “funny” ways of doing things (as my friends and colleagues would happily tell you) so don’t assume I’m telling you the gospel method of defining pipelines. This is how I like to do it, because I think it keeps things nicely separated.

Create a new directory in the root of your project called pipeline. This is going to hold individual .yaml files that make up the pipeline:

mkdir pipeline

In the root of the project, create a new azure-pipelines.yml file:

touch azure-pipelines.yml

Or, for Windows/PowerShell:

New-Item azure-pipelines.yml -ItemType File

Now we’re going to switch to VS Code and start building the pipeline. In the first stage of the pipeline we’re only going to produce the deployable artifacts; we’ll get to deploying to Netlify later.

Start with the following:

name: 'Site'
trigger:
  branches:
    include:
      - master
      - feature/*
    exclude:
      - develop/*

Here we’re specifying the name of the pipeline as it will appear in the Azure DevOps portal (‘Site’).

Then we are defining the triggers for the pipeline. The configuration I have is that only the master branch and any feature/* e.g. feature/my-post branches will trigger the pipeline. I like to develop locally whilst still pushing to the remote repo (so I can easily switch devices and continue elsewhere). However, when I’m in the middle of something like writing a new post, I don’t want the pipeline to be triggered. Any incomplete work I will do on a branch prefixed with develop/. It just helps me to keep organised.

Next, we’re going to add some variables. These allow us to define a value once and then re-use it elsewhere in the pipeline. There are other, more advanced uses for variables, but in this example I’m simply using them to keep the pipeline organised and simpler to update:

variables:
  - name: node-version-spec
    value: '11.x'
  - name: yarn-version-spec
    value: '1.x'
  - name: vm-image
    value: 'ubuntu-latest'

You’ll see these values in use shortly.

Next, we going to start building the first stage of our pipeline.

First pipeline stage

As I experiment more and more with Azure DevOps / Azure Pipelines, I find myself preferring the multi-stage pipelines to the previous incarnation of build pipelines and releases. I’m expecting new functionality to appear in the future, but for now the multi-stage pipelines provide exactly what we need.

The first stage will build our static site, and the second stage will deploy it. Read more about multi-stage pipelines at https://docs.microsoft.com/en-us/azure/devops/pipelines/process/stages?view=azure-devops&tabs=yaml.

The first stage is defined as follows:

stages:
- stage: Build
  jobs:
  - job: Prepare
    pool:
      vmImage: $(vm-image)
    steps:
    - template: pipeline/git-version.yml
    - template: pipeline/node.yml
    - template: pipeline/yarn.yml
    - template: pipeline/copy-files.yml
    - template: pipeline/publish.yml

That’s the entire stage defined, but note I am using templates so that I can keep the azure-pipelines.yml file clean. Each of the - template: pipeline/{template}.yml files contains the real details, and we’ll step through those now.

git-version.yml

I have my repository set up to be tagged with the build version when the build succeeds. This helps me to correlate a specific branch with a specific build. However, this is something I would normally use when I’m working in a team and multiple people are working in the same repository, or when I need an audit trail for what was deployed when. As this is just a demo for a blog, I’m going to skip over that stage for now. I may blog about this in the future.

node.yml

The node.yml file has the following contents:

steps:
- task: NodeTool@0
  displayName: 'Installing Node.js $(node-version-spec)'
  inputs:
    versionSpec: $(node-version-spec)

This command is part of the default set of tasks available in Azure DevOps and will ensure that node js is available on the build agent. The $(node-version-spec) syntax references the variable that we defined at the start of the build definition.

yarn.yml

The yarn.yml step involves a custom task that I installed into my Azure DevOps organisation. You can find it here: https://marketplace.visualstudio.com/items?itemName=geeklearningio.gl-vsts-tasks-yarn. Add it to your organisation, and then enter the following:

steps:
- task: YarnInstaller@3
  displayName: 'Install Yarn $(yarn-version-spec)'
  inputs:
    versionSpec: $(yarn-version-spec)
    checkLatest: true

- task: Yarn@3
  displayName: 'Install dependencies'
  inputs:
    projectDirectory: 'web'
    arguments: 'install'
    productionMode: false

- task: Yarn@3
  displayName: 'Build Site'
  inputs:
    projectDirectory: 'web'
    arguments: 'run build'
    productionMode: false

These steps will install the Yarn tool, install your site’s dependencies using Yarn, then build the site using Yarn. At the end of these stages, you should have a public folder containing the static assets ready for deployment.

You can test these steps locally by simply running the following commands from the web directory of your project:

yarn install
yarn run build

copy-files.yml

Now that we’ve ‘built’ the site, we want to ensure that the contents of the public directory are copied into the artifacts staging directory, so that we can publish them in the next stage.

The contents of copy-files.yml is as follows:

steps:
- task: CopyFiles@2
  displayName: 'Copy Static Files'
  inputs:
    sourceFolder: '$(System.DefaultWorkingDirectory)'
    contents: |
      web/public/**/*
    targetFolder: '$(Build.ArtifactStagingDirectory)'
    overwrite: true

This task will copy all files from the web/public directory (the globbing pattern **/* basically means all files from all sub-directories) to the artifact staging directory. Anything in the staging directory will be published in the following step.

publish.yml

The publish.yml file contains the steps to publish the contents of the web/public folder so that they are available to subsequent stages.

steps:
- task: PublishBuildArtifacts@1
  displayName: Publish
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)'
    artifactName: 'drop'

This config will result in a compressed archive drop being published. The pathtoPublish (casing is correct according to the Azure DevOps task - I’d have preferred pathToPublish but I need to pick my battles…).

Stage One Summary

I appreciate that we’ve covered a lot up to this point, so let’s summarise what we’ve done so far:

  1. Created a new git repo in Azure DevOps
  2. Pushed our default/starter site up to the remote repo
  3. Defined the following pipeline elements:
    • Pipeline named Site
    • Included only master and feature/* branches
    • Defined some re-usable variables
    • Installed node on the build agent
    • Installed Yarn on the build agent
    • Run yarn install to restore dependencies
    • Run yarn run build to build the Gatsby site
    • Copied the /web/public folder contents to the artifact staging directory
    • Published the artifact staging directory as a build artifact called drop

Now we’re ready to get this site deployed. To do that, you’re going to need a Netlify account.

We’ll cover that process in part 2 (still to come!).