Edward Thomson

GitHub Actions Day 26: Self-Hosted Runners

December 26, 2019  •  2:47 PM

This is day 26 of my GitHub Actions Advent Calendar. If you want to see the whole list of tips as they're published, see the index.

Advent calendars usually run through Christmas, but I'm going to keep posting about GitHub Actions through the end of December. Consider it bonus content!

This month I've talked a lot about the software installed on the runners that GitHub provides for running your workflows, and how to install new software.

But what if you wanted to customize your own runner instead of using the ones that GitHub Actions provides? You can use the GitHub Actions self-hosted runner to run workflows on any infrastructure that you have, whether it's an on-premises machine or a runner that you configure in the cloud.

Being able to set up a self-hosted runner is important if you have incredibly custom dependencies – some people still need to use software that has heavy license dependencies like hardware dongles. Or you might want to run a build on an ARM device that you have, instead of the GitHub Actions runners, which are amd64.

More commonly, you might want to talk to machines within your firewall to run tests against them. Or do a deployment step to servers within your firewall.

To set up a self-hosted runner, you first need to download the software to the machine you want to configure. To go the Settings tab in your repository, then select Actions in the left hand menu. There you can configure your self-hosted runners.

Self-Hosted Runners

Just click "Add Runner", and follow the instructions.

Setup

Once you've set up and started the self-hosted runner, it will start polling GitHub to look for workflow runs. You can configure a workflow to run on your self-hosted runner by setting the runs-on to self-hosted. Here I have a simple workflow that will run on my laptop, it just runs uname -a.

When I trigger this workflow, you can see that it accepts the job and then runs in.

Execution

And in GitHub itself, you can see the results of the run.

Results

It's easy to get workflows running on the GitHub Actions runners, or on a self-hosted runner within your network. And the GitHub Actions self-hosted runner will poll GitHub so that you don't need a hole in your firewall to be able to run workflows on machines located inside your firewall.

GitHub Actions Day 25: Sparkle a Christmas Tree

December 25, 2019  •  2:47 PM

This is day 25 of my GitHub Actions Advent Calendar. If you want to see the whole list of tips as they're published, see the index.

Today's Christmas, which means I'll spend it with my family. But I also thought it would be a good time to highlight a neat – and holiday-themed – use of GitHub Actions.

My buddy Martin set up a workflow that will run whenever somebody stars his repository. When that happens, he'll use curl to hit an external API that his internet-connected Christmas tree lights listen for. So whenever somebody stars his repository, his lights light up!

This is another inventive use of actions – listening for an event within the repository, and running a clever workflow.

Merry Christmas, Martin! I hope you get lots of lights on your tree today. And Merry Christmas to you, reader. Thanks for following along this month.

GitHub Actions Day 24: Caching Dependencies

December 24, 2019  •  2:47 PM

This is day 24 of my GitHub Actions Advent Calendar. If you want to see the whole list of tips as they're published, see the index.

Most software projects depend on a set of dependencies that need to be installed as part of the build and test workflows. If you're building a Node application, the first step is usually an npm install to download and install the dependencies. If you're building a .NET application, you'll install NuGet packages. And if you're building a Go application, you'll go get your dependencies.

But this initial step of downloading dependencies is expensive. By caching them, we can reduce this time spent setting up our workflow.

Basically, when the actions/cache action runs for the first time, at the beginning of our workflow, it will look for our dependency cache. Since this is the first run, it won't find it. Our npm install step will run as normal. But after the workflow is completed, the path that we specify will be stored in the cache.

Subsequent workflow runs will download that cache at the beginning of the run, meaning that our npm install step has everything that it needs, and doesn't need to spend time downloading.

The simplest setup is just to specify a cache key and the path to cache.

- uses: actions/cache@v1
  with:
    path: ~/.npm
    key: npm-packages

However, this setup is a little too simplistic, because caches are shared across all the workflows for your repository. That means that if you had a cache for the npm packages in your master branch, and a cache for the npm packages in a maintenance branch, then you'd always have to download the packages that changed between those two branches.

That is to say: when the master branch build runs, it will store the packages that it uses in the cache. When the maintenance branch build runs, it will restore the cache of packages from the master branch build. Then npm install will need to download all the packages that aren't in the master branch but are in the maintenance branch.

Instead, you can tailor the cache to exactly what it's storing. The best way to do this is to use a key that identifies exactly what's being cached. You can take a hash of the file that identifies the dependencies you're installing – in this case, we're using npm, so we'll hash the package-lock.json. This will give us a cache key that is tailored to our packages. We'll actually have multiple caches, one for each branch that changes the package.json, so each branch will restore efficiently.

- uses: actions/cache@v1
  with:
    path: ~/.npm
    key: npm-packages-${{ hashFiles('**/package-lock.json') }}

Okay, this is an improvement. But we still have a problem: since the key depends on the contents of package-lock.json, any time we change the dependencies at all, we invalidate the cache completely.

We can add one more key – in this case, the restore-keys – that can be used as a fuzzier match. It will match the prefixes of the cache keys. In this case, we could set the restore-keys to npm-packages-. If there's an exact match for the key, then that will be the cache that's restored. But on a cache miss, then it will look for the first cache with a key that starts with npm-packages-.

This means that npm install will have to download some dependencies, but probably not all of them. So it's a big improvement over the case when there's a total cache miss.

- uses: actions/cache@v1
  with:
    path: ~/.npm
    key: npm-packages-${{ hashFiles('**/package-lock.json') }}
    restore-keys: npm-packages-

Using the actions/cache action is a good way to reduce the time spent setting up your dependencies, and it works on a wide variety of platforms. So whether you're building a project with Node, .NET, Java, or another technology, it can speed up your build.

GitHub Actions Day 23: Upload Release Builds

December 23, 2019  •  2:47 PM

This is day 23 of my GitHub Actions Advent Calendar. If you want to see the whole list of tips as they're published, see the index.

Here's another nice way that GitHub Actions has simplified the way that I manage my open source projects: when I create a new release, GitHub Actions will run a release build and add the final build archive directly to the release itself as an asset.

Release Assets

This means that I don't have to download the build artifact from my CI build and upload it – or worse, run a build locally and upload it from my machine. Running the final release build within my CI system ensures that I always build it correctly every time.

Here I'll create a workflow that runs whenever I create a release. (If I were to leave out the types: [created] here, this workflow would run on every release activity, including editing or deleting an existing release, which is definitely not what I want.)

I'll do a checkout, set up Java and then run a build similar to what I do in a normal CI workflow. But I'll pass a property to maven (the Java build system that I use) so that my output filename has no suffix. By default, I add a suffix like SNAPSHOT so that people know that this is a prerelease, but for final release builds I want to make sure that there's no suffix.

I'll also ask maven for the filename that it created, so that I can use it in the upload task.

Finally, I'll run my colleague Jason Etco's very helpful upload-to-release action. This will take my build output and upload it to my release as an asset.

Now when I create a release in the GitHub UI, I don't have to worry about uploading any release assets myself. I don't have to remember the build property that I need to change to create the proper filename, and I don't have to upload it from my machine. Instead, a GitHub Actions workflow run will be queued to take care of all of this for me.

Just another way that I leverage GitHub Actions to help me manage my projects.

GitHub Actions Day 22: Automerge Security Updates

December 22, 2019  •  2:47 PM

This is day 22 of my GitHub Actions Advent Calendar. If you want to see the whole list of tips as they're published, see the index.

When GitHub started building GitHub Actions, it wasn't conceived of a just a CI/CD system – it was meant to automate common tasks in your repository. Of course building and releasing are two of the most common tasks, but I love breaking out of the build and release pipeline and thinking about how GitHub Actions can help me manage other parts of my application. For example: security.

GitHub provides automated security alerts for your repositories. When you turn these on, GitHub will periodically scan your repository and examine the dependencies that you use. So if you're building a Node application, GitHub will look at the npm packages that you use and see if any of them have security vulnerabilities.

When it finds a vulnerability, it can open a pull request with the fix - updating that package to a new version that has fixed the problem.

Dependabot

Of course, when this pull request is opened, it will run the pull request validation build that you've configured in your repository. So you'll quickly know that the security update pull request works and that all your tests pass.

But if you have good test coverage… why stop there? Why not automate this entire process and go ahead and merge the security update when your tests pass?

To do this, we can take advantage of github-script. We can use the github-script action to work with the Octokit API and merge the pull request.

In this workflow, we'll run our standard build job that runs our build and test on Node 8, 10 and 12. Then we'll add an automerge job that depends on the build job. If it succeeds, then the automerge job will run.

The automerge job has a conditional - it will ensure that the pull request is targeting the master branch, and that the dependabot[bot] user opened the pull request (ie, the PR was opened as part of a GitHub security update).

Now when a GitHub security update is opened, and the build runs and tests pass, it will be merged directly into the master branch.

Dependabot Automerge

I hope that this gives you inspiration on ways that you can simplify your work by automating manual tasks in your repository – like dealing with security updates – and that you leverage GitHub Actions and github-script to help with that automation.