Managing Dotfiles with Git

Professional cooks have a term: "mise en place", which translates literally to "everything in its place". In a kitchen, it means that your station is prepared and well-organized; all the ingredients for every dish that you cook are prepared and set out in front of you so that they're ready to use. But perhaps most importantly: your station is clean, because professional cooks have another phrase:

Messy station equals messy mind. Clean station equals clean mind.

This idea extends far beyond the kitchen and - yes - even into software engineering and IT. Don't believe me? Take a look at your computer... Unless you're excruciatingly well-organized, you've probably got a junk folder somewhere. I call mine "Temp" on the very optimistic idea that I'm just throwing things in there for a little while until I find the right place for them.

But the reality is that most things never leave; they just accumulate.

Just like in a kitchen, all this crud and all these little annoyances take a mental tax and make you less productive. That's why one of my guilty pleasures is to "repave" my machine, reinstalling everything onto a freshly formatted drive with no trace of the old detritus, then carefully reinstalling only the apps that I'm interested in and copying over only the data that's worth keeping.

I try to do this every year or two, both to my main personal machine and my main work machine. And it turns out that I just finished this up, since last week I had a little free time on my hands:

Just logged out of my last day at @GitHub. It's been fun, I've met great people and learned a lot, but I'm really excited about what's next.

— Edward Thomson (@ethomson) March 13, 2017

At a high level, this process is pretty straightforward: I back up my home directory (twice), format the drive, reinstall the operating system, and carefully copy the contents of my home directory back, tidying as I go. On the whole, this is pretty tedious, except for one set of files: my dotfiles.

Years of working with large networks of Unix machines has taught me to version control my dotfiles so that I can get up and running on any new machine quickly. I keep my dotfiles checked in to a git repository, except for the truly important ones - the ones that I need to keep secure, like my SSH keys - which I keep with me.

When I'm setting up a new workstation, I start by copying over my SSH keys since I'll actually need them in order to clone the rest of my dotfiles. (I skip this step if I'm setting up a new account on a remote machine, since I'll just use SSH Agent Forwarding to keep my keys on my local machine.)

I keep my most secure bits on a USB key that I keep with me. Most USB keys have some sort of attachment for a keyring - hence the name. Maybe it's a little lanyard, or a metal ring, or even a plastic tab of some sort.

But just because you can attach these to a keyring doesn't mean that you should attach these to a keyring. That little hook is going to fall apart over time as you pull your keys out of your pocket or they jostle around in your purse, and one day your USB key isn't going to be attached to your keyring anymore. And then you've lost a copy of your SSH private key.

What I use is a USB key that is made from a hunk of aluminum:

This is much, much less likely to fall off my keychain unexpectedly. Even with that in mind, of course, I still turn on full-disk encryption to protect me in case I accidentally leave my keys laying around after a night out at the pub.

This particular key is an encrypted HFS+ partition for Mac OS, but I have a separate key for setting up new Windows machines, which is encrypted with Bitlocker To Go. (But more on that later.)

After copying over my SSH public and private key into my home directory, I run a script that clones my dotfiles repository from my personal Visual Studio Team Services account. (I use VSTS for private repositories and GitHub for all my open source repositories.) I keep this on my USB key so that it's handy:

#!/bin/sh git clone --separate-git-dir=.dotfiles.git ethomson.visualstudio.com:DefaultCollection/personal/_git/dotfiles . rm .git echo '*' > .dotfiles.git/info/exclude

This script is pretty simple, but there are a few odd things going on:

1. It uses the --separate-git-dir option to git clone. It does this so that my git repository in my home directory is called .dotfiles.git, instead of the usual name, .git. This is important so that when I run git somewhere in my home directory it doesn't accidentally do work inside my dotfiles repository.

   In particular, I might be somewhere beneath my home directory - say in ~/src/my_new_project - and run git status. If I haven't yet inited a repository for my new project, then I'll actually see the status for the git repository in my home directory that controls my dotfiles. This is confusing at best; it's best to make this a separate git directory instead.

2. This removes the .git file that git clone creates. When I clone with a separate-git-dir, git will helpfully set up a .git file that points to that separate git dir so that future invocations of git can find it. This would be useful if my goal was to put the git directory on another device but still have it work transparently as if it were a normal git repository.

   That transparency is exactly the behavior I don't want, though. By removing this file, I'll get the isolation that I wanted in step 1, but I'll have to specify the git-dir every time I run git to work on my dotfiles. (More on that below.)

3. This sets up an info/exclude in the dotfiles git repository with a * wildcard. This means that all untracked files in my home directory will be ignored. As a result, I will have to explicitly add new files to my dotfiles repository. Without this, git status would show all the files underneath my home directory which is very noisy and - worse - makes it very easy to accidentally add things like my SSH keys.

   I could also accomplish this with a .gitignore, but then I've got a .gitignore hanging out in my home directory and the whole point of repaving my machine was to clean up the unnecessary cruft in my home directory.

As a consequence of using a separate git directory for my dotfiles, I can't simply run git commands - like git status - in my home directory. If I do, git will tell me that it can't find a repository:

fatal: Not a git repository (or any of the parent directories): .git

Instead, I need to explicitly pass the --git-dir option to git so that it can find my repository. The easiest way to do that is to set up an alias in my shell's startup configuration.

I have this in my .zshrc:

alias dotfiles="git --git-dir=$HOME/.dotfiles.git"

Now instead of running git status, I run dotfiles status:

On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   .zshrc

no changes added to commit (use "git add" and/or "git commit -a")

And it's straightforward to dotfiles add .zshrc, dotfiles commit and finally dotfiles push to get my changes back up to my hosted git repository. I simply replace git with dotfiles in any command I want to run when I want to work on my dotfiles repository.

The only thing to remember is that - by virtue of my info/excludes above - I'm ignoring all the files in my home directory that aren't already part of the repository. This means that when I create a new file in my home directory, git ignores its presence:

On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean

This is generally what I want; most of my home directory probably shouldn't be under version control or pushed to every computer that I use. But when I do create a file that I actually want to push to my dotfiles repository, it's easy for me to add:

dotfiles add -f .dotfile

The -f flag is necessary to override the info/excludes file. If I forget it, git will remind me. Once the file is staged, now git will begin tracking it, and dotfiles status will show:

On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   .dotfile

This straightforward setup lets me get my home directory configured on a new machine quickly and easily, and if I make any changes on that machine, I can share them with my home directory on every computer just by pushing and pulling.