Edward Thomson

Creating Mac Disk Images (DMG) with VSTS Build Agents

December 15, 2017  •  9:02 PM

I like to pretend that for most of my career, I've only worked on Unix platforms. But that's not exactly true; in fact, I've spent most of my time building cross-platform applications. But I pretend because writing cross-platform applications is hard. And writing cross-platform UI applications is especially hard. So it's exciting to be able to use Electron to be able to easily create cross-platform UI.

I've started working on an Electron app, and even though it's not done yet, and definitely not ready to release, one of the things that I've been thinking about is packaging it for distribution.

Most Electron apps are bundled as ZIP archives, and maybe tarballs on Linux. And that's fine but a really cool user experience is to provide a disk image for macOS. Instead of just a zipped up .app folder, this lets you add some branding, and lets you add a link to the Applications folder so that you can just drag and drop:

Example DMG Background

One of the problems with creating a DMG, though, is that you need a Mac to do it. You can't easily create DMGs on a Linux box or a Windows VM. You need a proper Mac, and historically, it's been hard to find Macs in the cloud for CI hosts.

This was especially problematic if you needed to build any native code. You'd need Windows, Linux and Mac hosts in order to target all your platforms and, until recently, no CI system offered hosted build environments for all the platforms.

But now, Visual Studio Team Services offers hosted Windows, Linux and Mac systems for continuous integration build hosts. And they're free to get started with.

This means that you can use the macOS build hosts to package up your .app into a nicely branded disk image.

Create a background image

One of the great things about using a DMG is that you can create a custom background for the disk image. You can add your logo and provide some simple instructions that people should drag and drop your app into the Applications folder.

Example DMG background

If you're using Photoshop, make sure you set it up as a high DPI image. Here, I want a window size of 500x300, so I'm creating an image that is 1000x600 at 144dpi. If you do this, make sure you use the Save File dialog and select a PNG; the quick export functionality will save as 72dpi.

Create a DMG Generator Repository

Create a Git repository that contains the necessary bits to generate a DMG with all the artwork. You'll need to create a new repository that includes:

  1. The background image that you created for you app.

  2. The create-dmg program written by Andrey Tarantsov.

    You can check this in directly, but I set this up as a submodule; the build step in VSTS can fetch submodules, even if they're hosted in other services. So you can have your proprietary software hosted in VSTS and it can reference Open Source submodules hosted in GitHub.

  3. A shell script to execute create-dmg with all the appropriate arguments. The exact icon placement dimensions will depend on the size of your background and its layout. I named mine build.sh; an example shell script for App.app (using example DMG background, above) might look like:

    #!/bin/sh
       
    create-dmg/create-dmg \
        --volname "App" \
        --background "background.png" \
        --window-pos 200 120 \
        --window-size 500 320 \
        --icon-size 80 \
        --icon "App.app" 125 175 \
        --hide-extension "App.app" \
        --app-drop-link 375 175 \
        "App.dmg" \
        "App.app"
    

    (Remember to set your shell script executable.)

Set up the CI/CD pipeline

It's easy to set up a build pipeline for a hosted Mac build agent. In your Visual Studio Team Services account, navigate to the Build page, and select New Definition.

New Build Definition

VSTS offers helpful build templates to get you started, but we're just running a shell script, so select empty process.

  1. In the "Process" tab, give it a name like Generate DMG, and set the agent queue to Hosted macOS Preview.

    Hosted macOS Preview

  2. In the "Get sources" tab, select where you've hosted the Git repository that contains your build script. Mine is - of course - hosted in a Git repository in Visual Studio Team Services, but the VSTS build process can build repositories that are hosted anywhere.

    Git Repository Hosting Providers

    If you added create-dmg as a submodule, then make sure to show "Advanced settings" and check the "Checkout submodules" option.

    Checkout Submodules

  3. Click the + button on build phase to add a new build task to Phase 1. In the task selection list, scroll or search for the "Command Line" task and click "Add".

    Add a Command Line task

    Select the new task, and give it a display name like "Run Generator". In the "Tool" option, enter "/bin/sh", since we wrote a shell script. Finally, in "Arguments", enter the name of your shell script, "build.sh".

    Generator Configuration

  4. Click the + button on the build phase to add another build task. In the task selection list, scroll or search for the "Publish Build Artifacts" task and click "Add".

    Add a Publish Build Artifacts task

    This task will add the resultant DMG as a build artifact that we can download or deploy.

    In the "Path to publish" option, enter the name of the DMG that your build script generated. In my example, this is "App.dmg".

    Finally, in the "Artifact name" option, enter the name you want to save this artifact as; this is what you'll download or deploy. I recommend using the name of your DMG here.

    Publish Details

And that's all - it's just a few simple steps to create a DMG.

Build Steps

Queue Your Build

Once you've configured your build, you can select "Save & queue" at the top of the page to save your build definition and queue the build to create a DMG.

Save and Queue

When you do, you'll get a notification that your build has been queued. You can select the build number to follow a link to that build.

Build Queued

And once the build completes successfully, you'll be able to navigate to the "Artifacts" tab and download your DMG.

Artifacts

Obviously you could - and should - also deploy this to a download site automatically, creating a deployment pipeline in VSTS.

Example

I created a build script that will download the latest version of Visual Studio Code, unzip it, and then bundle it up as a branded DMG. You can adapt it to your needs.

https://github.com/ethomson/vscode_dmg

When you build this repository in the Visual Studio Team Services build process, you'll end up with a DMG that includes a branded Visual Studio Code experience:

Visual Studio Code DMG

git-open supports Visual Studio Team Services

December 6, 2017  •  6:25 PM

I'm really excited that git-open 2.0 was just released. This newest version of git-open includes some changes that I contributed to support for Visual Studio Team Services and Team Foundation Server.

git-open already had great support for GitHub, so I was already using it for my open source projects. It also supports BitBucket and GitLab. But I wanted to use it for my day job, where all my repositories are hosted in VSTS.

What is git-open?

git-open is a nifty tool that will let you run a command on the command-line, git open, to open up a web browser to your Git repository hosting provider. This means that I can run a single command, right from within my Git repository, and it will open up a browser and navigate to my project in Visual Studio Team Services.

git-open

This is amazing because as much as I try to hang out in the command-line, that's not always useful for large projects. If I'm hacking on my (comparatively) little open source project, libgit2, it's easy enough to use ctags or to grep over the whole codebase. But for a big project that's not so easy.

What I really want is to use the great Roslyn-powered code search functionality in VSTS. So now I can open up a new browser pointing at VSTS without having to click around to get a new browser window open.

It will even look at the current branch that you're on and navigate there in the browser interface. This is perfect if you're starting your branch workflow by creating a new branch from a VSTS work item.

Create Branch from Work Item

When I start my branch workflow this way, I've always got a branch on the server that I fetch to my client. And then I can push my changes back and eventually open a pull request. And git open supports this workflow beautifully, it will look at the branch that I have checked out locally and try to navigate me there in the web browser. It can navigate to branches in Git repositories hosted in several hosting providers, including VSTS and TFS.

But it's not just branches - it can also navigate to the work item hub. This gives you a complete end-to-end branch workflow: create a new branch from a work item in VSTS, pull it down, and then you can use git open --issue to navigate back to the VSTS work items in your browser.

git open --issue

Getting Started

To get started with git-open, all you have to do is download the latest release, unzip it and then put the git-open shell script somewhere in your path.

git has an extension model where all the commands you run - say git pull - are actually separate commands. They're not all just built in to the git command itself. When you run git pull, git goes looking for an executable in your path named git-pull.

So if you want to add your own Git commands, you can just put an executable named git-something into your path. When you run git something, Git will find your git-something executable.

Tip
I have a directory in my home directory where I put useful shell scripts that I make sure is included in my path. On Unix systems this is ~/bin. On Windows machines, this is %HOMEPATH%/bin.

Pro-Tip
I also manage all these scripts - along with my dotfiles - in a Git repository. So when I start working on a new machine, I can just git clone my dotfiles repository.

Google Analytics for Your Podcast

October 27, 2017  •  5:57 PM

I built a little open source side project: Google Analytics Handler, which reports tracking information to Google Analytics on the server side.

It's for my new podcast, All Things Git, so that we can track the RSS and audio downloads from the podcast's website.

If this was 1997 and I was hosting the website on my own server, I could just crunch my Apache logs for the data. But it's not; it's 2017 so of course I'm hosting the podcast in the cloud. The website is an Azure Web App and the audio downloads are hosted in Azure CDN.

And it's nearly perfect! Performance? Amazing! Cost? Low! But log files? Not so much.

Martin suggested a hosted analytics platform to track RSS requests, and another to track audio downloads. But that's two new bits of analytics to go along with Google Analytics, which we use to track the page visits. Requests being tracked in three distinct places? Ugh.

Thankfully, Google Analytics offers the Measurement Protocol API - which lets you report events like page views manually. So I built an ASP handler to report requests on RSS and audio downloads to Google Analytics.

For RSS requests, the handler simply opens the RSS file that exists on disk and returns it to the client before reporting to Google Analytics. This is nice because it's a transparent change - the handler is loaded by the web configuration only in production. It's just a few lines in the Web.config:

<configuration>
  <system.webServer>
    <handlers>
      <add name="RssHandler"
           verb="*"
           path="rss.xml"
           type="GoogleAnalyticsHandler.GoogleAnalyticsHandler, GoogleAnalyticsHandler"
           resourceType="Unspecified" />
    </handlers>
  </system.webServer>
</configuration>

For audio, we don't host the audio directly on the web site, so for the actual podcast itself, the handler redirects to the audio files in Azure CDN. I set it up so that any request in the /episodes/audio folder is redirected straight to the CDN:

<configuration>
  <system.webServer>
    <handlers>
      <add name="AudioHandler"
           verb="*"
           path="/episodes/audio/*"
           type="GoogleAnalyticsHandler.GoogleAnalyticsHandler, GoogleAnalyticsHandler"
           resourceType="Unspecified" />
    </handlers>
  </system.webServer>
  <location path="episodes/audio">
    <appSettings>
      <add key="redirect-root" value="https://mycdn.azureedge.net/episodes/" />
    </appSettings>
  </location>
</configuration>

As soon as I deployed the handler and configuration to production, I was seeing results in the real-time tab of Google Analytics:

Google Analytics

So the Google Analytics Handler makes it very straightforward to add Google Analytics tracking to your media assets like audio and video, and to your non-HTML pages like text and XML.

Changing Titles in Google Authenticator

August 23, 2017  •  5:27 PM

Surely you know that the best practice for securing your accounts is to enable two-factor authentication:

When all that is between you and an attacker getting into your account is a single password, you’re running a risk that is far greater than what you need be taking. A password is one factor – “something you know”. Now if we add something you have such as your mobile phone and the email service verifies your identity when you first log on by sending an SMS to that thing you have, the security position of your email changes fundamentally.

Troy Hunt, 10 email security fundamentals for everyday people

And hopefully you're using an application as your second factor, instead of text messages. Text messages may not work when you travel to foreign countries, but you're also reliant upon your wireless carrier to keep your data secure:

Instead, use TOTP (Time-based One-Time Pad) to get a six digit number from a local application. There are many applications that support TOTP, but I keep it old school, and use the Google Authenticator application.

The problem with the Google Authenticator app, though, is that it doesn't let you edit the title of a website (the "issuer") once you've set it up. So you end up with a number that's missing a title, and there's no good way to identify it.

Here, the first entry is obviously for my Microsoft account, but the second entry…? I have no idea what it's for:

Google Authenticator Missing a Title

Thankfully, TOTP is a published standard, so you can actually create - and then scan - your own QR code based on the secret number that you're given when you turn on two-factor authentication:

Facebook 2FA Enablement

The QR code that you scan to set up a new account is generated by constructing a URL with the secret number and some metadata, and then encoding that with a QR generator. The format is:

otpauth://totp/account_name?secret=secret_key&issuer=Website_Title

The account_name - as the name suggests - reflects the name of your account on the website. This is your username or email address, generally. Google Authenticator shows this as the second line of the key.

The secret_key is the secret key that the web site gives you when you enable TOTP. (In the example above, it's XXXX ABCD XXXX ABCD).

Finally, the issuer is the name of the website itself. This is the larger header displayed above your key.

It's such a simple mechanism that you can just create a new URL with those values and then use your favorite QR generating tool to create a QR code for your custom URL. (Remember to URL-encode any of your values!)

If you don't have a QR generator (I didn't) then you can install the very simple qrencode package and generate a QR code into an image file.

Better still, you can specify ANSI as the output type:

% qrencode -t ANSI otpauth://totp/ethomson@edwardthomson.com?secret=XXXXABCDXXXXABCD&issuer=My%20Title

And it will dump a QR code straight to your console:

QR on the Console

Now you just point Google Authenticator at your terminal window, and you can see that it adds a secret with a custom title of "My Title":

Google Authenticator with a Custom Title

Voila!

Upgrading git for CVE 2017-1000117

August 14, 2017  •  12:11 PM

A security vulnerability in Git has been announced: a bug in URL parsing can cause git clone to execute arbitrary commands. These URLs look quite suspicious, so it's unlikely that you'd be convinced through social engineering to clone them yourself. But they can be hidden in repository submodules.

Unless you're a Continuous Integration build agent, I hope that it's quite uncommon that you git clone --recursive a repository that you do not trust. So this vulnerability is rather uncommon, but as with any security vulnerability that has the possibility of remote code execution, you should upgrade your Git clients immediately.

Git version 2.14.1 is the latest and greatest version of Git, and has been patched. But most people don't actually build from source, so your version of Git is probably provided to you by a distribution. You may have different versions available to you - ones that have had the patches applied by your vendor - so you may not be able to determine if you're vulnerable simply by looking at the version number.

Here's some simple steps to determine whether you're vulnerable and some upgrade instructions if you are.

Are you vulnerable?

You can easily (and safely) check to see if your version of Git is vulnerable to this recent security vulnerable. Run this from a command prompt:

git clone -q ssh://-q/ /tmp/gittest

Note: this will not actually clone any repositories to your system, and it will not execute any dangerous commands.

If you see:

fatal: strange hostname '-q' blocked

Congratulations - you are already running a version of Git that is not vulnerable.

If, instead, you see:

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Then your version of Git is vulnerable and you should upgrade immediately.

Windows

Windows is quite easy to upgrade. Simply grab the newest version of Git for Windows (version 2.14.1) from https://git-for-windows.github.io/.

macOS

Apple ships Git with Xcode but unfortunately, they do not update it regularly, even for security vulnerabilities. As a result, you'll need to upgrade to the version that is included by a 3rd party. Homebrew is the preferred package manager for macOS.

  1. If you have not yet installed Homebrew, you can install it by running:

    /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    

    at a command prompt.

  2. After that, you can use Homebrew to install git:

    brew install git
    
  3. Add the Homebrew install location (/usr/local) to your PATH.

    echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc
    
  4. Close all open Terminal sessions, quit Terminal.app, and re-open it.

Linux (Debian, Ubuntu)

If you're using the current version of Ubuntu or Debian, then they'll have the latest version ready. If you're on a stable system, like a server, you should be running an LTS release - a "long term support" version - where they backport security patches like this one. So you should simply need to:

  1. Get the latest information about the available software versions from the remote repository:

    Debian, Ubuntu:

    sudo apt-get update
    

    Red Hat, CentOS:

    sudo yum update
    
  2. Install the latest version of git:

    Debian, Ubuntu:

    sudo apt-get install git
    

    Red Hat, CentOS:

    sudo yum update git
    

Ensuring that you're patched

Now if you run:

git clone -q ssh://-q/ /tmp/gittest

at a command prompt, then you should see:

fatal: strange hostname '-q' blocked

And now you're patched against the git security vulnerability, CVE 2017-1000117.