Edward Thomson

Pull Requests in Visual Studio Online are great way to perform code reviews. But if you like reviewing code in your favorite editor, instead of on the web, they can be frustrating. Fortunately, you can set up git to fetch pull requests from Visual Studio Online.

Visual Studio Online creates branches for pull requests in the refs/pull/* namespace, which is not fetched in a default configuration. To set up your git client to fetch them, simply run:

git config --add remote.origin.fetch +refs/pull/*/merge:refs/remotes/origin/pr/*

Now on your next git fetch, you will download the pull requests, and you can simply check out the one you're interested in:

git checkout pr/20804

Another libgit2 Security Update

January 20, 2015  •  7:41 PM

On the heels of CVE 2014-9390, we are announcing another round of security updates to libgit2. Similar to the prior vulnerability, an attacker can construct a git commit that, when checked out, may cause files to be written to your .git directory which may lead to arbitrary code execution.

When attempting to write into a directory, we will follow symbolic links in the working directory, instead of removing the link and re-creating a directory in its place. On a case insensitive filesystem, this allows an attacker to produce a commit that creates a symbolic link to the .git directory, then creates a file in a folder with a name that differs only in case. The previously written symbolic link would then be followed, and the file would be written in the .git directory.

This vulnerability primarily affects Mac OS, as its filesystem, HFS+, is case insensitive by default and supports symbolic links. Git core is not affected by this vulnerability, nor are clients built on top of the git command-line interface.

Updated versions of libgit2 are being made available immediately, as are versions of LibGit2Sharp and Objective Git. We recommend that libgit2 users upgrade.

In addition, GitHub for Mac was updated yesterday to include a fix for this issue.

A big thanks goes out to Jeff Hostetler, who found this vulnerability while researching additional areas where we could write into .git. Jeff is a new member of the Microsoft Visual Studio team who comes to us with an enviable resume building version control systems and developer tools in general.

Thanks also to GitHub and Microsoft for their continued support of libgit2. I am particularly pleased that Microsoft is willing to invest in finding and fixing bugs that only affect other platforms like Mac OS.

92 Days to Fix a Bug?

January 12, 2015  •  11:20 AM

Microsoft (disclaimer: my employer) is apparently complaining about Google publishing the details of a security vulnerability in keeping with Google's stated policies on the matter, but two days before Microsoft would have been able to issue a general fix on Patch Tuesday.

In some of the discussion on Hacker News, somebody suggested that Microsoft had not been diligent about fixing this bug in a timely manner:

it seems unlikely it realistically took MSFT 92 days to put together a patch. It seems far more likely they sat on the disclosure and did not prioritize it appropriately, after google continued to push for an estimated release date (which they never got).

I wanted to repost my response to that here, (hopefully) without weighing in to whether Google or Microsoft is right or wrong here.

I very much doubt that this was simply "sat on". Not knowing the full breadth of this vulnerability, I can only speculate, but I'm making an educated guess based on several years of working at Microsoft and recently managing a fix for a security vulnerability in several versions of two products.

It can sound crazy to you (certainly, it sounded crazy to me at first), but taking 90 days to turn around a fix to a piece of software like Windows is not implausible.

First you have to understand the vulnerability: under what circumstances does it occur? Can we build some reliable tests so that we are convinced that we will know when we have fixed it?

Looking at the report of the vulnerability, are there other instances of problems that will lead to similar bugs? Because if you release a patch and then a week later somebody realizes a similar vulnerability, you're going to have to do all of this over again, and not with the luxury of 90 days before the announcement.

Then you have to identify what versions of the software are affected. This is where you start sweating, because were you working on the product seven years ago? If you were, do you remember anything about it? Well, you're going to have to learn fast and hope that the architecture hasn't changed too much.

Then you fix the bug, probably only in one of the affected versions at first, which is probably the version of the software that you have on your dev box. As you're fixing it, poke around in the code to think about similar vulnerabilities that the report didn't find. Recall that Windows is not a small piece of software and building it on your dev box and being able to test your fix may take several hours.

Now you port that bug fix over to all the other versions of this software that are affected and that you still support. For a piece of software like Windows, this is a long list. Hope that the architecture and the code you're fixing hasn't changed much or else you're digging in to remember how Windows 2008 worked and how to fix this problem there. Recall again that building each of these versions takes time.

Now you hand off your fixes to some other people who will independently verify your fixes on some of the supported versions. I say some, not all, because there's going to be another round of testing on the actual deliverables, which is the patch to the operating system.

Which another group is going to make. Most products at Microsoft have a build lab that will take a branch and create either the actual installer disk image or a patch to a previous version. In this case, they're going to create a patch to the latest supported version, for each supported version.

Fortunately, this can probably be done in parallel with the first round of testing if you're pretty sure that you nailed it. If you think that the testing (above) is likely to reveal some problem, then you should hold off handing it over to the build lab because these folks are some of the least appreciated parts of the development team. They're the ones who integrate all the various development teams feature branches into master, resolve the easy conflicts and find the people who need to resolve the hard ones. They have a full time job (and not a trivial one) before you're bringing your high priority build to them, and when you ask them to dust off the build machines for a seven year old version of the product, they're going to graciously accept. But to tell them you didn't get it right and request they start over on a new version is when you start bringing six packs with your request.

Once the patch is created, it goes through the real testing. Because somebody's going to install this patch on all the supported versions, and the SKUs within those versions, to make sure that it works. When I say "it works", I don't just mean that the patch fixes the bug in question (though of course it has to do that), I mean that it also has to not regress any functionality. And that the patch is able to be installed and uninstalled cleanly. This is some annoying work and the longer this bug has been around, the more annoying it is. Remember "Windows Essentials Business Server 2008"? Me neither, but somebody's going to be installing it on a VM and ensuring that your patch works there.

Let's assume that everything has gone well up to this point and you're ready to release it. Most product updates want to get included in Windows Update, of course, because you want your security fixes to just show up to the customer without them having to learn about them, download it and install it because so few people do.

If you're going to miss a patch tuesday, then you need to start asking yourself whether you want to a) try to buy yourself some more time, b) put up several KBs with the patch and ask people to install them manually or c) wait it out until the next patch tuesday. What you decide will probably be some combination of those depending on the severity of the bug. There's a possible fourth option, which is to convince somebody that your patch needs to go out before patch tuesday and while I'm sure that happens, I'm not sure how it happens. I suspect when you're dealing with a bug so critical as to warrant that level of pain for the organization, it will happen.

All of these steps take time. And a big organization like Microsoft moves slowly sometimes - it takes time to find the right people for all of these steps. This is especially difficult over the holidays where many of the "right people" here are out of the office.

And of course there's inevitably the clever person who says "hey guys? I just figured out another way to trigger this bug." And then you start over from the beginning.

Now you could argue that it's ridiculous that it took Microsoft 92 days to get a fix out the door, and maybe you'd be right. But that's another thing entirely. What's clear to me that Microsoft did not simply wait until day 89 to start working on this.

(In relating this story, I would be remiss if I didn't thank Junio Hamano of Google, the maintainer of git, who was kind enough to provide Microsoft with additional time to research and prepare patches for the range of our products that were affected by CVE 2014-9390.)

The Visual Studio ASCII Art Logo in TFS

September 10, 2014  •  10:42 AM

I noticed last night that Jared Parsons had cheerfully tweeted about the Visual Studio ASCII art logo that shows up when you clone a repository from TFS and Visual Studio Online:

In case you haven't seen this, it looks something like:

Several years ago, when we started planning our Git support in Visual Studio and Team Foundation Server, I spiked a Git server implementation that used TFVC object storage on the backend.1 This was a fun way for us to learn the details of the Git smart protocol that's used when you fetch or push to a server and one of the things you quickly notice about the smart protocol is that it multiplexes a few channels through a single TCP connection. There's the general data channel, which is where the objects you fetch get sent, but there's also a channel for normal priority messages (akin to stdout) and another for error messages (akin to stderr) so that the server can provide some feedback to the user.

ou may have noticed this at work - the git server will often send you little messages to tell you that it hasn't forgotten about you while it's busy preparing your request:

remote: Counting objects: 52789, done.
remote: Compressing objects: 100% (15421/15421), done.

As we were getting towards the end of this spike, management wanted to see what we'd been doing and how realistically we could cram Git into TFS. As I'm nervously preparing for this meeting, I suddenly want to throw a little bit of levity in to the meeting. So I quickly whipped up an ASCII art version of the Visual Studio logo and wired up our handler to send it out over the normal priority message channel. Now when we went to clone the demo repository from the server, you'd see the VS logo in the command prompt.

Thankfully, this got a pretty good laugh, and it convinced everybody that maybe we really could put Git into TFS in a very natural way.

I moved on to some other work and didn't think much more about this until my very talented coworker Philip Kelley was showing the actual, ready-to-ship implementation of our Git server in TFS. Once again, when you cloned a repository from the server you were greeted with the Visual Studio logo. As we got closer and closer to shipping, we kept waiting to hear from whatever department enforces our no easter egg policy.2 Or our marketing department, telling us that they didn't like our ASCII art bastardization of their logo.

And, of course, we did hear from them. They asked us to remove it… but only because we were using the old, rounded infinity logo from 2012 and prior, instead of the very angular new-style logo for VS 2013. So Martin Woodward hand-crafted an updated ASCII art logo (made up of the letters 'T', 'F' and 'S' no less!), checked that in, and it remains to this day.

In case you're curious, the original ASCII art logo was:

       ?????        $$$$$$$$ZZZ$$$
    ~?++==++?I    7$$$$$$$$$$Z???Z$
   ~?+==~~~=+I$Z 777$$$$$$$  +OOOOO$
   ??+=        $77777$$$I     $ZZZZZ
   7I?        II777777?       77$$$$
    ~77      +IIIII777         777777
     II~  =IIIIIIIIIZZ        III777
         7IIIII        $777777777     (TM)

Microsoft (R) Visual Studio (R) Team Foundation Server

This silly little bit of ASCII gives me a smile every time I see it; we hope it gives you one, too.

  1. Today we use an implementation where Git object storage goes to the database instead of through TFVC, a much saner approach. 

  2. Seriously, I signed something that said that putting an easter egg into my software was grounds for immediate dismissal. We rationalized this by saying that it shows up for everybody all the time, so it's clearly not an easter egg. 

Git Internals at That Conference 2014

August 15, 2014  •  2:29 PM

I had the privilege of speaking at That Conference earlier this week on my favorite topic: version control! I discussed how Git works under the covers, and my audience was kind enough to stay awake (which is the fundamental difference between Git and other version control systems).

A big thanks to everyone who attended and especially to the tireless volunteer organizers of That Conference: it was an amazing time.

If you attended the conference and want to see the slides, they're right here:

(Okay, they're still there even if you didn't attend. But only if you make sure to attend next year!)