TFS 2012 Controller Hangs on Startup

I work in an environment with multiple TFS collections and was recently working on a project to upgrade the build controllers for each collection from TFS 2010 to TFS 2012.

In general, since the TFS controller installs can work side by side, my process has been

-          Update the OS from Server 2008 R2 to Server 2008 R2 SP1

-          Install VS 2012

-          Install TFS 2012

-          Migrate the collection

-          Configure the TFS 2012 Controller and agents to point at the migrated collection

And this has been flawless over 6 collections to date. 

As the movie announcer says, “… until now.”

I’d done the first three steps weeks ago in preparation for the dev team to take a smallish outage to migrate the database.   On the morning of the migration, we did our due diligence, backed up the database, and upgraded the collection and attached it to our TFS 2012 server. 

Then, as with the preceding build controllers, I configured the TFS controller to point at the 2012 version of the collection.  Everything looked normal until I submitted a build. 

The build just hung. 

When I checked the controller, the controller and the agents were stopped.  Hm.  Ok, click restart.  Nothing happened.  Ok, restart the Build Service altogether.  Controller and agents would come up, and then about 5-10 seconds later stop again.  The build service would stay started, so build requests would still go through from Visual Studio, but they would hang and not show  what was wrong.

I did a bit of googling, and this article seemed to discuss similar symptoms, even down to the "Details..." error message.  That  “HTTP code 500: System.ServiceModel.ServiceActivationException” error threw us for a loop for a while, until we realized that on a different working controller, we were seeing the same errors in the event logs

So that wasn’t it.  Also, there was only one entry for http in the bindings.

Most confounding was the issue mentioned in a number of articles that featured similar symptoms.  They all poined at Program Files\Microsoft Team Foundation Server 11.0\Applciation Tier and either Web Services or Message queue, suggesting that there was a binding problem in the web.config file.

Except I didn’t have a web.config file in either location.  All I had was a web.config.template file.  It had some bindings in it, but I didn’t want to mess with them.

So at this point, I gave up. 

-          Uninstall TFS 2012

-          Uninstall TFS 2010

-          Install TFS 2012

Then my build worked first time. I’ve described this approach as the three finger salute, followed by a single finger salute.