Consolidating TFVC repositories to Azure DevOps Git

During the past months I have been working on a project for a client which had a main goal, that consisted of consolidating a lot of old TFVC repositories from different old and new TFS and VSTS versions. When I finished the project, the client had only one Azure DevOps environment with only Git repositories. In this article I would like to share my approach.

TFS2008 and TFS2013

The oldest repositories were TFVC in either TFS2008 or TFS2013 and had to be converted to Git repositories. In Azure DevOps you can import TFVC repositories into Git but there are some limitations in size and days of history (changesets). So, I searched the web for some other solutions and came across some older tools and projects that could help me import repositories into Git.

The problem here is that these toolseither didn’t support the older TFS and TFVC versions anymore or simply didn’t work that great due to complex branch architecture in all the repositories. Eventually I used the well-known tool git-tfs: https://github.com/git-tfs/git-tfs#introduction

So, no problem, you would think.Wrong! The support for TFS2008 was dropped in some version of the tool simply because the underlying architecture of TFS had changed a lot, and it was not possible to be backwards compatible anymore, and besides: who still uses TFS2008 these days,right?! Luckily all the older versions are kept as a reference and are downloadable from the GitHub page, so I downloaded the “latest” version which still had the TFS2008 assemblies and support.

As the documentation stated I had to install git-tfs, and I did so by using Chocolatey. This created a folder on my PC in C:\tools\gittfs. The install also added a PATH entry to my computer environment variable so the command “git tfs” could be used from any command or PowerShell prompt.

I created a simple PowerShell script that I executed for every repository, which created a new Git repository in Azure DevOps:

//Create an empty folder to put the source files in from TFVC
cd\
Remove-Item –path c:\temp –recurse -force
mkdir c:\temp

//Do the actual “clone” of the old TFVC repository path
cd c:\temp
git tfs clone “http://teamserverURL:8080” "$/COMPLETE_PATH_TO_SOURCEFOLDER"

//Upload the new Git repository to Azure DevOps
cd c:\temp\SOURCEFOLDERNAME
git remote add origin https://AZURE_DEVOPS_URL/PROJECTNAME/_git/REPONAME
git push -u origin --all

This was one hurdle I took. I converted a lot of repositories from TFVC to Git. But with one thing in mind: I didn’t include all the branches created in TFVC. I only included one direct “path” from the source repository. In this case not a big problem, but something to keep in mind.

I did something similar for the TFS2013, but first I had to use a newer version of the git-tfs tool. So, I changed the path of the previously used older C:\gittfs to C:\gitstfs_OLD because maybe I had to use the old one again if something went wrong along the way. And it did! So, I was happy to be able to switch between the old version of the tool and the newer one.

For TFS2013 repositories I use a similar script but with some small changes:

//Create an empty folder to put the source files in from TFVC
cd\
Remove-Item –path c:\temp –recurse -force
mkdir c:\temp

//Do the actual “clone” of the old TFVC repository of a specific branch
mkdir c:\temp\TFS2013
cd c:\temp\TFS2013
git tfs clone "http://teamserverURL:8080/tfs/COLLECTIONNAME" "$/COMPLETE_PATH_TO_SOURCEFOLDER" .

//Upload the new Git repository to Azure DevOps
git remote add origin https://AZURE_DEVOPS_URL/PROJECTNAME/_git/REPONAME
git push -u origin --all

Existing Azure DevOps Git repositories

Besides the old TFS2008 and TFS2013 repositories the organization was already using Azure DevOps mainly for work item management and some building and releasing of newer software products.

With the existing import functionality within Azure DevOps of importing Git repositories without limitations it was easy to “move” the repositories, with complete history and branches, from different projects to the one big project with all the new and old repositories together, existing side by side. This was a side goal to get all the repositories in one Azure DevOps project to be able to get some trace-ability between code changes and work items.

Combining Git repositories

The last step involved combining some Git repositories into new ones, because the number of repositories was to big to handle, and some code needed to be shared between products. After making a list of repositories which should be combined, which was a very hard task, I created the following script to combine two (or more) repositories into one new one.

//Create an empty folder to create a new repository in
cd\
Remove-Item –path c:\temp –recurse -force
mkdir c:\temp
cd c:\temp
mkdir c:\temp\NEWrepoNAME
cd c:\temp\NEWrepoNAME

//Initialize a new Git repository and do an initial commit because else you cannot //merge some other repository files into it, which will be done in the next step
git init
dir > deleteme.txt
git add .
git commit -m “Initial dummy commit”
git rm .\deleteme.txt
git commit -m “Clean up initial file”

//Clone the first old repository and move to a subfolder in the new repository
git remote add -f repoNAME1 https://AzuerDevOpsURL/PROJECTNAME/git/repoNAME1git merge repoNAME1/master --allow-unrelated-histories
mkdir repoNAME1_Migrateddir –exclude repoNAME1_Migrated | %{git mv $.Name repoNAME1_Migrated}
git commit -m “Move repoNAME1 files into subdir”
git remote remove repoNAME1

//Clone the second old repository and move into subfolder in the new repository
git remote add -f repoNAME2 https://AzuerDevOpsURL/PROJECTNAME/git/repoNAME2git merge repoNAME2/master --allow-unrelated-histories
mkdir repoNAME2_Migrateddir –exclude repoNAME2_Migrated, repoNAME1_Migrated | %{git mv $.Name repoNAME2_Migrated}
git commit -m “Move repoNAME2 files into subdir”
git remote remove repoNAME2

//Optional
Optionally: manually insert a .gitignore file in de root folder
Optionally: Rename migrated subfolders to original folder names

//Do the commit with some sensible comment
git add .
git commit -m “Combined repoNAME1 and repoNAME2 into NEWrepoNAME”

//Add an origin to the newly created combined repository and push changes
git remote add origin https://AzuerDevOpsURL/PROJECTNAME/_git/NEWrepoNAME
git push -u origin --all

Things to keep in mind here:

  • The new repository NEWrepoNAME needs to be created first, and be empty, in Azure DevOps.
  • We move the old repositories into subfolders of the new repository.
  • We do a merge of two (or more) repositories, therefore history viewing is more complex. An extra click is needed to view the history before the rename/move into subfolder action.
  • With the second (and third and fourth) repository you want to merge into the new one, you must exclude all the previous ones when moving to a subfolder.
  • We use different names (repoName2_Migrated) for the subfolders because if in the folder a child folder with the same name exists (in a lot of cases it will!) the moving of files will not move all the right files. The subfolder names can be renamed, after the move, to the original folder names again.
  • Don’t forget to delete the original two or more repositories you combined to prevent users making changes in the “old” repositories.

Feel free to contact me if this article was helpful of if you have questions or remarks.

DevOps Team Topologies

Onderstaand een link naar een artikel wat een aantal anti-type DevOps team structuren binnen organisaties beschrijft. Daarnaast ook een aantal team structuren die goed werken. Welke team structuur past het beste bij uw organisatie?

Thanks to Matthew and Manuel for setting up this nice website!

Source: DevOps Team Topologies

Microsoft DevOps Gold Competency Assessment – Passed!

Today I passed the new Azure DevOps Assessment (18473), happy!

After taking and passing the exams a couple of years ago, the requirements are changed for achieving Gold DevOps Partner status. The current exams are going to retire and are replaced by a new online assessment which I passed today.

More info on: https://partner.microsoft.com/cs-cz/membership/devops-competency#simple-tab-content-2

New Practical DevOps workshop bij Caesar Experts!

Op 26 en 27 september geef ik in samenwerking met Microsoft een tweedaagse Practical DevOps workshop bij Caesar Experts! Deelname is gratis, voor meer info en aanmelden: https://lnkd.in/g6vuQEF

26 September 2017 9:00 – 27 September 2017 17:00

Caesar Experts
Janssoniuslaan 80, 3528 AJ Utrecht

Software engineering practices have changed drastically in the last few years. Agile practices, DevOps, Cloud Computing, Open Source and similar developments have proven their effectiveness. Microsoft’s platforms for software development and cloud computing have also embraced these concepts and are at the leading edge in their domain.

In this two-day training you will learn how software development works with Microsoft’s tools and languages. Based on a consistent sample, you will learn how to

  • setup a development environment with the latest version of Visual Studio and Visual Studio Team Services,
  • highlight and learn to benefit from the latest developments in the Visual Studio C# IDE (e.g. PerfTips, live code analysis, CodeLens, diagnostic tools, (etc.),
  • use cross-platform build- and test-automation tools like Visual Studio Web and Load Tests, Visual Studio Team Services, Gulp, xUnit and Jasmine to automate your build- and quality assurance processes,
  • manage your Technical Debt,
  • create highly available and secure environments for your web APIs and web applications in Microsoft Azure with Azure Resource Manager (ARM),
  • setup a multi-stage (dev/test/prod) environment in Azure,
  • automate deployment to Microsoft Azure with Visual Studio Release Management,
  • using Application Telemetry with Application Insights,
  • discuss the advantages of the upcoming container technology Docker,
  • we will use a sample (web) application during the course.

Training Structure
The training is a combination of approx. 50% presentations with embedded demos and 50% hands-on time where you have your fingers on the keyboard and try what you have learned. Those who are new to the topics can follow a provided hands-on lab guide step by step. In areas you are already familiar with, you can put the guides to the side and create an individual sample with the latest tools and technologies. Attendees should already have basic knowledge about Visual Studio, .NET, C#, HTML and JavaScript. Detailed knowledge about the latest versions of these tools, frameworks and languages is not required.

Prerequisites
You will need your own laptop to do the hands-on labs. Ideally, you have the latest version of Visual Studio installed (detailed instructions about the necessary Visual Studio configuration will be provided) and you have access to your own Microsoft Azure subscription. If you do not have access to Visual Studio and/or a Microsoft Azure account, Microsoft will provide a limited number (first come, first served) of free Azure Passes that you can use to run a VM with Visual Studio in it. In that case you just need internet access and a remote desktop client on your laptop.

For the labs about build- and test-automation, you need your own Visual Studio Team Services subscription (free edition available).

 

Day 1

  • Welcome and introduction.
  • Setup development environment for the course.
  • Discuss the sample application used during the course.
  • Setup Continuous Integration (including first steps in test automation).
  • Managing Technical Debt with SonarQube.

Day 2

  • Adding an artifact repository to your release pipeline.
  • Continuous deployment with Visual Studio Release Management.
  • Configuration as code.
  • Application Telemetry with Application Insights.
  • Operating web applications in Azure App Services and Docker.

Workshop Practical DevOps (Microsoft) at Caesar Experts B.V.

A.s. woensdag (morgen) en donderdag is het weer zover. De volgende workshop Practical DevOps in samenwerking met Microsoft georganiseerd bij Caesar Experts.

Heb je je niet ingeschreven? Jammer, maar houd de aankondigingen in de gaten voor komende sessies, zodat je dit niet hoeft te missen.

Heb je je wel ingeschreven? Dan graag tot morgen!

Parkeren kan alleen op de P+R Papendorp, ingang Mercatorlaan, Utrecht. Vanuit hier is het 9 minuten lopen naar de Janssoniuslaan 80 naar Caesar Experts. Uitrijkaarten voor de P+R zijn te verkrijgen bij de receptie bij Caesar Experts. Mocht je minder mobiel zijn geef dit dat dan door aan mij op f.biesheuvel@caesarexperts.nl dan zorgen we dat je opgehaald wordt of direct onder het pand kan parkeren.