Hosting an Azure DevOps build agent on Windows using Amazon EC2
Azure Pipelines is Microsoft’s solution for running automatic builds and tests in Azure DevOps. By default, jobs are run using a Microsoft-hosted agent, which spins up a new virtual machine (VM) or container each time a pipeline is executed. However, there is also the option to run jobs in a “self-hosted agent”, which you might consider in these situations:
- You need to install custom software on the VM.
- You want to speed up your pipeline by relying on machine-level caching.
In this article, I describe my experience setting up a self-hosted agent on a virtual machine in the cloud. In my case, I created a Windows Server machine using Amazon Web Services (AWS).
Creating a virtual machine
I decided to deploy my VM on an Amazon EC2 instance. I launched a “free tier eligible” t2.micro instance using the Amazon Machine Image (AMI) named “Microsoft Windows Server 2019 Base.” Warning: Using a free tier eligible AMI does not mean you won’t incur any costs as you still need to pay for e.g. the EBS volume that makes up the hard disk.
I created a new keypair and downloaded it to my local machine. To install custom software on my VM, I needed to connect to it using a Remote Desktop Protocol (RDP) client. For this, I turned to the following AWS article:
Note: The article recommends verifying the identity of the computer by comparing the thumbprint of the certificate, but I could not find the suggested “Get System Log” action in AWS until I switched back to the “old” EC2 console.
I set up the security group for my EC2 instance to allow all outbound traffic so that the agent is able to clone the source code, download and install packages, etc. To my delight, I got away with denying all inbound traffic — it seems the agent uses HTTP long polling to listen for jobs in Azure Pipelines.
Creating an agent pool in Azure DevOps
I added a new agent pool for my build agent on Azure DevOps. From the dashboard, I chose “Organization Settings” and selected “Agent pools” under the “Pipelines” header.
Clicking the “Add pool” button to the right brings up a new “Add agent pool” pane.
Under “Pool type”, I chose the “Self-hosted” option. I named my agent pool and clicked the “Create” button. The new pool now appeared on the list.
Provisioning the agent pool for my project
Just because an agent pool exists in an organization doesn’t mean it’s available to every project. After creating the agent pool in my Organization Settings, I had to go to also go to the same place under my Project Settings and add the agent pool there (there is an option to use an existing pool).
Deploying the Azure Pipelines agent
To make my new virtual machine a build agent, I needed to install the Azure Pipelines agent. I downloaded the agent software from the New Agent process in DevOps as per this Microsoft article.
I copied the zip file onto my build machine, extracted its contents into a new Agent folder and ran the config.cmd file.
The script prompted me for my server URL. The URL is the same address you use when you navigate to the DevOps instance in your browser, e.g.
https://dev.azure.com/username/When prompted to choose an authentication type, I pressed Enter to default to using a Personal Access Token (PAT). I created my PAT in DevOps as (per this Microsoft article) and entered this token when prompted by the script. I entered the name of the agent pool I created earlier. After choosing a name for my agent, I was prompted for my work folder, which I once again let default (to _work). When asked if I wanted to run the agent as a service, I entered Y, and finally accepted the default User account NT AUTHORITY\NETWORK SERVICE.
Creating the pipeline
In order to run builds in the agent pool that I just created, I had to modify the pipeline YAML. The syntax for using a self-hosted agent pool differs somewhat from the syntax for using a Microsoft-hosted one. The YAML for a Microsoft-hosted agent looks something like this, and specifies which virtual machine image to use:
pool:
vmImage: 'ubuntu-latest'For my self-hosted agent, I changed the YAML to instead specify that it should use my named agent pool, like this:
pool: 'MyAgentPool'Here’s a useful post from StackOverflow about running scripts with working directories other than the root in Azure Pipelines.
That’s it! I am all set to kick off builds on my new build agent.
