Build GitHub repositories

Azure DevOps Services

Azure Pipelines can automatically build and validate every pull request and commit to your GitHub repository. This article describes how to configure the integration between GitHub and Azure Pipelines.

If you're new to pipelines integration with GitHub, follow the steps in Create your first pipeline. Come back to this article to learn more about configuring and customizing the integration between GitHub and Azure Pipelines.

Organizations and users

GitHub and Azure Pipelines are two independent services that integrate well together. Each of them have their own organization and user management. This section makes a recommendation on how to replicate the organization and users from GitHub to Azure Pipelines.

Organizations

GitHub's structure consists of organizations and user accounts that contain repositories. See GitHub's documentation.

Azure DevOps' structure consists of organizations that contain projects. See Plan your organizational structure.

Azure DevOps can reflect your GitHub structure with:

• A DevOps organization for your GitHub organization or user account
• DevOps Projects for your GitHub repositories

To set up an identical structure in Azure DevOps:

1. Create a DevOps organization named after your GitHub organization or user account. It will have a URL like https://dev.azure.com/your-organization.
2. In the DevOps organization, create projects named after your repositories. They’ll have URLs like https://dev.azure.com/your-organization/your-repository.
3. In the DevOps Project, create pipelines named after the GitHub organization and repository they build, such as your-organization.your-repository. Then, it's clear which repositories they're for.

Following this pattern, your GitHub repositories and Azure DevOps Projects will have matching URL paths. For example:

Users

Your GitHub users don’t automatically get access to Azure Pipelines. Azure Pipelines is unaware of GitHub identities. For this reason, there’s no way to configure Azure Pipelines to automatically notify users of a build failure or a PR validation failure using their GitHub identity and email address. You must explicitly create new users in Azure Pipelines to replicate GitHub users. Once you create new users, you can configure their permissions in Azure DevOps to reflect their permissions in GitHub. You can also configure notifications in DevOps using their DevOps identity.

GitHub organization roles

GitHub organization member roles are found at https://github.com/orgs/your-organization/people (replace your-organization).

DevOps organization member permissions are found at https://dev.azure.com/your-organization/_settings/security (replace your-organization).

Roles in a GitHub organization and equivalent roles in an Azure DevOps organization are shown below.

GitHub user account roles

A GitHub user account has one role, which is ownership of the account.

DevOps organization member permissions are found at https://dev.azure.com/your-organization/_settings/security (replace your-organization).

The GitHub user account role maps to DevOps organization permissions as follows.

GitHub repository permissions

GitHub repository permissions are found at https://github.com/your-organization/your-repository/settings/collaboration (replace your-organization and your-repository).

DevOps project permissions are found at https://dev.azure.com/your-organization/your-project/_settings/security (replace your-organization and your-project).

Equivalent permissions between GitHub repositories and Azure DevOps Projects are as follows.

If your GitHub repository grants permission to teams, you can create matching teams in the Teams section of your Azure DevOps project settings. Then, add the teams to the security groups above, just like users.

Pipeline-specific permissions

To grant permissions to users or teams for specific pipelines in a DevOps project, follow these steps:

1. Visit the project's Pipelines page (for example, https://dev.azure.com/your-organization/your-project/_build).
2. Select the pipeline for which to set specific permissions.
3. From the '...' context menu, select Security.
4. Select Add... to add a specific user, team, or group and customize their permissions for the pipeline.

Access to GitHub repositories

Azure Pipelines must be granted access to your repositories to trigger their builds, and fetch their code during builds.

There are three authentication types for granting Azure Pipelines access to your GitHub repositories while creating a pipeline.

GitHub app authentication

The Azure Pipelines GitHub App is the recommended authentication type for continuous integration pipelines. After you install the GitHub App in your GitHub account or organization, your pipeline will run without using your personal GitHub identity. Builds and GitHub status updates will be performed using the Azure Pipelines identity. The app works with GitHub Checks to display build, test, and code coverage results in GitHub.

To use the GitHub App, install it in your GitHub organization or user account for some or all repositories. The GitHub App can be installed and uninstalled from the app's homepage.

After installation, the GitHub App will become Azure Pipelines' default method of authentication to GitHub (instead of OAuth) when pipelines are created for the repositories.

If you install the GitHub App for all repositories in a GitHub organization, you don't need to worry about Azure Pipelines sending mass emails or automatically setting up pipelines on your behalf. As an alternative to installing the app for all repositories, repository admins can install it one at a time for individual repositories. This requires more work for admins, but has no advantage nor disadvantage.

Permissions needed in GitHub

Installation of Azure Pipelines GitHub app requires you to be a GitHub organization owner or repository admin. In addition, to create a pipeline for a GitHub repository with continuous integration and pull request triggers, you must have the required GitHub permissions configured. Otherwise, the repository will not appear in the repository list while creating a pipeline. Depending on the authentication type and ownership of the repository, ensure that the appropriate access is configured.

• If the repo is in your personal GitHub account, install the Azure Pipelines GitHub App in your personal GitHub account, and you’ll be able to list this repository when creating the pipeline in Azure Pipelines.

• If the repo is in someone else's personal GitHub account, the other person must install the Azure Pipelines GitHub App in their personal GitHub account. You must be added as a collaborator in the repository's settings under "Collaborators". Accept the invitation to be a collaborator using the link that is emailed to you. Once you’ve done so, you can create a pipeline for that repository.

• If the repo is in a GitHub organization that you own, install the Azure Pipelines GitHub App in the GitHub organization. You must also be added as a collaborator, or your team must be added, in the repository's settings under "Collaborators and teams".

• If the repo is in a GitHub organization that someone else owns, a GitHub organization owner or repository admin must install the Azure Pipelines GitHub App in the organization. You must be added as a collaborator, or your team must be added, in the repository's settings under "Collaborators and teams". Accept the invitation to be a collaborator using the link that is emailed to you.

GitHub App permissions

The GitHub App requests the following permissions during installation:

Troubleshooting GitHub App installation

GitHub may display an error such as:

You do not have permission to modify this app on your-organization. Please contact an Organization Owner.

This means that the GitHub App is likely already installed for your organization. When you create a pipeline for a repository in the organization, the GitHub App will automatically be used to connect to GitHub.

Create pipelines in multiple Azure DevOps organizations and projects

Once the GitHub App is installed, pipelines can be created for the organization's repositories in different Azure DevOps organizations and projects. However, if you create pipelines for a single repository in multiple Azure DevOps organizations, only the first organization's pipelines can be automatically triggered by GitHub commits or pull requests. Manual or scheduled builds are still possible in secondary Azure DevOps organizations.

OAuth authentication

OAuth is the simplest authentication type to get started with for repositories in your personal GitHub account. GitHub status updates will be performed on behalf of your personal GitHub identity. For pipelines to keep working, your repository access must remain active. Some GitHub features, like Checks, are unavailable with OAuth and require the GitHub App.

To use OAuth, select Choose a different connection below the list of repositories while creating a pipeline. Then, select Authorize to sign into GitHub and authorize with OAuth. An OAuth connection will be saved in your Azure DevOps project for later use, and used in the pipeline being created.

Permissions needed in GitHub

To create a pipeline for a GitHub repository with continuous integration and pull request triggers, you must have the required GitHub permissions configured. Otherwise, the repository will not appear in the repository list while creating a pipeline. Depending on the authentication type and ownership of the repository, ensure that the appropriate access is configured.

• If the repo is in your personal GitHub account, at least once, authenticate to GitHub with OAuth using your personal GitHub account credentials. This can be done in Azure DevOps project settings under Pipelines > Service connections > New service connection > GitHub > Authorize. Grant Azure Pipelines access to your repositories under "Permissions" here.

• If the repo is in someone else's personal GitHub account, at least once, the other person must authenticate to GitHub with OAuth using their personal GitHub account credentials. This can be done in Azure DevOps project settings under Pipelines > Service connections > New service connection > GitHub > Authorize. The other person must grant Azure Pipelines access to their repositories under "Permissions" here. You must be added as a collaborator in the repository's settings under "Collaborators". Accept the invitation to be a collaborator using the link that is emailed to you.

• If the repo is in a GitHub organization that you own, at least once, authenticate to GitHub with OAuth using your personal GitHub account credentials. This can be done in Azure DevOps project settings under Pipelines > Service connections > New service connection > GitHub > Authorize. Grant Azure Pipelines access to your organization under "Organization access" here. You must be added as a collaborator, or your team must be added, in the repository's settings under "Collaborators and teams".

• If the repo is in a GitHub organization that someone else owns, at least once, a GitHub organization owner must authenticate to GitHub with OAuth using their personal GitHub account credentials. This can be done in Azure DevOps project settings under Pipelines > Service connections > New service connection > GitHub > Authorize. The organization owner must grant Azure Pipelines access to the organization under "Organization access" here. You must be added as a collaborator, or your team must be added, in the repository's settings under "Collaborators and teams". Accept the invitation to be a collaborator using the link that is emailed to you.

Revoke OAuth access

After authorizing Azure Pipelines to use OAuth, to later revoke it and prevent further use, visit OAuth Apps in your GitHub settings. You can also delete it from the list of GitHub service connections in your Azure DevOps project settings.

Personal access token (PAT) authentication

PATs are effectively the same as OAuth, but allow you to control which permissions are granted to Azure Pipelines. Builds and GitHub status updates will be performed on behalf of your personal GitHub identity. For builds to keep working, your repository access must remain active.

To create a PAT, visit Personal access tokens in your GitHub settings. The required permissions are repo, admin:repo_hook, read:user, and user:email. These are the same permissions required when using OAuth above. Copy the generated PAT to the clipboard and paste it into a new GitHub service connection in your Azure DevOps project settings. For future recall, name the service connection after your GitHub username. It will be available in your Azure DevOps project for later use when creating pipelines.

Permissions needed in GitHub

To create a pipeline for a GitHub repository with continuous integration and pull request triggers, you must have the required GitHub permissions configured. Otherwise, the repository will not appear in the repository list while creating a pipeline. Depending on the authentication type and ownership of the repository, ensure that the following access is configured.

• If the repo is in your personal GitHub account, the PAT must have the required access scopes under Personal access tokens: repo, admin:repo_hook, read:user, and user:email.

• If the repo is in someone else's personal GitHub account, the PAT must have the required access scopes under Personal access tokens: repo, admin:repo_hook, read:user, and user:email. You must be added as a collaborator in the repository's settings under "Collaborators". Accept the invitation to be a collaborator using the link that is emailed to you.

• If the repo is in a GitHub organization that you own, the PAT must have the required access scopes under Personal access tokens: repo, admin:repo_hook, read:user, and user:email. You must be added as a collaborator, or your team must be added, in the repository's settings under "Collaborators and teams".

• If the repo is in a GitHub organization that someone else owns, the PAT must have the required access scopes under Personal access tokens: repo, admin:repo_hook, read:user, and user:email. You must be added as a collaborator, or your team must be added, in the repository's settings under "Collaborators and teams". Accept the invitation to be a collaborator using the link that is emailed to you.

Revoke PAT access

After authorizing Azure Pipelines to use a PAT, to later delete it and prevent further use, visit Personal access tokens in your GitHub settings. You can also delete it from the list of GitHub service connections in your Azure DevOps project settings.

CI triggers

Continuous integration (CI) triggers cause a pipeline to run whenever you push an update to the specified branches or you push specified tags.

trigger:
- main
- releases/*

# specific branch build
trigger:
  branches:
    include:
    - main
    - releases/*
    exclude:
    - releases/old*

trigger:
  branches:
    include:
      - refs/tags/{tagname}
    exclude:
      - refs/tags/{othertagname}

trigger:
  branches:
    include:
    - '*'  # must quote since "*" is a YAML reserved character; we want a string

# specific branch build with batching
trigger:
  batch: true
  branches:
    include:
    - main

# specific path build
trigger:
  branches:
    include:
    - main
    - releases/*
  paths:
    include:
    - docs
    exclude:
    - docs/README.md

# specific tag
trigger:
  tags:
    include:
    - v2.*
    exclude:
    - v2.0

# A pipeline with no CI trigger
trigger: none

Skipping CI for individual commits

You can also tell Azure Pipelines to skip running a pipeline that a push would normally trigger. Just include [skip ci] in the message or description of any of the commits that are part of a push, and Azure Pipelines will skip running CI for this push. You can also use any of the following variations.

• [skip ci] or [ci skip]
• skip-checks: true or skip-checks:true
• [skip azurepipelines] or [azurepipelines skip]
• [skip azpipelines] or [azpipelines skip]
• [skip azp] or [azp skip]
• ***NO_CI***

Using the trigger type in conditions

It is a common scenario to run different steps, jobs, or stages in your pipeline depending on the type of trigger that started the run. You can do this using the system variable Build.Reason. For example, add the following condition to your step, job, or stage to exclude it from PR validations.

condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

Behavior of triggers when new branches are created

It is common to configure multiple pipelines for the same repository. For instance, you may have one pipeline to build the docs for your app and another to build the source code. You may configure CI triggers with appropriate branch filters and path filters in each of these pipelines. For instance, you may want one pipeline to trigger when you push an update to the docs folder, and another one to trigger when you push an update to your application code. In these cases, you need to understand how the pipelines are triggered when a new branch is created.

Here is the behavior when you push a new branch (that matches the branch filters) to your repository:

• If your pipeline has path filters, it will be triggered only if the new branch has changes to files that match that path filter.
• If your pipeline does not have path filters, it will be triggered even if there are no changes in the new branch.

Wildcards

When specifying a branch, tag, or path, you may use an exact name or a wildcard. Wildcards patterns allow * to match zero or more characters and ? to match a single character.

• If you start your pattern with * in a YAML pipeline, you must wrap the pattern in quotes, like "*-releases".
• For branches and tags:
  • A wildcard may appear anywhere in the pattern.
• For paths:
  • In Azure DevOps Server 2022 and higher, including Azure DevOps Services, a wildcard may appear anywhere within a path pattern and you may use * or ?.
  • In Azure DevOps Server 2020 and lower, you may include * as the final character, but it doesn't do anything differently from specifying the directory name by itself. You may not include * in the middle of a path filter, and you may not use ?.

trigger:
  branches:
    include:
    - main
    - releases/*
    - feature/*
    exclude:
    - releases/old*
    - feature/*-working
  paths:
    include:
    - docs/*.md

PR triggers

Pull request (PR) triggers cause a pipeline to run whenever a pull request is opened with one of the specified target branches, or when updates are made to such a pull request.

pr:
- main
- releases/*

pr:
  branches:
    include:
    - '*'  # must quote since "*" is a YAML reserved character; we want a string

# specific branch
pr:
  branches:
    include:
    - main
    - releases/*
    exclude:
    - releases/old*

# specific path
pr:
  branches:
    include:
    - main
    - releases/*
  paths:
    include:
    - docs
    exclude:
    - docs/README.md

# auto cancel false
pr:
  autoCancel: false
  branches:
    include:
    - main

pr:
  autoCancel: boolean # indicates whether additional pushes to a PR should cancel in-progress runs for the same PR. Defaults to true
  branches:
    include: [ string ] # branch names which will trigger a build
    exclude: [ string ] # branch names which will not
  paths:
    include: [ string ] # file paths which must match to trigger a build
    exclude: [ string ] # file paths which will not trigger a build
  drafts: boolean # whether to build draft PRs, defaults to true

# no PR triggers
pr: none

If you have an open PR and you push changes to its source branch, multiple pipelines may run:

• The pipelines that have a PR trigger on the PR's target branch will run on the merge commit (the merged code between the source and target branches of the pull request), regardless if there exist pushed commits whose messages or descriptions contain [skip ci] (or any of its variants).
• The pipelines triggered by changes to the PR's source branch, if there are no pushed commits whose messages or descriptions contain [skip ci] (or any of its variants). If at least one pushed commit contains [skip ci], the pipelines will not run.

Finally, after you merge the PR, Azure Pipelines will run the CI pipelines triggered by pushes to the target branch, if the merge commit's message or description doesn't contain [skip ci] (or any of its variants).

Protected branches

You can run a validation build with each commit or pull request that targets a branch, and even prevent pull requests from merging until a validation build succeeds.

To configure mandatory validation builds for a GitHub repository, you must be its owner, a collaborator with the Admin role, or a GitHub organization member with the Write role.

1. First, create a pipeline for the repository and build it at least once so that its status is posted to GitHub, thereby making GitHub aware of the pipeline's name.

2. Next, follow GitHub's documentation for configuring protected branches in the repository's settings.

   For the status check, select the name of your pipeline in the Status checks list.

   

Contributions from external sources

If your GitHub repository is open source, you can make your Azure DevOps project public so that anyone can view your pipeline's build results, logs, and test results without signing in. When users outside your organization fork your repository and submit pull requests, they can view the status of builds that automatically validate those pull requests.

You should keep in mind the following considerations when using Azure Pipelines in a public project when accepting contributions from external sources.

• Access restrictions
• Validate contributions from forks
• Important security considerations

Access restrictions

Be aware of the following access restrictions when you're running pipelines in Azure DevOps public projects:

• Secrets: By default, secrets associated with your pipeline aren’t made available to pull request validations of forks. See Validate contributions from forks.
• Cross-project access: All pipelines in an Azure DevOps public project run with an access token restricted to the project. Pipelines in a public project can access resources such as build artifacts or test results only within the project and not in other projects of the Azure DevOps organization.
• Azure Artifacts packages: If your pipelines need access to packages from Azure Artifacts, you must explicitly grant permission to the Project Build Service account to access the package feeds.

Contributions from forks

When you create a pipeline, it’s automatically triggered for pull requests from forks of your repository. You can change this behavior, carefully considering how it affects security. To enable or disable this behavior:

1. Go to your Azure DevOps project. Select Pipelines, locate your pipeline, and select Edit.
2. Select the Triggers tab. After enabling the Pull request trigger, enable or disable the Build pull requests from forks of this repository check box.

By default with GitHub pipelines, secrets associated with your build pipeline aren’t made available to pull request builds of forks. These secrets are enabled by default with GitHub Enterprise Server pipelines. Secrets include:

• A security token with access to your GitHub repository.
• These items, if your pipeline uses them:
  • Service connection credentials
  • Files from the secure files library
  • Build variables marked secret

To bypass this precaution on GitHub pipelines, enable the Make secrets available to builds of forks check box. Be aware of this setting's effect on security.

For more information, see Repository protection - Forks.

You can define centrally how pipelines build PRs from forked GitHub repositories using the Limit building pull requests from forked GitHub repositories control. It's available at organization and project level. You can choose to:

• Disable building pull requests from forked repositories
• Securely build pull requests from forked repositories
• Customize rules for building pull requests from forked repositories

Starting with Sprint 229, to improve the security of your pipelines, Azure Pipelines no longer automatically builds pull requests from forked GitHub repositories. For new projects and organizations, the default value of the Limit building pull requests from forked GitHub repositories setting is Disable building pull requests from forked repositories.

When you choose the Securely build pull requests from forked repositories option, all pipelines, organization or project-wide, cannot make secrets available to builds of PRs from forked repositories, cannot make these builds have the same permissions as normal builds, and must be triggered by a PR comment. Projects can still decide to not allow pipelines to build such PRs.

When you choose the Customize option, you can define how to restrict pipeline settings. For example, you can ensure that all pipelines require a comment in order to build a PR from a forked GitHub repo, when the PR belongs to non-team members and non-contributors. But, you can choose to allow them to make secrets available to such builds. Projects can decide to not allow pipelines to build such PRs, or to build them securely, or have even more restrictive settings than what is specified at the organization level.

The control is off for existing organizations. Starting September 2023, new organizations have Securely build pull requests from forked repositories turned on by default.

Important security considerations

A GitHub user can fork your repository, change it, and create a pull request to propose changes to your repository. This pull request could contain malicious code to run as part of your triggered build. Such code can cause harm in the following ways:

• Leak secrets from your pipeline. To mitigate this risk, don’t enable the Make secrets available to builds of forks check box if your repository is public or untrusted users can submit pull requests that automatically trigger builds. This option is disabled by default.

• Compromise the machine running the agent to steal code or secrets from other pipelines. To mitigate this:

  • Use a Microsoft-hosted agent pool to build pull requests from forks. Microsoft-hosted agent machines are immediately deleted after they complete a build, so there’s no lasting impact if they're compromised.

  • If you must use a self-hosted agent, don’t store any secrets or perform other builds and releases that use secrets on the same agent, unless your repository is private and you trust pull request creators.

Comment triggers

Repository collaborators can comment on a pull request to manually run a pipeline. Here are a few common reasons for why you might want to do this:

• You may not want to automatically build pull requests from unknown users until their changes can be reviewed. You want one of your team members to first review their code and then run the pipeline. This is commonly used as a security measure when building contributed code from forked repositories.
• You may want to run an optional test suite or one more validation build.

To enable comment triggers, you must follow the following two steps:

1. Enable pull request triggers for your pipeline, and make sure that you didn’t exclude the target branch.
2. In the Azure Pipelines web portal, edit your pipeline and choose More actions, Triggers. Then, under Pull request validation, enable Require a team member's comment before building a pull request.
   • Choose On all pull requests to require a team member's comment before building a pull request. With this workflow, a team member reviews the pull request and triggers the build with a comment once the pull request is deemed safe.
   • Choose Only on pull requests from non-team members to require a team member's comment only when a PR is made by a non-team member. In this workflow, a team member doesn't need a secondary team member's review to trigger a build.

With these two changes, the pull request validation build won’t be triggered automatically, unless Only on pull requests from non-team members is selected and the PR is made by a team member. Only repository owners and collaborators with 'Write' permission can trigger the build by commenting on the pull request with /AzurePipelines run or /AzurePipelines run <pipeline-name>.

The following commands can be issued to Azure Pipelines in comments:

Troubleshoot pull request comment triggers

If you have the necessary repository permissions, but pipelines aren't getting triggered by your comments, make sure that your membership is public in the repository's organization, or directly add yourself as a repository collaborator. Pipelines can’t see private organization members unless they are direct collaborators or belong to a team that is a direct collaborator. You can change your GitHub organization membership from private to public here (replace Your-Organization with your organization name): https://github.com/orgs/Your-Organization/people.

Informational runs

An informational run tells you Azure DevOps failed to retrieve a YAML pipeline's source code. Source code retrieval happens in response to external events, for example, a pushed commit. It also happens in response to internal triggers, for example, to check if there are code changes and start a scheduled run or not. Source code retrieval can fail for multiple reasons, with a frequent one being request throttling by the git repository provider. The existence of an informational run doesn't necessarily mean Azure DevOps was going to run the pipeline.

An informational run looks like in the following screenshot.

You can recognize an informational run by the following attributes:

• Status is Canceled
• Duration is < 1s
• Run name contains one of the following texts:
  • Could not retrieve file content for {file_path} from repository {repo_name} hosted on {host} using commit {commit_sha}.
  • Could not retrieve content for object {commit_sha} from repository {repo_name} hosted on {host}.
  • Could not retrieve the tree object {tree_sha} from the repository {repo_name} hosted on {host}.
  • Could not find {file_path} from repository {repo_name} hosted on {host} using version {commit_sha}. One of the directories in the path contains too many files or subdirectories.
• Run name generally contains the BitBucket / GitHub error that caused the YAML pipeline load to fail
• No stages / jobs / steps

Learn more about informational runs.

Checkout

When a pipeline is triggered, Azure Pipelines pulls your source code from the Azure Repos Git repository. You can control various aspects of how this happens.

Preferred version of Git

The Windows agent comes with its own copy of Git. If you prefer to supply your own Git rather than use the included copy, set System.PreferGitFromPath to true. This setting is always true on non-Windows agents.

Checkout path

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

Submodules

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

The build pipeline will check out your Git submodules as long as they are:

• Unauthenticated: A public, unauthenticated repo with no credentials required to clone or fetch.

• Authenticated:

  • Contained in the same project as the Azure Repos Git repo specified above. The same credentials that are used by the agent to get the sources from the main repository are also used to get the sources for submodules.

  • Added by using a URL relative to the main repository. For example

    • This one would be checked out: git submodule add ../../../FabrikamFiberProject/_git/FabrikamFiber FabrikamFiber

      In this example the submodule refers to a repo (FabrikamFiber) in the same Azure DevOps organization, but in a different project (FabrikamFiberProject). The same credentials that are used by the agent to get the sources from the main repository are also used to get the sources for submodules. This requires that the job access token has access to the repository in the second project. If you restricted the job access token as explained in the section above, then you won't be able to do this. You can allow the job access token to access the repo in the second project by either (a) explicitly granting access to the project build service account in the second project or (b) using collection-scoped access tokens instead of project-scoped tokens for the entire organization. For more information about these options and their security implications, see Access repositories, artifacts, and other resources.

    • This one would not be checked out: git submodule add https://[email protected]/fabrikam-fiber/FabrikamFiberProject/_git/FabrikamFiber FabrikamFiber

Alternative to using the Checkout submodules option

In some cases you can't use the Checkout submodules option. You might have a scenario where a different set of credentials are needed to access the submodules. This can happen, for example, if your main repository and submodule repositories aren't stored in the same Azure DevOps organization, or if your job access token does not have access to the repository in a different project.

If you can't use the Checkout submodules option, then you can instead use a custom script step to fetch submodules. First, get a personal access token (PAT) and prefix it with pat:. Next, base64-encode this prefixed string to create a basic auth token. Finally, add this script to your pipeline:

git -c http.https://<url of submodule repository>.extraheader="AUTHORIZATION: Basic <BASE64_ENCODED_STRING>" submodule update --init --recursive

Be sure to replace "<BASE64_ENCODED_STRING>" with your Base64-encoded "pat:token" string.

Use a secret variable in your project or build pipeline to store the basic auth token that you generated. Use that variable to populate the secret in the above Git command.

Sync tags

The checkout step uses the --tags option when fetching the contents of a Git repository. This causes the server to fetch all tags as well as all objects that are pointed to by those tags. This increases the time to run the task in a pipeline, particularly if you have a large repository with a number of tags. Furthermore, the checkout step syncs tags even when you enable the shallow fetch option, thereby possibly defeating its purpose. To reduce the amount of data fetched or pulled from a Git repository, Microsoft has added a new option to checkout to control the behavior of syncing tags. This option is available both in classic and YAML pipelines.

Whether to synchronize tags when checking out a repository can be configured in YAML by setting the fetchTags property, and in the UI by configuring the Sync tags setting.

steps:
- checkout: self
  fetchTags: true

Default behavior

• For existing pipelines created before the release of Azure DevOps sprint 209, released in September 2022, the default for syncing tags remains the same as the existing behavior before the Sync tags options was added, which is true.
• For new pipelines created after Azure DevOps sprint release 209, the default for syncing tags is false.

Shallow fetch

You may want to limit how far back in history to download. Effectively this results in git fetch --depth=n. If your repository is large, this option might make your build pipeline more efficient. Your repository might be large if it has been in use for a long time and has sizeable history. It also might be large if you added and later deleted large files.

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

In these cases this option can help you conserve network and storage resources. It might also save time. The reason it doesn't always save time is because in some situations the server might need to spend time calculating the commits to download for the depth you specify.

Don't sync sources

You may want to skip fetching new commits. This option can be useful in cases when you want to:

• Git init, config, and fetch using your own custom options.

• Use a build pipeline to just run automation (for example some scripts) that do not depend on code in version control.

steps:
- checkout: none  # Don't sync sources

Clean build

You can perform different forms of cleaning the working directory of your self-hosted agent before a build runs.

In general, for faster performance of your self-hosted agents, don't clean the repo. In this case, to get the best performance, make sure you're also building incrementally by disabling any Clean option of the task or tool you're using to build.

If you do need to clean the repo (for example to avoid problems caused by residual files from a previous build), your options are below.

steps:
- checkout: self  # self represents the repo where the initial Pipelines YAML file was found
  clean: boolean  # whether to fetch clean each time
  fetchDepth: number  # the depth of commits to ask Git to fetch
  lfs: boolean  # whether to download Git-LFS files
  submodules: true | recursive  # set to 'true' for a single level of submodules or 'recursive' to get submodules of submodules
  path: string  # path to check out source code, relative to the agent's build directory (e.g. \_work\1)
  persistCredentials: boolean  # set to 'true' to leave the OAuth token in the Git config after the initial fetch

git clean -ffdx
git reset --hard HEAD

jobs:
- job: string  # name of the job, A-Z, a-z, 0-9, and underscore
  ...
  workspace:
    clean: outputs | resources | all # what to clean up before the job runs

Label sources

You may want to label your source code files to enable your team to easily identify which version of each file is included in the completed build. You also have the option to specify whether the source code should be labeled for all builds or only for successful builds.

In the Tag format you can use user-defined and predefined variables that have a scope of "All." For example:

$(Build.DefinitionName)_$(Build.DefinitionVersion)_$(Build.BuildId)_$(Build.BuildNumber)_$(My.Variable)

The first four variables are predefined. My.Variable can be defined by you on the variables tab.

The build pipeline labels your sources with a Git tag.

Some build variables might yield a value that is not a valid label. For example, variables such as $(Build.RequestedFor) and $(Build.DefinitionName) can contain white space. If the value contains white space, the tag is not created.

After the sources are tagged by your build pipeline, an artifact with the Git ref refs/tags/{tag} is automatically added to the completed build. This gives your team additional traceability and a more user-friendly way to navigate from the build to the code that was built. The tag is considered a build artifact since it is produced by the build. When the build is deleted either manually or through a retention policy, the tag is also deleted.

Pre-defined variables

When you build a GitHub repository, most of the predefined variables are available to your jobs. However, since Azure Pipelines doesn’t recognize the identity of a user making an update in GitHub, the following variables are set to system identity instead of user's identity:

• Build.RequestedFor
• Build.RequestedForId
• Build.RequestedForEmail

Status updates

There are two types of statuses that Azure Pipelines posts back to GitHub - basic statuses and GitHub Check Runs. GitHub Checks functionality is only available with GitHub Apps.

Pipeline statuses show up in various places in the GitHub UI.

• For PRs, they’re displayed on the PR conversations tab.
• For individual commits, they’re displayed when hovering over the status mark after the commit time on the repo's commits tab.

PAT or OAuth GitHub connections

For pipelines using PAT or OAuth GitHub connections, statuses are posted back to the commit/PR that triggered the run. The GitHub status API is used to post such updates. These statuses contain limited information: pipeline status (failed, success), URL to link back to the build pipeline, and a brief description of the status.

Statuses for PAT or OAuth GitHub connections are only sent at the run level. In other words, you can have a single status updated for an entire run. If you have multiple jobs in a run, you can’t post a separate status for each job. However, multiple pipelines can post separate statuses to the same commit.

GitHub Checks

For pipelines set up using the Azure Pipelines GitHub app, the status is posted back in the form of GitHub Checks. GitHub Checks allow for sending detailed information about the pipeline status and test, code coverage, and errors. The GitHub Checks API can be found here.

For every pipeline using the GitHub App, Checks are posted back for the overall run and each job in that run.

GitHub allows three options when one or more Check Runs fail for a PR/commit. You can choose to "rerun" the individual Check, rerun all the failing Checks on that PR/commit, or rerun all the Checks, whether they succeeded initially or not.

Clicking on the "Rerun" link next to the Check Run name will result in Azure Pipelines retrying the run that generated the Check Run. The resultant run will have the same run number and will use the same version of the source code, configuration, and YAML file as the initial build. Only those jobs that failed in the initial run and any dependent downstream jobs will be run again. Clicking on the "Rerun all failing checks" link will have the same effect. This is the same behavior as clicking "Retry run" in the Azure Pipelines UI. Clicking on "Rerun all checks" will result in a new run, with a new run number and will pick up changes in the configuration or YAML file.

Limitations

• For best performance, we recommend a maximum of 50 pipelines in a single repository. For acceptable performance, we recommend a maximum of 100 pipelines in a single repository. The time required to process a push to a repository increases with the number of pipelines in that repository. Whenever there's push to a repository, Azure Pipelines needs to load all YAML pipelines in that repository to figure out if any of them need to run, and loading each pipeline incurs a performance penalty. In addition to performance issues, having too many pipelines in a single repository can lead to throttling on GitHub's side, as Azure Pipelines may make too many requests in a short amount of time.

• The use of extends and include templates in a pipeline impacts the rate at which Azure Pipelines makes GitHub API requests and can lead to throttling on GitHub's side. Before running a pipeline, Azure Pipelines needs to generate the complete YAML code, so it needs to fetch all template files.

• Azure Pipelines loads a maximum of 2000 branches from a repository into dropdown lists in the Azure DevOps Portal, for example in the Select a branch window for the Default branch for manual and scheduled builds setting, or when choosing a branch when running a pipeline manually.

  If you don't see your desired branch in the list, type the desired branch name manually in the Default branch for manual and scheduled builds field.

  If you click the ellipsis and open the Select a branch dialogue and close it without choosing a valid branch from the drop-down list, you may see a Some settings need attention message and a This setting is required message below Default Branch for manual and scheduled builds. To work around this message, reopen the pipeline and enter the name directly in the Default branch for manual and scheduled builds field.

FAQ

Problems related to GitHub integration fall into the following categories:

• Connection types: I’m not sure what connection type I’m using to connect my pipeline to GitHub.
• Failing triggers: My pipeline isn’t being triggered when I push an update to the repo.
• Failing checkout: My pipeline is being triggered, but it fails in the checkout step.
• Wrong version: My pipeline runs, but it’s using an unexpected version of the source/YAML.
• Missing status updates: My GitHub PRs are blocked because Azure Pipelines didn’t report a status update.

Connection types

To troubleshoot triggers, how do I know the type of GitHub connection I'm using for my pipeline?

Troubleshooting problems with triggers very much depends on the type of GitHub connection you use in your pipeline. There are two ways to determine the type of connection - from GitHub and from Azure Pipelines.

• From GitHub: If a repo is set up to use the GitHub app, then the statuses on PRs and commits will be Check Runs. If the repo has Azure Pipelines set up with OAuth or PAT connections, the statuses will be the "old" style of statuses. A quick way to determine if the statuses are Check Runs or simple statuses is to look at the "conversation" tab on a GitHub PR.

  • If the "Details" link redirects to the Checks tab, it’s a Check Run and the repo is using the app.
  • If the "Details" link redirects to the Azure DevOps pipeline, then the status is an "old style" status and the repo isn’t using the app.

• From Azure Pipelines: You can also determine the type of connection by inspecting the pipeline in Azure Pipelines UI. Open the editor for the pipeline. Select Triggers to open the classic editor for the pipeline. Then, select YAML tab and then the Get sources step. You'll notice a banner Authorized using connection: indicating the service connection that was used to integrate the pipeline with GitHub. The name of the service connection is a hyperlink. Select it to navigate to the service connection properties. The properties of the service connection will indicate the type of connection being used:

  • Azure Pipelines app indicates GitHub app connection
  • oauth indicates OAuth connection
  • personalaccesstoken indicates PAT authentication

How do I switch my pipeline to use GitHub app instead of OAuth?

Using a GitHub app instead of OAuth or PAT connection is the recommended integration between GitHub and Azure Pipelines. To switch to GitHub app, follow these steps:

1. Navigate here and install the app in the GitHub organization of your repository.
2. During installation, you'll be redirected to Azure DevOps to choose an Azure DevOps organization and project. Choose the organization and project that contain the classic build pipeline you want to use the app for. This choice associates the GitHub App installation with your Azure DevOps organization. If you choose incorrectly, you can visit this page to uninstall the GitHub app from your GitHub org and start over.
3. In the next page that appears, you don’t need to proceed creating a new pipeline.
4. Edit your pipeline by visiting the Pipelines page (e.g., https://dev.azure.com/YOUR_ORG_NAME/YOUR_PROJECT_NAME/_build), selecting your pipeline, and clicking Edit.
5. If this is a YAML pipeline, select the Triggers menu to open the classic editor.
6. Select the "Get sources" step in the pipeline.
7. On the green bar with text "Authorized using connection", select "Change" and select the GitHub App connection with the same name as the GitHub organization in which you installed the app.
8. On the toolbar, select "Save and queue" and then "Save and queue". Select the link to the pipeline run that was queued to make sure it succeeds.
9. Create (or close and reopen) a pull request in your GitHub repository to verify that a build is successfully queued in its "Checks" section.

Why isn't a GitHub repository displayed for me to choose in Azure Pipelines?

Depending on the authentication type and ownership of the repository, specific permissions are required.

• If you're using the GitHub App, see GitHub App authentication.
• If you're using OAuth, see OAuth authentication.
• If you're using PATs, see Personal access token (PAT) authentication.

When I select a repository during pipeline creation, I get an error "The repository {repo-name} is in use with the Azure Pipelines GitHub App in another Azure DevOps organization."

This means that your repository is already associated with a pipeline in a different organization. CI and PR events from this repository won't work as they’ll be delivered to the other organization. Here are the steps you should take to remove the mapping to the other organization before proceeding to create a pipeline.

1. Open a pull request in your GitHub repository, and make the comment /azp where. This reports back the Azure DevOps organization that the repository is mapped to.

2. To change the mapping, uninstall the app from the GitHub organization, and reinstall it. As you reinstall it, make sure to select the correct organization when you’re redirected to Azure DevOps.

Failing triggers

I just created a new YAML pipeline with CI/PR triggers, but the pipeline isn't being triggered.

Follow each of these steps to troubleshoot your failing triggers:

• Are your YAML CI or PR triggers overridden by pipeline settings in the UI? While editing your pipeline, choose ... and then Triggers.

  

  Check the Override the YAML trigger from here setting for the types of trigger (Continuous integration or Pull request validation) available for your repo.

  

• Are you using the GitHub app connection to connect the pipeline to GitHub? See Connection types to determine the type of connection you have. If you’re using a GitHub app connection, follow these steps:

  • Is the mapping set up properly between GitHub and Azure DevOps? Open a pull request in your GitHub repository, and make the comment /azp where. This reports back the Azure DevOps organization that the repository is mapped to.

    • If no organizations are set up to build this repository using the app, go to https://github.com/<org_name>/<repo_name>/settings/installations and complete the configuration of the app.

    • If a different Azure DevOps organization is reported, then someone has already established a pipeline for this repo in a different organization. We currently have the limitation that we can only map a GitHub repo to a single DevOps org. Only the pipelines in the first Azure DevOps org can be automatically triggered. To change the mapping, uninstall the app from the GitHub organization, and reinstall it. As you reinstall it, make sure to select the correct organization when you’re redirected to Azure DevOps.

• Are you using OAuth or PAT to connect the pipeline to GitHub? See Connection types to determine the type of connection you have. If you’re using a GitHub connection, follow these steps:

  1. OAuth and PAT connections rely on webhooks to communicate updates to Azure Pipelines. In GitHub, navigate to the settings for your repository, then to Webhooks. Verify that the webhooks exist. Usually you should see three webhooks - push, pull_request, and issue_comment. If you don't, then you must re-create the service connection and update the pipeline to use the new service connection.

  2. Select each of the webhooks in GitHub and verify that the payload that corresponds to the user's commit exists and was sent successfully to Azure DevOps. You may see an error here if the event couldn’t be communicated to Azure DevOps.

• The traffic from Azure DevOps could be throttled by GitHub. When Azure Pipelines receives a notification from GitHub, it tries to contact GitHub and fetch more information about the repo and YAML file. If you have a repo with a large number of updates and pull requests, this call may fail due to such throttling. In this case, see if you can reduce the frequency of builds by using batching or stricter path/branch filters.

• Is your pipeline paused or disabled? Open the editor for the pipeline, and then select Settings to check. If your pipeline is paused or disabled, then triggers do not work.

• Have you updated the YAML file in the correct branch? If you push an update to a branch, then the YAML file in that same branch governs the CI behavior. If you push an update to a source branch, then the YAML file resulting from merging the source branch with the target branch governs the PR behavior. Make sure that the YAML file in the correct branch has the necessary CI or PR configuration.

• Have you configured the trigger correctly? When you define a YAML trigger, you can specify both include and exclude clauses for branches, tags, and paths. Ensure that the include clause matches the details of your commit and that the exclude clause doesn't exclude them. Check the syntax for the triggers and make sure that it is accurate.

• Have you used variables in defining the trigger or the paths? That is not supported.

• Did you use templates for your YAML file? If so, make sure that your triggers are defined in the main YAML file. Triggers defined inside template files are not supported.

• Have you excluded the branches or paths to which you pushed your changes? Test by pushing a change to an included path in an included branch. Note that paths in triggers are case-sensitive. Make sure that you use the same case as those of real folders when specifying the paths in triggers.

• Did you just push a new branch? If so, the new branch may not start a new run. See the section "Behavior of triggers when new branches are created".

My CI or PR triggers have been working fine. But, they stopped working now.

First, go through the troubleshooting steps in the previous question, then follow these additional steps:

• Do you have merge conflicts in your PR? For a PR that didn't trigger a pipeline, open it and check whether it has a merge conflict. Resolve the merge conflict.

• Are you experiencing a delay in the processing of push or PR events? You can usually verify a delay by seeing if the issue is specific to a single pipeline or is common to all pipelines or repos in your project. If a push or a PR update to any of the repos exhibits this symptom, we might be experiencing delays in processing the update events. Here are some reasons why a delay may be happening:

  • We are experiencing a service outage on our status page. If the status page shows an issue, then our team must have already started working on it. Check the page frequently for updates on the issue.
  • Your repository contains too many YAML pipelines. For best performance, we recommend a maximum of 50 pipelines in a single repository. For acceptable performance, we recommend a maximum of 100 pipelines in a single repository. The more pipelines there are, the slower the processing of a push to that repository. Whenever there is push to a repository, Azure Pipelines needs to load all YAML pipelines in that repository, to figure out if any of them need to run, and each new pipeline incurs a performance penalty.

I do not want users to override the list of branches for triggers when they update the YAML file. How can I do this?

Users with permissions to contribute code can update the YAML file and include/exclude additional branches. As a result, users can include their own feature or user branch in their YAML file and push that update to a feature or user branch. This may cause the pipeline to be triggered for all updates to that branch. If you want to prevent this behavior, then you can:

1. Edit the pipeline in the Azure Pipelines UI.
2. Navigate to the Triggers menu.
3. Select Override the YAML continuous Integration trigger from here.
4. Specify the branches to include or exclude for the trigger.

When you follow these steps, any CI triggers specified in the YAML file are ignored.

Failing checkout

I see the following error in the log file during checkout step. How do I fix it?

remote: Repository not found.
fatal: repository <repo> not found

This could be caused by an outage of GitHub. Try to access the repository in GitHub and make sure that you’re able to.

Wrong version

A wrong version of the YAML file is being used in the pipeline. Why is that?

• For CI triggers, the YAML file that is in the branch you are pushing is evaluated to see if a CI build should be run.
• For PR triggers, the YAML file resulting from merging the source and target branches of the PR is evaluated to see if a PR build should be run.

Missing status updates

My PR in GitHub is blocked since Azure Pipelines didn’t update the status.

This could be a transient error that resulted in Azure DevOps not being able to communicate with GitHub. Retry the check-in GitHub if you use the GitHub app. Or, make a trivial update to the PR to see if the problem can be resolved.

Related articles

• Scheduled triggers
• Pipeline completion triggers