Azure Pipelines Resources Repositories Setup and Best Practices

Setting up Azure Pipelines resources repositories is a crucial step in automating your CI/CD process. This setup allows you to manage your code and infrastructure as code, making it easier to collaborate and deploy your applications.

You can connect your Azure Pipelines resources repositories to your Azure DevOps project using Git. Azure Pipelines supports multiple Git repositories, including Azure Repos, GitHub, and GitLab.

To set up your Azure Pipelines resources repositories, you'll need to create a new pipeline and connect it to your Git repository. This will allow you to automate your build, test, and deployment processes.

Azure Pipelines provides a free tier, which allows you to run 300 minutes of build time per month for free. This is a great option for small projects or teams just starting out with Azure Pipelines.

Azure Pipelines Configuration allows you to define resources that can be used throughout your pipeline. You can consume artifacts from a pipeline by defining a pipelines resource, which is unique to Azure Pipelines.

Only Azure Pipelines can use the pipelines resource, and you can set triggers for your continuous deployment (CD) workflows on a pipeline resource. The pipeline is a unique value that you can use to reference the pipeline resource later in your pipeline.

To define a pipeline resource, you can specify the source, which is the name of the pipeline that produced the pipeline artifact. You can also use the label defined by pipeline to reference the pipeline resource from other parts of your pipeline.

You can use the repository keyword to specify an external repository, which is useful if your pipeline has templates in another repository or you want to use multi-repo checkout with a repository that requires a service connection. Trigger evaluation only occurs on the default branch, so make sure to set the correct default branch or merge the YAML file into the current default branch.

To trigger a pipeline resource, you can use branch conditions or tags filters. Branch conditions can be used to trigger the pipeline on a specific branch, while tags filters can be used to trigger the pipeline based on specific tags set on the continuous integration (CI) or CD pipeline.

Here are some examples of pipeline resource definitions:

Branch filters are a powerful tool in Azure Pipelines Configuration that allow you to specify which branches trigger builds. You can use wildcard characters to match multiple branches at once.

To specify a branch filter, simply type the branch specification, such as "features/modules/*", and press Enter. This will trigger builds for any branch that matches this pattern.

You can also use branch filters to exclude certain branches from triggering builds. For example, if you want to trigger builds for all branches except those that start with "old", you can use the syntax "releases/*" and exclude "releases/old".

It's worth noting that branch filters are processed before path filters, so if you have both set up, the branch filter will take precedence.

Here's a breakdown of how branch filters work:

By using branch filters, you can fine-tune your Azure Pipelines Configuration to only trigger builds for the branches that matter most to your project.

When configuring Azure Pipelines, choosing the right Git version is crucial for a smooth build process.

The Windows agent comes with its own copy of Git, which is recommended by Microsoft. However, you have the option to supply your own Git version.

To use the version of Git installed on the agent machine, you can set the System.PreferGitFromPath pipeline variable to true in your pipelines.

On self-hosted agents, you can create a .env file in the agent root directory with the line System.PreferGitFromPath=true to enable this setting.

Here are the steps to override the default Git version:

This setting is always true on non-Windows agents, so you don't need to take any further action on those platforms.

To get started with Azure Pipelines configuration, you need to understand the setup process. The first step is to create a main repository where your pipeline lives, and then add Git submodules to it. You can add submodules using the `git submodule add` command, specifying the URL of the submodule repository.

In Azure DevOps, you can set the option "Limit job authorization scope to the current project for non-release pipelines" to Off to avoid permission issues when working with submodules. This is especially important if you're using a script to perform read-only operations on a repository in a public project.

Here are the steps to make things work:

1. Set the option "Limit job authorization scope to the current project for non-release pipelines" to Off.

2. Add the submodules as a URL relative to the main repository.

3. Set the `submodules` key to `true` under `stages: – stage: checkout jobs: – job: steps: – checkout: self` in your pipeline code.

4. Reference the submodule repositories in `resources.repositories.repository`.

5. Leverage the `use` statement and refer to the submodule repositories you defined in step 4.

For example, if you're referencing a submodule repository in a different project, you can use a relative path like `../../../OtherProject/_git/SubModuleInOtherProject` while adding the submodule. In your pipeline, reference the name with the `Project/RepoName` and you'll be fine.

By following these steps, you'll be able to get Git submodules in private Azure DevOps repositories to work in a pipeline.

You can consume artifacts from an external CI build system, like Jenkins, TeamCity, or CircleCI, using a builds resource. This allows you to integrate your pipeline with other build systems and leverage their capabilities.

A build resource can be from any external CI system, and you can write an extension to consume artifacts from your build service. You can introduce a new type of service as part of builds.

To use a builds resource, you need to define it in your pipeline. The build definition includes the resource type, which can be Jenkins, TeamCity, or CircleCI, and the trigger settings.

Here are the trigger settings for hosted Jenkins:

* Triggers are supported for hosted Jenkins only where Azure DevOps has line of sight with the Jenkins server.

You can set triggers for your continuous deployment (CD) workflows on a pipeline resource. The trigger settings include the pipeline resource trigger conditions, which can be based on branch, tags, or stages.

Here are the pipeline resource trigger conditions:

You can also use stages filters for evaluating trigger conditions for CD pipelines. Stages use the AND operator, and on successful completion of all the provided stages, the CD pipeline triggers.

Here are the stages filters:

You can also use tags filters for default version evaluation and for triggers. Tags use the AND operator.

Here are the tags filters:

Validating fork contributions in Azure Pipelines is a straightforward process. You can create forks only within the same organization that your project is part of.

To build pull requests from Azure Repos forks, you can create them just like you would within the same repository or project.

Building pull requests from forks is no different from building pull requests within the same repository or project.

Repository Management is crucial for Azure Pipelines. You must add a checkout step to consume a Repository in your Pipeline, and for Azure Repos Git, you must specify type: git.

If the Azure Repo exists in the same Azure DevOps Project, you can use name: someRepo. However, if it exists in a different Azure DevOps Project, you must use name: someProject/someRepo. For other types of repos, you must use name: orgName/someRepo.

You can use a repository resource to reference an additional repository in your pipeline, which is helpful when you have submodules in your main repository. To do this, you need to specify an ID to reference the resource in use, the type of repository (Git), and the name of the repository for your submodule.

Here's a summary of how to reference a repository in your pipeline:

Remember to explicitly reference any Azure Repos Git repositories you want to use in the pipeline as a checkout step in the job that uses the repository. This is especially important when using a "Protect access to repositories in YAML pipelines" feature, which is enabled by default for new organizations and projects created after May 2020.

The branch you choose as the default when manually queuing a build is the one from which your build will get the latest sources. This branch has no bearing when the build is triggered through continuous integration (CI).

You'll usually set this to be the same as the default branch of the repository, such as "master". The default branch has no bearing when the build is triggered through continuous integration (CI).

You can control which branches get CI triggers with a simple syntax, specifying the full name of the branch or a wildcard. For example, you can use "main" or "releases/*".

Here's a breakdown of the syntax:

You cannot use variables in triggers, as variables are evaluated at runtime (after the trigger has fired). If you use templates to author YAML files, then you can only specify triggers in the main YAML file for the pipeline.

In the above example, the pipeline will be triggered if a change is pushed to main or to any releases branch. However, it won't be triggered if a change is made to a releases branch that starts with old.

Repos are a crucial part of any development project, and Azure Pipelines provides a robust way to manage them.

Repositories can access any Azure DevOps repositories in authorized projects unless Protect access to repositories in YAML pipelines is enabled.

You can clone multiple repositories in the same project as your pipeline by using multi-repo checkout.

To clone a repo from another project that is not public, you will need to authenticate as a user who has access to that project.

Use a secret variable to store credentials securely.

Secret variables are not automatically made available to scripts as environment variables.

For Azure Repos, you can use a personal access token with the Code (Read) permission.

You can specify the branches where you want to trigger builds using wildcard characters.

Here are the different options for specifying the repository:

If you prefer to use your own Git version rather than the one included with the Windows agent, set System.PreferGitFromPath to true.

To access Azure Repos Git repositories, you must explicitly reference them in your pipeline code, especially if "Protect access to repositories in YAML pipelines" is enabled. This setting is always true on non-Windows agents.

In YAML pipelines, you can reference Azure Repos Git repositories using a "uses" statement in the checkout stage. This is a critical step to avoid Git commands failing due to lack of access.

If you don't explicitly reference the repository, it behaves as if it has a "checkout: self" step, and the self repository has checked out.

Sync tags are a feature in Azure Repos Git that can be a real time-saver, but they also have some limitations.

The sync tags feature is supported in Azure Repos Git with Azure DevOps Server 2022.1 and higher.

To configure sync tags, you can set the fetchTags property in the Checkout step of your pipeline. This setting takes priority over the Sync tags setting configured in the pipeline settings UI.

You can also configure the Sync tags setting from the properties of the Get sources task in your pipeline.

If you have a large repository with many tags, syncing tags can increase the time to run the task in a pipeline.

To avoid this, you can configure the fetchTags setting to control the behavior of syncing tags.

Here are the steps to configure the Sync tags setting in YAML:

Using Git LFS is a great way to manage large files in your repository. Git LFS must be configured prior to checking out submodules, especially if a submodule contains LFS files.

If you're using a Microsoft-hosted macOS or Linux agent, you're in luck - it comes preconfigured with Git LFS. However, if you're using a Windows agent or a self-hosted macOS or Linux agent, you may need to take extra steps.

To work around this issue, you can add a step before checking out your submodules in your YAML file. This step will configure Git LFS, ensuring that your submodules can handle large files.

If you're using a Windows agent, you might want to consider using the version of Git that's already installed on the agent machine, rather than the one bundled with the agent software.

To do this, you can set a pipeline variable named System.PreferGitFromPath to true in your pipelines. This will allow the agent to use the version of Git that's installed on the machine.

Alternatively, you can create a file named .env in the agent root directory and add a System.PreferGitFromPath=true line to the file.

Here are the steps to set the pipeline variable:

By using the version of Git that's already installed on the agent machine, you can avoid any potential issues that might arise from using the bundled version.

In Azure Pipelines, the build process is divided into several stages, each with its own set of tasks. The build stage is the first stage in the pipeline and is responsible for compiling and building the code.

The build stage typically includes tasks such as resolving dependencies, compiling code, and packaging the application. The deploy stage is the final stage in the pipeline and is responsible for deploying the application to a production environment.

The deploy stage typically includes tasks such as deploying to Azure, deploying to a container registry, and configuring the application for production. The test stage is an optional stage that can be used to run automated tests on the application.

Cleaning the local repo on the agent can be a bit tricky, but it's essential to understand the implications. In general, for faster performance of your self-hosted agents, don't clean the repo.

Cleaning the repo isn't effective if you're using a Microsoft-hosted agent because you'll get a new agent every time. This means you won't have to worry about cleaning the repo in this case.

However, if you're using a self-hosted agent, you may need to clean the repo to avoid problems caused by residual files from a previous build. Depending on how your agent pools are configured, you may get a new agent for subsequent pipeline runs or stages, so not cleaning is not a guarantee that subsequent runs will be able to access outputs from previous runs.

You can use Build artifacts to share outputs of a pipeline run with subsequent runs, which is a good workaround for this issue.

To check out files, you'll want to add a checkout step to your pipeline. In Azure Pipelines, this is done by referencing the submodule repositories in the uses statement.

For example, if you have a repository with a submodule, you'll need to explicitly reference the submodule repository in the uses statement. This is because "Protect access to repositories in YAML pipelines" is enabled by default, which requires you to reference Azure Repos Git repositories in the pipeline.

You can do this by adding a uses statement to your checkout stage, like this: uses: repositories: - AzureFwChildPolFleetMgntID - AzureFwChildPolMarShipID.

If you're using large file storage (LFS) in your repository, you'll need to add a checkout step with lfs set to true. This will allow you to check out files from LFS.

Here's an example of how to do this: checkout: lfs: true.

Note that if you're using TFS or a self-hosted agent, you'll need to install git-lfs on the agent for this to work.

Adding a uses statement to your stage is a crucial step in pipeline configuration. You must explicitly reference the Azure Repo Git repositories of submodules to clone the code you want to use in the pipeline.

Protect access to repositories in YAML pipelines is enabled by default for new organizations and projects created after May 2020. This means your YAML pipelines must explicitly reference any Azure Repos Git repositories you want to use in the pipeline as a checkout step.

If your pipeline does not have an explicit checkout step, it behaves as if it has a "checkout: self" step and the self repository has checked out. However, using a script that provides authentication to the repo, such as a PAT, does not require a reference in a checkout step.

To explicitly reference a repository, use a uses statement like this: uses: repositories - AzureFwChildPolFleetMgntID-AzureFwChildPolMarShipID. This statement tells the pipeline to reference the specified repositories.

Here's a breakdown of the uses statement:

By adding a uses statement to your stage, you ensure that your pipeline can access the necessary repositories and clone the code required for the pipeline to run successfully.

Pipeline triggers are a crucial part of Azure Pipelines, allowing you to automate the build and deployment process. You can configure CI triggers to run a pipeline whenever you push an update to a specified branch or tag.

CI triggers are enabled by default for YAML pipelines, unless the Disable implied YAML CI trigger setting is enabled. To configure validation builds for an Azure Repos Git repository, you must be a project administrator of its project.

You can opt out of CI triggers entirely by specifying trigger: none in your YAML file. This will prevent the pipeline from running whenever someone checks in code.

Pull request (PR) triggers cause a pipeline to run whenever you open a pull request or push changes to it. In Azure Repos Git, this functionality is implemented using branch policies. To enable PR validation, navigate to the branch policies for the desired branch and configure the Build validation policy.

Here's a breakdown of how PR triggers work:

If you have an open PR and you push changes to its source branch, multiple pipelines may run. However, if at least one pushed commit contains [skip ci], the pipelines will not run.

Manual or scheduled triggers allow you to run a pipeline at a specific time or on demand. When triggered manually or scheduled, the values of the version, branch, and tags properties define the artifact version.

Resource pipeline completion triggers are used to trigger a pipeline when one of its resource pipelines completes. The artifacts version is the version of the triggering pipeline. The values of the version, branch, and tags properties are ignored.

Pipeline security and authorization are crucial aspects of Azure Pipelines, and understanding them can help you secure your resources and repositories. Pipelines run with collection scoped access tokens unless the Limit job authorization scope setting is enabled.

To limit the job authorization scope to the current project, you can enable the Limit job authorization scope to current project setting for YAML pipelines and classic build pipelines. This setting does not apply to classic release pipelines. If your Azure Repos Git repository is in a different project than your pipeline, you must grant permission to the build service identity for your pipeline to the second project.

The Limit job authorization scope setting allows you to reduce the scope of access for all pipelines to the current project, but it can impact your pipeline if you are accessing an Azure Repos Git repository in a different project in your organization. To avoid this, you can use the resource administration experience to authorize all pipelines to access the resource, or you can add a resource to a YAML file and authorize it on the failed build.

Repository Access Protection is a crucial aspect of Pipeline Security and Authorization. Pipelines can access any Azure DevOps repositories in authorized projects, unless certain settings are enabled.

Protecting access to repositories in YAML pipelines is a key feature. With this option enabled, pipelines must explicitly reference any Azure Repos Git repositories used in the pipeline as a checkout step or uses statement.

Pipelines run with collection scoped access tokens unless the relevant setting for the pipeline type is enabled. This setting allows you to reduce the scope of access for all pipelines to the current project.

To configure this setting, navigate to Pipelines, Settings at either Organization settings or Project settings. If enabled at the organization level, the setting is grayed out and unavailable at the project settings level.

There are exceptions where you don't need to explicitly reference an Azure Repos Git repository before using it in your pipeline. These exceptions include:

Resource authorization in YAML pipelines is also a critical aspect. Resources must be authorized before they can be used in pipelines. Resource owners control the users and pipelines that can access their resources.

There are several ways to authorize a YAML pipeline to use resources. These include:

Approval checks are a powerful tool for controlling when a resource runs in your pipeline. They can be set up to require a specific YAML template, which helps ensure that resources are used only under certain conditions and enhances security.

You can use approval checks to manually control when a resource runs, and templates to require that any pipeline using a resource or environment extends from a specific YAML template. This is a key part of enhancing pipeline security with templates.

Setting a required template approval ensures that your resource is used only under specific conditions, and enhances security. This is a crucial step in protecting your pipeline from unauthorized access.

By using approval checks and templates, you can take control of who has access to your resources and environments, and when they can be used. This helps prevent unauthorized changes to your pipeline.

You can opt out of CI triggers entirely by specifying trigger: none. This is a straightforward way to prevent your pipeline from running automatically whenever someone checks in code.

If you want to disable CI triggers for a specific pipeline, you can do so by editing the YAML file in the pipeline's branch. Simply add `trigger: none` to the YAML file and you're good to go.

Here are some scenarios where you might want to disable CI triggers:

Remember, disabling CI triggers doesn't mean you can't run the pipeline manually. You can still trigger the pipeline to run whenever you need to.