Mastering Azure Pipelines Stages and Pipeline Execution

Azure Pipelines stages are a powerful way to organize and execute complex workflows. Stages can be thought of as a series of steps that are executed in a specific order.

Each stage in Azure Pipelines can contain multiple jobs, which are executed independently of each other. This allows for a high degree of flexibility and control over the pipeline execution process.

There are three types of stages in Azure Pipelines: Deploy, Test, and Build. The Build stage is responsible for compiling and packaging the code, the Test stage runs automated tests, and the Deploy stage deploys the application to production.

The order in which stages are executed can be customized to fit the specific needs of your pipeline. This is achieved through the use of stage dependencies, which allow you to specify the order in which stages should be executed.

In Azure DevOps, you can create a pipeline using either YAML or the UI editor. YAML pipelines are 100% code-based, allowing for reusability and validation through pull requests.

For more insights, see: Azure Devops Pipeline Yaml

To create a YAML pipeline, you'll need to add a YAML file to your project, typically named azure-pipelines.yaml. This file will serve as the foundation for your pipeline configuration.

You can configure the pipeline to trigger on specific events, such as pushing commits to the main branch. This is done using the trigger keyword in the YAML file.

The pool keyword is used to select the environment where your pipeline will run. In the example, the author chose ubuntu-latest, which is faster and aligns with their Linux-based app service.

A pipeline is composed of stages, which are collections of jobs that can be run concurrently or sequentially. For a simple project, you might define two stages: Build and Test.

Here's a breakdown of the pipeline configuration elements:

By understanding these configuration elements, you'll be well on your way to creating and customizing your own Azure pipelines.

You can manually trigger a YAML pipeline stage to have more control over when it runs.

This feature is useful for stages that require human intervention or approval, such as production deployment.

To use this feature, add the trigger: manual property to a stage.

For example, you might have a pipeline with stages for building, testing, and deploying to production, but only want the production stage to run manually.

The production stage runs a "hello world" output script, just like the development stage, but it requires a manual trigger to start.

This allows you to have a unified pipeline that can be run automatically, except for the stages that require manual intervention.

You can add the trigger: manual property to a stage to enable this feature.

By doing so, you can have more control over when each stage runs and ensure that critical stages, like production deployment, are only triggered when ready.

For more insights, see: Azure Pipelines Trigger

When defining a stage in Azure Pipelines, you'll need to specify the stage ID as the first property. This ID is used to identify the stage.

For more insights, see: How to See How Deploy a Stage on Azure Ado

The displayName property is optional but useful for giving your stage a human-readable name, making it easier to understand the pipeline's flow.

You can specify any stages that must complete before this one using the dependsOn property, which can be a single string or a list of strings. This allows for more complex pipeline dependencies.

To control lock requests from this stage in relation to other exclusive lock requests, you can set the lockBehavior property to either sequential or runLatest. sequential will block other lock requests, while runLatest will cancel any existing runLatest builds for the same branch.

If you want to prevent a stage from being skipped, you can set the isSkippable property to false. By default, stages are skippable, allowing you to skip them if needed.

Specifying an exclusive lock at the stage level can help ensure only one pipeline run can access a stage at a time, which is useful for stages that deploy to shared resources like Azure resource groups.

When you run the pipeline for the first time, you'll receive an email to approve the deployment.

You'll see a detailed view of the pipeline execution, including all stages, jobs, and steps.

The pipeline will continue to run after approval, executing all stages, jobs, and steps as it's the first run.

On subsequent pipeline triggers, you'll notice that stages, jobs, and steps are skipped, except for checking if the application has already been deployed.

This optimization is a result of the pipeline learning from its previous execution.

Pipeline stages are a crucial part of Azure Pipelines, allowing you to break down your build, test, and deployment processes into separate stages.

You can specify stages in your pipeline by using the "stages" keyword, and each stage can have its own pool, jobs, and steps. For example, in a deployment stage, you can have a job with two tasks: one to download the artifact and another to deploy it to Azure.

Stages can also have dependencies, which control the order in which they run. By default, stages run sequentially, but you can use triggers to specify when a stage should start. For instance, you can use an "After stage" trigger to start a stage after all the dependent stages have completed.

Here are some key facts about pipeline stages:

By using pipeline stages effectively, you can create more efficient and organized build, test, and deployment processes in Azure Pipelines.

Pipelines must contain at least one stage with no dependencies. This ensures that there's always a clear starting point for the pipeline.

You can control the dependencies by setting triggers on each stage of the release pipeline. Triggers are what determine when a stage runs.

Stages run with a trigger or by being manually started. This means you have control over when each stage begins.

With an After release trigger, a stage starts as soon as the release starts, in parallel with other stages that have the same trigger. This allows you to run multiple stages simultaneously.

Worth a look: Azure Release Pipeline

Using an After stage trigger, a stage will start after all the dependent stages complete. This is useful for modeling fan-out and fan-in behavior for stages.

Here's a breakdown of the different types of triggers:

By understanding how to specify dependencies, you can create more efficient and effective pipelines that meet your needs.

A YAML pipeline is a powerful tool for automating your deployment process. You can create a YAML pipeline in Azure DevOps using the YAML editor or the UI editor, but YAML pipelines have some advantages, such as being 100% code-based and allowing you to reuse snippets.

To create a YAML pipeline, you'll need to configure the trigger, pool, and stages. The trigger specifies when the pipeline should be triggered, such as when pushing commits to the main branch. In my case, I used the "trigger" keyword to trigger the pipeline on the main branch.

The pool specifies the environment where the pipeline will run. You can choose from various options, such as "ubuntu-latest" or Windows. I selected "ubuntu-latest" because it's faster and I'm using Linux for my app service.

A fresh viewpoint: How to Create Azure Pipeline with Yaml

A YAML pipeline consists of stages, jobs, and steps. Stages are collections of jobs, and jobs are collections of steps. Steps can be scripts or predefined tasks from Azure DevOps. Each stage can have its own trigger, pool, and conditions.

Here are the common triggers for stages:

You can also specify conditions for each stage, such as forcing a stage to run even if a previous stage fails or specifying a custom condition. If you customize the default condition of the preceding steps for a stage, you remove the conditions for completion and success.

A different take: Azure Pipelines or Condition

YAML pipelines don't support queuing policies, which means each run is independent and unaware of other runs. This can be a problem if you have successive commits that trigger multiple pipelines, and you need to control the order of execution.

If you need to sequence and control the order of execution, you can use manual approvals. This allows you to manually control when a stage should run, which is especially useful for deployments to production environments.

Manual approvals are supported on environments, and you can define checks that are required before a stage can start consuming a resource. This gives you more control over the pipeline and can help prevent errors.

Queuing policies give you control over how multiple releases are queued into a stage. You can choose from three options: Number of parallel deployments, Deploy all in sequence, and Deploy latest and cancel the others.

Here's a breakdown of the options:

For example, if you have five releases (R1-R5) and you specify a limit of 3 parallel deployments, the first three releases will be deployed in parallel, and the remaining two will be deployed one after the other.

Specify approvals can be a game-changer in controlling deployments to production environments.

You can manually control when a stage should run using approval checks, which are a mechanism available to the resource owner to control if and when a stage in a pipeline can consume a resource.

Curious to learn more? Check out: Azure Resource

Approval checks are commonly used to control deployments to production environments, and you can add them at the start or end of each stage in the pipeline.

These checks are a way to define what must be satisfied before a stage consuming a resource can start, and as an owner of a resource like an environment, you can define checks that are required before a stage that consumes the resource can start.

For more information, see Approvals, and remember that manual approval checks are currently supported on environments.

A pipeline is a sequence of stages that run in a specific order. You can create a pipeline using YAML or the UI editor in Azure DevOps.

You can specify the stages in your pipeline using the stages keyword. If you don't need any logical boundaries in your pipeline, you can directly specify the jobs in your YAML file.

To add a stage to your release pipeline, select the release pipeline in the Releases page, select the action to Edit it, and then select the Pipeline tab. You can configure several properties and options for a stage in a release pipeline, such as editing the name, designating a stage owner, deleting the stage, changing the order of stages, saving a copy of the stage as a stage template, and managing security settings.

Worth a look: Implementing Ci/cd Using Azure Pipelines Pdf

You can also specify queuing policies to control how multiple releases are queued into a stage. However, YAML pipelines don't support queuing policies, so you may need to use manual approvals to manually sequence and control the order of execution.

Here are the queuing policy options:

You can also specify dependencies between stages using the dependsOn keyword. This allows you to control the order in which stages run and can help you model fan-out and fan-in behavior for stages.

Here are some examples of stages that run sequentially and in parallel:

By specifying the stages and dependencies in your pipeline, you can create a more complex and flexible deployment process that meets the needs of your organization.

The build stage is pretty simple. We first define a stage with the name 'build', and inside we'll add a job with the name 'build'. This job will have 3 steps: dotnet build, dotnet publish, and Publish pipeline artifact.

Consider reading: Azure Devops Pipeline Publish Artifacts

The first two steps are pretty straightforward. We use the dotnet SDK to build our app, and then we use the 'dotnet publish' command to publish the contents in a folder defined by the variable $(build.artifactStagingDirectory).

A key part of the build stage is publishing the contents of the folder to a location. In this example, the location is named 'pipeline' and the artifact name is 'drop'. You can use whatever you want, but for simplicity, the same names were used.

Once the publish is completed, we have to upload the contents of the folder to a location. To do this, we use the Publish pipeline artifact step.

Here are the three steps of the build job in detail: