Azure OpenTelemetry Fundamentals and Best Practices

Azure OpenTelemetry is a powerful tool for monitoring and analyzing distributed systems. It helps you understand the performance and behavior of your applications by providing a unified way to collect and export telemetry data.

OpenTelemetry is an open-source project that provides a standard for collecting and exporting telemetry data from applications. This standard is adopted by Azure Monitor and other monitoring tools, making it easy to integrate with your existing infrastructure.

To get started with Azure OpenTelemetry, you'll need to install the OpenTelemetry SDK for your programming language of choice. This will allow you to instrument your application with OpenTelemetry APIs and export telemetry data to Azure Monitor.

The OpenTelemetry SDK provides a set of APIs that you can use to instrument your application, including support for tracing, metrics, and logs. By using these APIs, you can collect and export telemetry data in a standardized way that's easy to understand and analyze.

First, you'll need to have an Azure subscription, which you can create for free.

To use Azure OpenTelemetry, you'll also need to have Azure Monitor set up, specifically application insights, which can be found in the Azure Monitor documentation.

Next, you'll need to install the Opentelemetry SDK for Python, which is available for download.

Make sure you have Python 3.8 or later installed on your system, as this is the minimum required version.

Instrumentation and data collection are crucial components of Azure OpenTelemetry. The Azure Monitor OpenTelemetry Distro automatically collects data by bundling OpenTelemetry instrumentation libraries.

You can also collect more data automatically by including instrumentation libraries from the OpenTelemetry community. However, be aware that community libraries are not officially supported and may introduce breaking changes in the future.

To add a community library, use the ConfigureOpenTelemetryMeterProvider or ConfigureOpenTelemetryTracerProvider methods after adding the NuGet package for the library. The list of community instrumentation libraries can be found here.

The following instrumentation libraries are available for Azure OpenTelemetry:

To install the Azure Monitor Opentelemetry Distro, you'll need to use pip. Install the package using pip, and you're good to go.

The command to install the package is straightforward: use pip to install the Azure Monitor Opentelemetry Distro.

Automatic data collection is a powerful feature that can save you time and effort. The Azure Monitor OpenTelemetry Distro automatically collects data by bundling OpenTelemetry instrumentation libraries.

You can collect more data automatically when you include instrumentation libraries from the OpenTelemetry community. However, be aware that community libraries are not officially supported, and some might introduce future breaking changes.

To add a community library, you'll need to use the ConfigureOpenTelemetryMeterProvider or ConfigureOpenTelemetryTracerProvider methods after adding the NuGet package for the library.

Here are some community instrumentation libraries you can consider:

Just remember that you can't use community instrumentation libraries with GraalVM Java native applications.

The OpenTelemetry API offers a way to collect custom telemetry from your application, allowing you to gather data that's not automatically collected by the OpenTelemetry Instrumentation Libraries.

You can collect custom telemetry using the OpenTelemetry API, language-specific logging/metrics libraries, or the Application Insights Classic API.

The table below shows the currently supported custom telemetry types for different languages.

<

Sampling is a great way to reduce your data ingestion volume, which in turn reduces your cost.

You can enable sampling in Azure Monitor with a custom fixed-rate sampler that populates events with a sampling ratio.

The sampler is designed to preserve your traces across services and is interoperable with older Application Insights Software Development Kits (SDKs).

Metrics and Logs are unaffected by sampling.

The sampler expects a sample rate of between 0 and 1 inclusive, where a rate of 0.1 means approximately 10% of your traces are sent.

Rate-limited sampling is available starting from 3.4.0 and is now the default.

For Spring Boot native applications, the sampling configurations of the OpenTelemetry Java SDK are applicable.

The OTEL_TRACES_SAMPLER_ARG environment variable can be used to specify the sampling rate, with a valid range of 0 to 1.

A value of 0.1 means 10% of your traces are sent.

If you're unsure what to set the sampling rate as, start at 5% (i.e., 0.05 sampling ratio) and adjust the rate based on the accuracy of the operations shown in the failures and performance panes.

A higher rate generally results in higher accuracy.

However, any sampling will affect accuracy, so it's recommended to alert on OpenTelemetry metrics, which are unaffected by sampling.

Azure OpenTelemetry offers a range of advanced features to help you monitor and optimize your applications.

With Azure OpenTelemetry, you can use distributed tracing to analyze the performance of your microservices-based applications. This allows you to gain a deeper understanding of how your application is performing and identify bottlenecks.

One of the key features of Azure OpenTelemetry is its ability to handle high cardinality data, which means it can handle a large number of unique values without impacting performance. This is especially useful for applications with a large number of users or transactions.

Custom exceptions are a powerful tool for developers to manually report exceptions beyond what instrumentation libraries report. This allows you to draw attention to failures in relevant experiences, including the failures section and end-to-end transaction views.

You can log custom exceptions using an Activity or ILogger. For example, you can start a new activity named "ExceptionExample" and set the activity status to "Error" when an exception is thrown.

To log an exception using an Activity, you can use the following code:

```

using (var activity = activitySource.StartActivity("ExceptionExample"))

{

try

{

throw new Exception("Test exception");

}

catch (Exception ex)

{

activity?.SetStatus(ActivityStatusCode.Error);

activity?.RecordException(ex);

}

}

```

Similarly, you can log an exception using ILogger:

```

var logger = loggerFactory.CreateLogger("ExceptionExample");

try

{

throw new Exception("Test Exception");

}

catch (Exception ex)

{

logger.Log(

logLevel: LogLevel.Error,

eventId: 0,

exception: ex,

message: "Hello {name}.",

args: new object[] { "World" });

}

```

You can also use opentelemetry-api to update the status of a span and record exceptions. To do this, you need to add opentelemetry-api-1.0.0.jar (or later) to your application and set the status to error and record an exception in your code.

Here are the steps to record exceptions manually using opentelemetry-api:

```

import io.opentelemetry.api.trace.Span;

import io.opentelemetry.api.trace.StatusCode;

Span span = Span.current();

span.setStatus(StatusCode.ERROR, "errorMessage");

span.recordException(e);

```

By following these steps, you can manually report exceptions and improve the visibility of failures in your application.

Custom Spans are a powerful feature that allows you to add custom spans to your application's telemetry data.

You can add custom spans using the OpenTelemetry API or by applying the @WithSpan annotation. The simplest way to add your own spans is by using OpenTelemetry's @WithSpan annotation, which will automatically populate the requests and dependencies tables in Application Insights.

By default, the span will end up in the dependencies table with dependency type InProc, but you can change this by setting the attribute kind = SpanKind.SERVER to ensure it appears in the Application Insights requests table.

To use the OpenTelemetry API, you'll need to inject OpenTelemetry, create a Tracer, and then create a span, make it current, and end it. This will allow you to add custom spans that appear in the requests and dependencies tables in Application Insights.

Here's a summary of the steps to add custom spans:

By using custom spans, you can gain a deeper understanding of your application's behavior and performance, and make data-driven decisions to improve it.

You can export telemetry data from Azure OpenTelemetry using the OpenTelemetry exporter.

The Azure monitor OpenTelemetry Exporter configurations can be passed directly into configure_azure_monitor.

To export telemetry data, you can use the OpenTelemetry exporter, which allows you to export data to Azure Monitor.

Azure monitor OpenTelemetry Exporter configurations are related to exporting, and you can find additional information here.

Azure Monitor's OpenTelemetry distro comes with a set of officially supported instrumentations that allow for automatic collection of telemetry data. These instrumentations are enabled by default, but can be opted-out of if needed.

The officially supported instrumentations include Azure Core Tracing OpenTelemetry, OpenTelemetry Django Instrumentation, OpenTelemetry FastApi Instrumentation, and several others. These instrumentations are listed in the table below:

These instrumentations are not only enabled by default but are also included in the Azure Monitor Distro client library for Python, making it easy to get started with telemetry collection.

Let's take a look at the officially supported instrumentations that come bundled with the Azure monitor distro. These instrumentations are enabled by default, but you can opt-out if needed.

By default, the following instrumentations are enabled: Azure Core Tracing OpenTelemetry, OpenTelemetry Django Instrumentation, OpenTelemetry FastApi Instrumentation, OpenTelemetry Flask Instrumentation, OpenTelemetry Psycopg2 Instrumentation, OpenTelemetry Requests Instrumentation, OpenTelemetry UrlLib Instrumentation, and OpenTelemetry UrlLib3 Instrumentation.

Here's a list of the supported instrumentations, including their corresponding library names and supported versions:

If you need to add support for another OpenTelemetry instrumentation, you can submit a feature request.

The Azure Monitor Opentelemetry Distro client library for Python is a game-changer for instrumenting your application. It's a "one-stop-shop" solution that requires only one line of code to get started.

This library captures telemetry via OpenTelemetry instrumentations and reports it to Azure Monitor via the Azure Monitor exporters. Prior to using this SDK, make sure you understand Data Collection Basics, especially the section on telemetry types.

To get the most out of this library, you'll need to understand the mapping of OpenTelemetry terminology to Application Insights terminology. This will help you navigate the different telemetry types and how they're reported.

Here are some key features of the Azure Monitor OpenTelemetry Distro client library for Python:

By using this library, you'll be able to easily instrument your application and start collecting valuable telemetry data.

The Azure Monitor OpenTelemetry package bundles a series of OpenTelemetry and Azure Monitor components to enable the collection and sending of telemetry to Azure Monitor.

Manual instrumentation can be achieved using the configure_azure_monitor function, while automatic instrumentation is not yet supported. This is a key concept to understand when working with Azure Monitor OpenTelemetry.

To configure Azure Monitor OpenTelemetry, you can pass parameters directly into the configure_azure_monitor function, which takes priority over environment variables. This includes connection string, enable live metrics, logger name, instrumentation options, resource, span processors, and views.

Some key environment variables to be aware of when configuring OpenTelemetry include OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_LOGS_EXPORTER, OTEL_METRICS_EXPORTER, and OTEL_TRACES_EXPORTER, which can be used to customize telemetry collection and export.

Here are the OpenTelemetry environment variables that can be used to configure Azure Monitor OpenTelemetry:

You can add custom properties to your telemetry using span attributes. This is useful for adding extra information to your spans that might not be captured by autoinstrumentation.

Application Insights uses attributes to set optional fields in its schema, like Client IP. To set the user IP, you can use the custom property example, but replace the lines of code with the following: `span.setAttribute("client_ip", "192.168.1.100");`. This will populate the client_IP field for requests.

You can also use attributes to set the kind of span. For example, if your method represents a background job not already captured by autoinstrumentation, you should set the attribute kind = SpanKind.SERVER to ensure it appears in the Application Insights requests table. This is because Application Insights maps ActivityKind.Server and ActivityKind.Consumer to requests.

Here are some common span attributes and their uses:

By using span attributes, you can add extra context and information to your spans, making it easier to understand and analyze your application's behavior.

Configurations are a crucial part of setting up Azure Monitor OpenTelemetry. You can access OpenTelemetry configurations through environment variables while using the Azure Monitor OpenTelemetry Distros.

The APPLICATIONINSIGHTS_CONNECTION_STRING environment variable should be set to the connection string for your Application Insights resource. This is a must-have for Azure Monitor OpenTelemetry to work properly.

You can also set the APPLICATIONINSIGHTS_STATSBEAT_DISABLED environment variable to true to opt out of internal metrics collection. This is a good option if you want to disable metrics collection for your application.

Resource attributes are key-value pairs that can be used to specify the OpenTelemetry Resource associated with your application. You can set the OTEL_RESOURCE_ATTRIBUTES environment variable to a dictionary of key-value pairs to customize your resource attributes.

The OTEL_SERVICE_NAME environment variable sets the value of the service.name resource attribute. If service.name is also provided in OTEL_RESOURCE_ATTRIBUTES, then OTEL_SERVICE_NAME takes precedence.

Here are some common OpenTelemetry environment variables:

The configure_azure_monitor function also supports additional configuration parameters, including connection_string, enable_live_metrics, logger_name, instrumentation_options, resource, span_processors, and views.

This package bundles OpenTelemetry and Azure Monitor components to enable telemetry collection and sending to Azure Monitor. For manual instrumentation, use the configure_azure_monitor function.

AUTOMATIC instrumentation is not yet supported. You'll need to use the manual method for now.

The Azure Monitor OpenTelemetry exporters are the main components for accomplishing this. They can be used directly through this package.

To understand how OpenTelemetry and Azure Monitor components work, please go to the exporter documentation.