A Comprehensive Guide to Azure Telemetry Best Practices

As you start implementing Azure Telemetry, it's essential to follow best practices to ensure you're getting the most out of your data. Azure Telemetry is designed to collect and analyze data from your Azure resources, but without proper setup, you might end up with a mess of unorganized data.

To avoid this, it's crucial to understand the importance of data context, which is discussed in more detail in the "Data Context" section. By including relevant metadata, you can easily identify and troubleshoot issues.

A good starting point is to set up Azure Monitor, which provides a centralized platform for collecting and analyzing telemetry data. According to the "Azure Monitor" section, this service can help you monitor your resources in real-time, identify potential issues, and optimize your Azure environment for better performance.

Instrumentation is code added to an application that allows us to monitor the application’s performance, debug errors, track metrics, and write trace information.

Instrumenting your application is helpful because it provides more insight into how the application or service is functioning, beyond a simple healthbeat response code.

Instrumentation becomes crucial in distributed systems to track requests processing through the system.

To add the Application Insights client library to our application, we can install the Microsoft.ApplicationInsights nuget package from the Package Manager window or console.

The library provides a thread-safe wrapper class around the calls to send the instrumentation (telemetry) data to Azure.

We can use the TelemetryClient class to track the various types of instrumentation data we want to send, such as sending metrics with TrackMetric(), or exceptions with TrackException().

The TelemetryClient class is thread-safe and exposes different methods to track instrumentation data.

Creating a new TelemetryClient instance only makes sense if it needs a configuration that's separate from the rest of the telemetry.

A singleton instance of TelemetryClient is already registered in the DependencyInjection container, which shares TelemetryConfiguration with the rest of the telemetry.

Azure telemetry collection is a powerful tool that helps you monitor your app's performance, health, and usage. Most Application Insights data has a latency of under 5 minutes, although some data can take longer, especially for larger log files.

You can select the location when you create a new Application Insights resource, which is a great feature. Application Insights collects telemetry about your app, including web server telemetry, web page telemetry, and performance counters.

Here are some examples of what you can collect with Azure telemetry:

This data can be used to monitor your app's performance, health, and usage, which is super helpful for troubleshooting and optimization.

Most data has a latency of under 5 minutes, which is quite impressive considering the vast amounts of data being collected.

Some data can take longer to collect, especially larger log files, which is a common occurrence.

Latency of under 5 minutes is typical for most Application Insights data.

To track telemetry that's not automatically collected, you can get an instance of TelemetryClient using constructor injection and call the required TrackXXX() method on it.

A singleton instance of TelemetryClient is already registered in the DependencyInjection container, which shares TelemetryConfiguration with the rest of the telemetry.

Creating a new TelemetryClient instance is only recommended if it needs a configuration that's separate from the rest of the telemetry.

You can use the GetMetric API to send custom metrics to Application Insights, similar to how you can track more telemetry from a controller.

In Application Insights Agent 2.0.0-beta1 and later, ASP.NET Core applications hosted in IIS are supported.

Data Management is crucial for Azure telemetry. You can choose the retention period for raw data, from 30 to 730 days. Aggregated data is retained for 90 days, and debug snapshots are retained for 15 days.

Azure Monitor pipeline at edge is a powerful solution designed to facilitate high-scale data ingestion and routing from edge environments to seamlessly enable observability across cloud, edge, and multicloud. It uses the OpenTelemetry Collector. Currently, in public preview, it can be deployed on a single Arc-enabled Kubernetes cluster, and it can collect OpenTelemetry Protocol (OTLP) logs.

The OpenTelemetry Collector Azure Data Explorer Exporter is a data exporter component that can be plugged into the OpenTelemetry Collector. It supports ingestion of data from many receivers into Azure Data Explorer, Azure Synapse Data Explorer, and Real-Time Analytics in Fabric.

Collection is a crucial aspect of data management, and it's essential to understand how telemetry is collected and processed.

Application Insights collects telemetry about your app, including web server telemetry, web page telemetry, and performance counters. This data can be used to monitor your app's performance, health, and usage.

You can select the location when you create a new Application Insights resource, giving you flexibility in where your data is stored.

Most Application Insights data has a latency of under 5 minutes, but some data can take longer, especially for larger log files.

Here's a breakdown of what data is collected:

This data is collected through various means, including the OpenTelemetry SDK and instrumentation libraries, which are bundled together in the Azure Monitor OpenTelemetry Distro.

Data is sent to an Application Insights Log Analytics workspace, where you can choose the retention period for raw data, from 30 to 730 days.

You can choose the retention period for raw data, which ranges from 30 to 730 days.

Aggregated data is retained for a fixed 90 days.

Debug snapshots are retained for a relatively short 15 days.

Monitoring is a crucial aspect of Azure Telemetry. You can monitor a wide range of aspects of your application and trigger various actions with alerts. Alerts can be configured to monitor specific metrics, logs, and diagnostic settings.

Application Insights allows you to dive deep into metrics data to understand usage patterns and trends. You can also configure streaming export of platform logs and metrics to the destination of your choice.

To confirm data is flowing, run your application and then open Application Insights in the Azure portal. It might take a few minutes for data to show up, but once it does, you can start analyzing your application's performance.

Some key metrics to monitor include Average Response Time, which is available as an SLI for Azure App Service. There are over 30 SLIs available for Azure App Service, so you don't have to manually calculate and report on them.

Here are some ways to monitor your application:

By monitoring your application, you can quickly identify issues and make data-driven decisions to improve its performance.

Performance is a crucial aspect of Azure Telemetry, and understanding how it works is essential for optimizing your application's performance.

SDK versions 2.4.1 and later collect performance counters if the application is running in Web Apps (Windows). This is a significant improvement, as it allows for more detailed monitoring of your application's performance.

You can also collect performance counters if your application targets netstandard2.0 or later, as long as it's running on Windows with SDK versions 2.7.1 and later.

However, if your application targets the .NET Framework, all versions of the SDK support performance counters.

To get a better understanding of your application's performance, it's essential to know the specific counters that are supported. Here's a breakdown of the supported counters:

Sampling is an essential aspect of Azure Telemetry, allowing you to collect data from your ASP.NET Core application.

The Application Insights SDK for ASP.NET Core supports both fixed-rate and adaptive sampling. By default, adaptive sampling is enabled.

Adaptive sampling is a great feature, as it allows your application to dynamically adjust the sampling rate based on the available resources.

For more information on configuring adaptive sampling, see Configure adaptive sampling for ASP.NET Core applications.

You can customize the Application Insights SDK for ASP.NET Core in the ConfigureServices() method of your Startup.cs class. This is where you'll make most of your configuration changes, unless you're directed otherwise.

The default configuration can be changed, but be aware that modifying TelemetryConfiguration.Active isn't supported in ASP.NET Core applications.

If you're building a Worker Service application, you won't be able to use the Application Insights SDK. Instead, you should use Application Insights for Worker Service applications, which are non-HTTP applications.

Adding Application Insights to your project creates files and adds code to some of your files. Be sure to manually delete the added code or files along with any API calls you added in your project if you decide to fully remove Application Insights.

Application Insights supports ASP.NET Core 3.1, but only with the SDK version 2.8.0 and Visual Studio 2019 or later.

You'll need to plan carefully when it comes to deploying Application Insights resources. To determine the right number of resources, check out the Application Insights deployment planning guide.

To manage your Application Insights resources efficiently, consider using PowerShell. With PowerShell, you can create and update resources, set the pricing plan, and even get the instrumentation key.

Here are some specific tasks you can perform using PowerShell:

When deploying Application Insights resources, it's essential to understand the number of resources required to cover your application or components across environments.

To determine the right number of resources, check out the Application Insights deployment planning guide.

You'll need to have a deployed instance of the Azure Application Insights service. Deploy one to your Azure subscription if you haven’t already.

Taking the infrastructure-as-code route and deploying your Application Insights resource via Azure Resource Manager (ARM) templates is a recommended approach. Instructions can be found online.

After deployment, take note of the instrumentation key, which can be retrieved through code by calling the Azure APIs or viewed from the Azure Portal directly on the overview tab of the resource.

The instrumentation key is not considered a secret value, so it's safe to check this value into source control and/or use it as a configuration value in public client applications.

Managing resources with PowerShell is a powerful way to streamline your work. You can create and update Application Insights resources with just a few commands.

To start, you'll need to create and update Application Insights resources using PowerShell. This will help you get a handle on your resources and set them up correctly.

With PowerShell, you can set the pricing plan for your Application Insights resources. This is an important step, as it will affect how much you pay for your resources.

You can also use PowerShell to get the instrumentation key for your Application Insights resources. This key is essential for tracking and monitoring your application.

Adding a metric alert is another important task that can be done with PowerShell. This will help you stay on top of any issues that may arise with your application.

Finally, you can use PowerShell to add an availability test to your Application Insights resources. This will give you valuable insights into how your application is performing.

Here are the key tasks you can perform with PowerShell:

Application Insights is billed through the Log Analytics workspace into which its log data is ingested. The default Pay-as-you-go Log Analytics pricing tier includes 5 GB per month of free data allowance per billing account.

You can learn more about Azure Monitor logs pricing options.

When handling sensitive data, it's essential to keep it secure. Application Insights doesn't handle sensitive data by default.

Avoid putting sensitive data in URLs as plain text to prevent unauthorized access. This is a good practice to follow, especially when dealing with personal or other sensitive details.

During development and testing, it's crucial to check the sent data in your IDE and browser's debugging output windows to catch any potential issues.

For long-term data, consider archiving it according to the guidelines outlined in Data collection, retention, and storage in Application Insights.

Troubleshooting is an essential part of any app's maintenance, and Azure Telemetry provides some fantastic tools to help you identify and fix issues.

To get started, explore user flows to understand how users move through your app, as this can help you pinpoint where problems are occurring.

By configuring a snapshot collection, you can see the state of your source code and variables at the exact moment an exception is thrown, giving you valuable insight into what went wrong.

To further troubleshoot, use the API to send your own events and metrics for a detailed view of your app's performance and usage.

Availability tests can also be used to constantly check your app from around the world, helping you identify any issues that may be specific to certain regions.

Here are some specific steps to take: