Configure and Use Azure Diagnostic Settings for Better Insights

Azure diagnostic settings are a powerful tool for gaining deeper insights into your Azure resources. They allow you to collect diagnostic data from your resources and store it in a designated location for later analysis.

To get started with Azure diagnostic settings, you'll need to create a diagnostic setting for each resource that you want to monitor. This can be done through the Azure portal, where you can select the resource and create a new diagnostic setting.

Diagnostic settings can be configured to collect a wide range of data, including metrics, logs, and performance data. By collecting these types of data, you'll be able to identify trends and patterns in your resource usage that can inform your decision-making.

You can choose to send the collected data to a variety of destinations, including Azure Storage, Azure Monitor, and third-party services.

To configure your Azure diagnostic settings, you can choose from several options. You can add a diagnostic setting and configure log data collection by selecting the logs you wish to monitor, such as Azure AD activity logs.

You can also use Resource Manager template samples for diagnostic settings in Azure Monitor, which allow you to establish a diagnostic setting for an Azure resource. This can be useful for collecting authorized logs and metrics for different resource types.

To enable logging, you can use the Azure portal, Azure Storage, or follow specific steps for your Azure resource, such as your Azure CDN endpoint.

Resource-specific logs have several advantages over Azure diagnostics logs, including making it easier to work with the data in log queries, providing better discoverability of schemas and their structure, improving performance across ingestion latency and query times, and providing the ability to grant Azure role-based access control rights on a specific table.

Resource-specific logs are created for each log category selected in the diagnostic setting, allowing for more granular control over data collection and management.

You can select the collection mode for logs in the diagnostic setting, choosing between Azure diagnostics or resource-specific mode. Most Azure resources write data to the workspace in one of these modes without giving you a choice.

For new diagnostic settings, it's recommended to specify resource-specific mode, as this mode makes the data easier to manage and might help you avoid complex migrations later.

If you modify an existing diagnostic setting to resource-specific mode, data that was already collected remains in the AzureDiagnostics table until it's removed according to your retention setting for the workspace. New data is collected in the dedicated table.

To query data across both tables, you can use the union operator.

Here are the two category groups available for resource logs:

The "Audit" category group is a subset of the "All" category group, but the Azure portal and REST API consider them separate settings.

To enable logging in Azure, you can follow these steps. Firstly, sign in to the Azure portal to access your resources. From there, navigate to All resources and select your CDN profile.

To enable diagnostics logs for your CDN endpoint, select the endpoint you want to monitor and then click on Diagnostics logs in the Monitoring section. You can also enable logging with Azure Storage by selecting Archive to a storage account and choosing CoreAnalytics.

When configuring diagnostic settings, you'll need to enter a name for your settings and select the log categories you want to monitor. You can also choose the log destination type, such as a Log Analytics workspace. To store logs in a storage account, select the subscription and storage account for the logs.

Here are the steps to enable logging with Azure Storage:

Remember to name your new diagnostic setting and select the logs you want to monitor. You can also check the checkbox beside the log destination type you wish to send your logs to.

You have several options when it comes to sending your Azure platform logs and metrics to different destinations.

Azure Monitor logs can be sent to a Log Analytics workspace, which allows you to correlate resource log data with other monitoring data and perform complex analysis.

You can also send your logs to Event Hubs or Azure Storage, which can be useful for real-time processing or long-term storage.

Here are the current destinations available for sending logs:

Each destination has its own unique features and use cases, so it's worth considering which one best fits your needs.

A Log Analytics workspace is a crucial component for storing and analyzing logs from various Azure resources.

You can send resource logs to a Log Analytics workspace to enable Azure Monitor Logs features. This allows you to correlate resource log data with other monitoring data, consolidate log entries from multiple resources, and perform complex analysis.

Two types of collection modes for resource logs exist: Azure diagnostics and resource-specific. Azure diagnostics writes all data to the AzureDiagnostics table, while resource-specific writes data to individual tables for each resource category.

There are four options for destinations when creating a diagnostic setting for Azure AD audit logs: Send to Log Analytics workspace, Archive to a storage account, Stream to an event hub, or Send to partner solution.

A Log Analytics workspace can be created in the Azure Portal by searching for "Log Analytics workspace" and selecting Log Analytics workspaces. You can also use the "Send to Log Analytics workspace" option when creating a diagnostic setting for Azure AD audit logs.

Here are the steps to create a Log Analytics workspace:

1. Open the Azure Portal and sign in with an account assigned adequate permission.

2. Search for "Log Analytics workspace" and select Log Analytics workspaces.

3. Click Create log analysis workspace.

4. Enter the required details and click Review + Create.

To stream logs to an event hub, you'll need to create a diagnostic setting. This setting defines the categories of logs and metric data sent to the event hub. You can select from various destinations, including Log Analytics workspace, Event Hubs, and Azure Storage.

Firstly, give your diagnostic log settings a name. You can then select Stream to an event hub and choose CoreAnalytics. Next, select the subscription and event hub namespace for the logs. This is where you'll send your logs.

If you're sending logs to a Log Analytics workspace, you can select the Resource specific toggle on the Diagnostics settings screen. This will allow you to send resource logs to the event hub in JSON format with a records element that contains the records in each payload. The schema depends on the resource type, as described in the Common and service-specific schema for Azure resource logs.

Here are the steps to follow:

Each PT1H.json blob contains a JSON object with events from log files that were received during the hour specified in the blob URL. During the present hour, events are appended to the PT1H.json file as they're received, regardless of when they were generated. The minute value in the URL, m=00 is always 00 as blobs are created on a per hour basis.

Storage is a great destination option for your Azure resource logs. You can send logs to Azure Storage to retain them for archiving.

A storage container is created in the storage account as soon as an event occurs in one of the enabled log categories.

To store logs in Azure Storage, each event is stored in a blob in the following format, using a common top-level schema but unique for each Azure service.

Logs are written to blobs based on the time that the log was received, not the time it was generated. This means that a given blob can contain log data from multiple hours.

Azure Storage allows you to retain logs for a longer period, and a blob can contain data from the previous 48 hours.

Here's a summary of the key benefits of storing logs in Azure Storage:

You can create diagnostic settings to send platform logs and metrics to multiple destinations, such as Log Analytics workspace, Event Hubs, and Azure Storage. This allows you to collect and analyze data from different sources in one place.

Each Azure resource requires its own diagnostic setting, which defines the categories of logs and metric data sent to the destinations. You can choose from various categories, depending on the resource type.

To create a diagnostic setting, you can use the Azure portal, Azure PowerShell, or Azure CLI. In the Azure portal, you can configure diagnostic settings from the Azure Monitor menu or from the menu for the resource.

Here are the steps to create a diagnostic setting in the Azure portal:

1. Go to the Azure Monitor menu or the menu for the resource.

2. If no settings exist, click Add diagnostic setting.

3. Check the box for each category of data you want to send to destinations.

4. Check the box for each destination and add additional information as needed.

5. Click Save.

You can also create a diagnostic setting using PowerShell or Azure CLI. The Set-AzDiagnosticSetting cmdlet in PowerShell and the az monitor diagnostic-settings create command in Azure CLI allow you to create a diagnostic setting with multiple destinations.

For example, you can use the following PowerShell cmdlet to create a diagnostic setting with all three destinations:

```powershell

Set-AzDiagnosticSetting -Name KeyVault-Diagnostics -ResourceId /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.KeyVault/vaults/mykeyvault -Category AuditEvent -MetricCategory AllMetrics -Enabled $true -StorageAccountId /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.Storage/storageAccounts/mystorageaccount -WorkspaceId /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/oi-default-east-us/providers/microsoft.operationalinsights/workspaces/myworkspace -EventHubAuthorizationRuleId /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.EventHub/namespaces/myeventhub/authorizationrules/RootManageSharedAccessKey

```

Similarly, you can use the following Azure CLI command to create a diagnostic setting with all three destinations:

```azure-cli

az monitor diagnostic-settings create \

–name KeyVault-Diagnostics \

–resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.KeyVault/vaults/mykeyvault \

–logs ‘[{“category”: “AuditEvent”,”enabled”: true}]’ \

–metrics ‘[{“category”: “AllMetrics”,”enabled”: true}]’ \

–storage-account /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.Storage/storageAccounts/mystorageaccount \

–workspace /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/oi-default-east-us/providers/microsoft.operationalinsights/workspaces/myworkspace \

–event-hub-rule /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myresourcegroup/providers/Microsoft.EventHub/namespaces/myeventhub/authorizationrules/RootManageSharedAccessKey

```

Data collection is a crucial aspect of Azure diagnostic settings. Most Azure resources write data to the workspace in either Azure diagnostics or resource-specific mode without giving you a choice.

To manage data more efficiently, specify resource-specific mode for any new diagnostic settings as it makes data easier to manage and might help you avoid complex migrations later. You can modify an existing diagnostic setting to resource-specific mode, but data that was already collected remains in the AzureDiagnostics table until it's removed according to your retention setting for the workspace.

To send your vault diagnostics data to Log Analytics, you can select the Send to Log Analytics checkbox and select a Log Analytics workspace. Dedicated tables for each event are created in your workspace, and you can query any of these tables directly.

You can choose between two modes when collecting data in Azure: Azure diagnostics mode and resource-specific mode. Azure diagnostics mode is a legacy method where all data from any diagnostic setting is collected in the AzureDiagnostics table, which can contain a large number of columns.

In resource-specific mode, data is collected in dedicated tables for each resource type, making it easier to manage. This mode is recommended for new diagnostic settings because it will eventually become the standard for all Azure services.

You can modify an existing diagnostic setting to resource-specific mode, but data that was already collected remains in the AzureDiagnostics table until it's removed according to your retention setting for the workspace. New data is collected in the dedicated table.

To switch to resource-specific mode, use the union operator to query data across both tables. This will help you avoid complex migrations later on.

Here's a comparison of the two modes:

For example, if you're using Azure Backup, you'll want to use resource-specific mode to send vault diagnostics data to dedicated Log Analytics tables for backup. This will make it easier to query and manage your data.

Azure Backup users can collect diagnostics events to monitor their backup activities. These events provide detailed data on backup-related artifacts.

Azure Backup provides six types of diagnostics events: Core Azure Backup Data, Addon Azure Backup Job Data, Addon Azure Backup Policy Data, Addon Azure Backup Storage Data, Addon Azure Backup Protected Instance Data, and Azure Backup Operations.

Data for these events can be sent to a storage account, a Log Analytics workspace, or an event hub. If sending to a storage account, it must be in the same region as the Recovery Services vaults.

Azure Active Directory offers two logs to track user activity: UserRiskEvents and RiskyUsers. These logs help identify and analyze risky user events and activities.

Here are the types of events collected by Azure Active Directory:

These logs can be sent to a storage account or a Log Analytics workspace, and can be viewed in the Azure Active Directory -> Security -> Reports section.

Legacy events in Azure diagnostics mode will eventually be deprecated, so it's essential to consider the alternatives.

For Recovery Services vaults, a legacy event called Azure Backup Reporting Data is still supported for backward compatibility, but it's recommended to move to the new events as soon as possible.

The new events provide better discoverability of schemas and their structure, making the data much easier to work with in log queries. They also improve performance across both ingestion latency and query times.

Here are the benefits of choosing the new events:

For Backup vaults, all diagnostics events are sent to the resource-specific tables only, so no migration is needed.

Azure Site Recovery events are sent from the same Recovery Services vault as Azure Backup events.

To receive these events in your Log Analytics workspace, you must choose the resource-specific mode for the two tables mentioned: Azure Site Recovery Jobs and Azure Site Recovery Replicated Items Details.

Choosing the resource-specific mode for Azure Site Recovery events for any other table prevents the required data from being sent.

Azure Site Recovery Jobs is available as both resource-specific and legacy tables.

To send Azure Site Recovery events to Log Analytics, create a new diagnostic setting, select Azure diagnostics, and select the relevant Azure Site Recovery events.

If you're currently sending Azure Site Recovery events to Log Analytics, do not choose the resource-specific mode for these events. Instead, create an additional diagnostic setting.

Here are the steps to follow:

To collect activity data, you can add a Diagnostic Setting in Azure AD. This allows you to track activity logs, which is a crucial step in monitoring your Azure AD activity.

You can create a Diagnostic Setting by scrolling down the menu options to the Monitoring section and clicking “Diagnostic settings.” From there, you can add a new diagnostic setting and select the log categories you wish to monitor.

To create a Diagnostic Setting using Azure CLI, use the az monitor diagnostic-settings create command. This command allows you to create a diagnostic setting with all three destinations, including Log Analytics workspace, storage account, and event hub rule.

Azure Log Analytics is a powerful tool for analyzing activity logs. You can access the logs by navigating to Azure Active Directory -> Monitoring, and then view Sign-in logs, Audit logs, and Provisioning logs.

The Kusto Query language is a SQL-like language that allows you to write queries to analyze your activity logs. You can learn more about Azure Log Analytics and the Kusto Query language by reading Microsoft's tutorials.

Application Insights is a powerful tool for collecting and analyzing data from your applications.

You should store diagnostic logs for Application Insights in a Log Analytics workspace, but don't send the logs to the same workspace that the Application Insights resource is based on.

This configuration can cause duplicate telemetry to be displayed because Application Insights is already storing this data.

Sending Application Insights logs to a different workspace is a good idea.

To do this, restrict the Application Insights user's access to only the Log Analytics workspace linked with the Application Insights resource.

Set the access control mode to Requires workspace permissions and manage permissions through Azure role-based access control.

Configuring data collection in Azure is a straightforward process that requires some planning. To start, you need to select the collection mode, which can be either Azure diagnostics or resource-specific mode. Resource-specific mode is the recommended choice as it makes data easier to manage and might help you avoid complex migrations later.

There are several ways to configure data collection, including adding a diagnostic setting and configuring log data collection. To do this, you must add a diagnostic setting in Azure AD and select the logs you wish to monitor.

One of the key steps in configuring data collection is to add a diagnostic setting. This involves clicking on the "Diagnostic settings" page and then clicking "+ Add diagnostic settings". From there, you can enter a name for the diagnostic setting and select the log categories you wish to monitor.

Another important aspect of data collection is sending resource logs to an event hub. This allows you to send logs outside of Azure to a third-party SIEM or other log analytics solutions. The schema for resource logs sent to an event hub depends on the resource type and is described in the Common and service-specific schema for Azure resource logs.

Here are the steps to send resource logs to an event hub:

You can also use PowerShell or Azure CLI to create a diagnostic setting with multiple destinations. For example, you can use the Set-AzDiagnosticSetting cmdlet in PowerShell to create a diagnostic setting with storage account, workspace, and event hub destinations.

Here are the arguments for the Set-AzDiagnosticSetting cmdlet:

Similarly, you can use the az monitor diagnostic-settings create command in Azure CLI to create a diagnostic setting with multiple destinations.

Here are the arguments for the az monitor diagnostic-settings create command: