Azure Activity Log Management and Storage Solutions

Managing Azure Activity Logs can be a daunting task, especially with the vast amount of data generated daily. Azure Monitor provides a centralized platform to collect, store, and analyze logs from various Azure services.

Azure Monitor allows you to store activity logs for up to 30 days, which can be extended to 400 days with Azure Monitor Logs. This extended retention period provides a larger window for auditing and compliance purposes.

Azure Storage is another option for storing Azure Activity Logs, offering a cost-effective solution for long-term data retention. With Azure Storage, you can store logs for up to 30 days or longer, depending on your storage needs.

Azure Monitor Logs provides a scalable and secure solution for storing and analyzing Azure Activity Logs, making it an ideal choice for large-scale deployments.

You can retrieve activity log events in several ways, including using the Get-AzLog cmdlet in PowerShell, the az monitor activity-log command in the CLI, or the Azure Monitor REST API.

These methods allow you to access the activity log from different platforms and tools, making it convenient to manage and analyze your data.

To send the activity log to a Log Analytics workspace, select Export Activity Logs, which enables the Azure Monitor Logs feature and allows you to correlate activity log data with other monitoring data.

You can send the activity log from any single subscription to up to five workspaces, and activity log data is stored in a table called AzureActivity that you can retrieve with a log query in Log Analytics.

Here's a summary of the benefits of sending activity log data to a Log Analytics workspace:

Note that the default retention period in Log Analytics is 90 days.

To send activity logs to a workspace, you can select Export Activity Logs. This will enable the Azure Monitor Logs feature, allowing you to correlate activity log data with other monitoring data collected by Azure Monitor.

Activity log data in a Log Analytics workspace is stored in a table called AzureActivity. You can retrieve all records in the administrative category using a query like this: "AzureActivity | where Category == 'Administrative'".

To view a count of activity log records for each category, use a query like this: "AzureActivity | summarize count() by Category". This will give you a count of records for each category.

You can send the activity log from any single subscription to up to five workspaces. The default retention period in Log Analytics is 90 days, and you incur no data ingestion or retention charges for activity log data stored in a Log Analytics workspace.

Here are the benefits of sending activity logs to a Log Analytics workspace:

To access your workspace, go to the same services mentioned above for Tenant or Subscription logs, but now select Log Analytics from the Monitoring section of the navigation pane. This will open up a view to run queries against the logs forwarded to that workspace.

Sending data to Azure Event Hubs allows you to send activity log entries outside of Azure to third-party SIEM or log analytics solutions.

Event Hubs consume activity log events in JSON format with a records element containing the records in each payload.

The schema for event Hubs depends on the category and is described in the Azure activity log event schema.

Activity log events from Event Hubs are consumed in a format that includes a records element.

You can create a log profile using PowerShell that writes the activity log to both a storage account and an event hub.

This approach enables you to manage and retrieve data in a more comprehensive manner.

If you need to retrieve activity log events, you have several options. You can use the Get-AzLog cmdlet in PowerShell, the az monitor activity-log command in the CLI, or the Azure Monitor REST API.

Using these methods allows you to access activity log events from various tools and platforms. For example, you can use the Get-AzLog cmdlet to retrieve the activity log from PowerShell, or the az monitor activity-log command to retrieve the activity log from the CLI.

These methods provide flexibility and convenience, enabling you to work with activity log events in a way that suits your needs.

Here are the alternative event retrieval methods in a concise format:

By using these methods, you can effectively manage and retrieve activity log events, making it easier to monitor and analyze your Azure resources.

When you're working with Azure activity logs, it's essential to understand the data structure and retention policies. The data structure has undergone some changes, with some columns being deprecated and replaced with new ones.

The columns that have been deprecated include Category, ActivityStatus, ActivitySubstatus, OperationName, and ResourceProvider, which still exist in AzureActivity but have no data. You can find the replacements for these columns in the updated schema, which are CategoryValue, ActivityStatusValue, ActivitySubstatusValue, OperationNameValue, and ResourceProviderValue.

You might need to modify your log queries if you're using the deprecated columns, as they have different formats. Additionally, be aware that the values in these columns might be all uppercase, so use the =~ operator for a case-insensitive comparison.

Here's a table to help you understand the column changes:

Values are success, start, accept, failureActivityStatus

Values same as JSONActivityStatusValue

Values change to succeeded, started, accepted, failedThe valid values change as shown.subStatusActivitySubstatusActivitySubstatusValueoperationNameOperationNameOperationNameValueREST API localizes the operation name value. Log Analytics UI always shows English.resourceProviderNameResourceProviderResourceProviderValue

Activity log events are retained in Azure for 90 days and then deleted. This means you don't have to worry about storage costs during this time, regardless of the volume of logs.

The Export activity logs experience sends the same data as the legacy method used to send the activity log with some changes to the structure of the AzureActivity table.

Some columns in the AzureActivity table are deprecated, but they still exist with no data. The replacements for these columns contain the same data as the deprecated column, but in a different format.

You might need to modify log queries that use the deprecated columns. This is because the valid values for some columns have changed. For example, the ActivityStatus column now has values like "succeeded", "started", "accepted", and "failed".

To handle queries that use these columns, use the =~ operator for a case-insensitive comparison. This is because the values in some columns might be all uppercase.

The following columns have been added to AzureActivity in the updated schema: Authorization_d, Claims_d, and Properties_d.

Establishing log retention policies is crucial for effective log storage management. Determine how long logs should be retained based on compliance and business requirements.

Log retention policies should be configured to automatically delete or archive logs after a certain period. This helps manage storage space and ensure compliance with regulatory requirements.

Activity logs are retained in Azure for 90 days before being deleted. During this time, there's no charge for entries regardless of volume.

For more advanced functionality, such as longer retention periods, consider creating a diagnostic setting. This allows you to route log entries to another location based on your specific needs.

Creating hierarchies for your logs can greatly simplify log management and search. Organizing logs into hierarchies or groups based on the Azure service, application, or resource generating the logs is a great way to do this.

Having a clear structure makes it easier to identify and retrieve specific logs. By creating log hierarchies, you can quickly locate logs that are relevant to a particular issue or area of your application.

This approach also helps you to scale your log management as your application grows. As your application generates more logs, a well-structured hierarchy will help you to efficiently manage and search through them.

By structuring your logs in this way, you can save time and reduce the effort required to find and analyze specific logs.

Monitoring in Azure is crucial in a cloud-based environment, where it involves collecting and analyzing performance metrics, resource utilization, and other data to ensure efficient operation.

Azure offers a range of services and tools for monitoring that can help you gain insights into your infrastructure's health and performance.

Monitoring helps you identify and troubleshoot issues before they become major problems, and it's essential for maintaining a healthy and efficient Azure environment.

You can access your logs and monitor your Azure environment through the Log Analytics Workspace, where you can send all of your logs to a central location for log access.

The Log Analytics Workspace provides a single location for log access, and you can execute queries in Kusto Query Language (KQL) to analyze your logs.

Monitoring is especially useful during incidents and investigations, where interactive dashboards can provide a real-time, at-a-glance view of your Azure environment's health.

Activity log insights provide a set of dashboards that monitor the changes to resources and resource groups in a subscription, presenting data about which users or services performed activities and their status.

Azure's security features are designed to help you detect and respond to security threats and vulnerabilities. This is crucial because Azure environments are frequently targeted by cyberattacks.

Logging and monitoring are essential for maintaining visibility into security-related events. Azure's logging and monitoring capabilities help you stay on top of potential security issues.

Security is a top priority in Azure, making it critical to maintain visibility into security-related events. Cyberattacks can have serious consequences, so it's essential to be proactive about security.

Azure's security features are designed to help you detect and respond to security threats and vulnerabilities, making it easier to stay secure.

To get the most out of your Azure activity log, you need to understand what's happening next.

Platform logs are a crucial part of the process, providing a detailed record of all events and activities.

Reviewing your activity log event schema will help you make sense of the data and identify areas for improvement.

Activity log insights offer valuable insights into your Azure usage and can inform your decision-making.

Here are the next steps to take:

Azure Activity Log Features offer a robust set of capabilities to help you manage and monitor your Azure resources.

You can use Azure Activity Log to diagnose issues with your Azure resources, such as monitoring for subscription changes or resource creations.

Azure Activity Log Features also allow you to filter logs by category, resource group, and time range, making it easier to pinpoint specific events.

The Azure Portal provides a quick and easy method of reviewing Tenant and Subscription logs, but it has its limitations, only providing logs from the last 30 days for Tenant logs and the last 90 days for Subscription logs.

To access Tenant logs, search for Azure Activity Directory and go to that service, then navigate to the Monitoring section and select either Sign-in Logs or Audit Logs.

Sign-in Logs provide a view of user sign-ins, with filters available based on various fields, and each row can be clicked to open a pane with detailed information about the event.

You can also download up to 100,000 rows per log type in CSV or JSON format, or configure the logs to automatically forward to a Log Analytics Workspace, a storage account, or EventHub.

Audit Logs offer a similar view, but with a download option allowing up to 250,000 events.

To view Subscription logs, search for Activity log and access the service, where you'll find a formatted table with filters to search through the last 90 days of logs.

Activity logs can only be downloaded in CSV format, and each operation is made up of a series of events, making it challenging to review the details of a large number of events.

Fortunately, you can programmatically query this data using the Get-Azlog PowerShell cmdlet or the az monitor activity-log CLI command.

The Graph API offers a unique way to retrieve Azure AD activity reports, providing access to Tenant logs, both sign-in and audit logs. This method is not configured or accessed directly via the portal.

To use the Graph API, you'll need to develop custom code using one of the supported languages, as the documentation for this API endpoint can be found here: https://learn.microsoft.com/en-us/graph/api/resources/azure-ad-auditlog-overview?view=graph-rest-1.0.

This method allows for the export of logs to a file locally or the building of an integration for a SIEM, giving you more flexibility in how you manage your Azure AD activity reports.