AWS Log S3 Integration and Configuration Guide

AWS Log S3 integration is a powerful way to store and manage your log data. This allows you to easily monitor and analyze your application's performance.

To get started with AWS Log S3 integration, you'll need to set up a bucket in your AWS account. This is where your log data will be stored.

A bucket in AWS S3 is essentially a container that holds your log data. You can think of it like a file folder, but instead of files, it holds objects like log files.

To integrate AWS Log S3, you'll need to configure your AWS services to send log data to your S3 bucket. This can be done through the AWS Management Console or the AWS CLI.

To set up S3 log collection, you'll need to create an Amazon Simple Notification Service (SNS) topic in your AWS account and an IAM role to grant Alert Logic access to your S3 buckets. You can use an AWS CloudFormation template provided by Alert Logic to set this up, or do it manually.

First, set up the SNS topic and IAM role in your AWS account. This will allow Alert Logic to receive notifications from your S3 buckets. Once you've done this, you'll need to configure collection in the Alert Logic console.

In the Alert Logic console, go to the Application Registry page and click on the AWS tile. Select Amazon S3 and enter a name for this log collection instance. Choose the type of logs to collect from your S3 bucket and enter the ARN for the IAM role granting access to your S3 bucket. Don't forget to enter the SNS topic ARN and the S3 bucket name.

If you want to collect logs from a specific folder in your S3 bucket, enter the prefix that identifies the folder in the S3 Object Key Prefix field. Leave this field blank if you want to collect all objects in the bucket.

Here's a summary of the required fields:

After you've filled in these fields, click ADD and wait a few minutes for the application to create and appear in your application list.

To set up S3 log ingestion, you'll need to provide the name of the S3 bucket and optionally a prefix, which operates like a folder, allowing you to ingest only specific files.

One prefix per bucket is allowed, and changing it will only ingest new files with the new prefix. You can also find your AWS account ID on the AWS Security Credentials page, which you'll need to enter in the process.

To troubleshoot S3 ingestion issues, check if the AWS source is enabled, if the log files exist, and if you're exceeding the data volume limit per your subscription. You can also try the manual method or search for errors on the page.

Here are some common issues to check for when troubleshooting S3 ingestion:

To set up log ingestion, you'll need to configure your S3 bucket and IAM role. In the Alert Logic console, click the menu icon and then click Configure, followed by Application Registry. From there, click the AWS tile and then Amazon S3.

You'll need to enter a name for your Amazon S3 log collection instance, and select the type of logs to collect from the S3 bucket. You'll also need to enter the ARN for the IAM role granting access to your S3 bucket, as well as the External ID and S3 Bucket Name.

If you want to collect logs from a particular folder in your S3 bucket, enter the prefix that identifies the folder, followed by a slash. For example, "logs/". Leave this field blank if you want Alert Logic to collect all objects in the bucket.

You'll also need to enter the ARN of the SNS topic created earlier that receives S3 notifications, as well as the SNS Topic Region.

Here's a summary of the required fields:

Once you've entered all the required fields, click ADD. Wait a few minutes for the application to create and appear in your application list.

Data Processing is a crucial step in log ingestion and processing. It's where the magic happens, turning raw log data into actionable insights.

Log data is parsed, normalized, and stored in a data lake, allowing security teams to write detections, detect anomalies, and conduct investigations on logs in the context of days, weeks, or months of data.

Normalization fields are applied to all log records, standardizing names for attributes and empowering users to correlate data across all log types. This makes it easier to identify patterns and trends.

Panther's search tools empower you to investigate your normalized logs for suspicious activity or vulnerabilities.

To integrate Amazon S3 with Alert Logic, you must complete the log collection configuration process in the Alert Logic console. This configuration is an account-level integration, allowing you to configure more than one instance of Amazon S3 log collection.

To access the Application Registry page, click the menu icon and navigate to Configure > Application Registry. In the Application Registry, click the AWS tile, and then click Amazon S3.

Here's a step-by-step guide to configuring Amazon S3 log collection:

Note that it may take approximately 10 minutes for the application to be configured correctly.

To set up an SNS topic and IAM role, you'll need to create an Amazon Simple Notification Service (SNS) topic where AWS publishes S3 notifications. This is crucial for sending notifications to Alert Logic.

First, sign in to the Amazon SNS console and choose to create a Standard topic. Then, enter a descriptive Name for the topic, such as "my-sns-topic."

The SNS topic access policy must be configured to allow S3 to publish notifications to the topic and grant Alert Logic permission to receive and process S3 notifications. You can use the S3 SNS access policy document (JSON file) provided by Alert Logic to configure the policy.

To do this, download the S3 SNS access policy document, and edit it to include the necessary information, such as the S3 bucket name and region, and the SNS topic ARN. Then, scroll to the end of the form and choose Create topic.

Alternatively, you can use an AWS CloudFormation template that Alert Logic provides to set up the SNS topic and IAM role. However, setting them up manually gives you more control over the configuration.

Once you've set up the SNS topic, you'll need to configure the IAM role to grant Alert Logic access to your S3 buckets. This involves creating an IAM role and attaching the necessary policies to it.

Here's a summary of the steps to set up an SNS topic and IAM role:

By following these steps, you'll be able to set up an SNS topic and IAM role that allows Alert Logic to receive and process S3 notifications.

S3 Server Access Logging is a feature provided by Amazon Web Services that allows you to log all requests made to your S3 bucket. This can be useful for monitoring and troubleshooting purposes, as well as for compliance and auditing.

Panther can collect S3 logs to help you identify suspicious activity in real time. Your normalized data is then retained to power future security investigations.