Trigger Lambda from S3 Bucket with Python Step-by-Step Tutorial

To trigger a Lambda function from an S3 bucket, you'll need to create an S3 bucket notification. This notification will be sent to your Lambda function whenever an object is uploaded to the bucket.

You can create an S3 bucket notification by going to the S3 dashboard, selecting your bucket, and clicking on the "Properties" tab. From there, click on "Events" and then "Edit" to add a new event notification.

The event notification will specify the event type, such as "Object created", and the Lambda function that should be triggered. You can also specify the prefix and suffix of the object key to filter which objects trigger the Lambda function.

For example, if you want to trigger the Lambda function for objects with a specific prefix, you can enter the prefix in the "Prefix" field.

To get started with triggering a Lambda function from an S3 bucket, you'll need to ensure you have the right prerequisites in place.

First and foremost, you'll need to have the .NET 6 SDK or higher installed on your machine, but I'll be using .NET 8 to build the WebAPI.

A basic understanding of AWS Lambda is also required, and I've written articles about this topic that you can refer to for more information.

In addition to AWS Lambda, you'll also need a basic understanding of Amazon S3, which you can also find more information about in another article.

To access your AWS resources programmatically, you'll need to have the AWS CLI and profile configured. This will make it easier to interact with your AWS account.

You'll also need to have the AWS Toolkit extension for Visual Studio installed, which will provide you with a more seamless development experience.

Finally, you'll need to have Visual Studio IDE installed, and I'm currently using Visual Studio 2022 Preview to take advantage of the .NET 8 SDK.

Here's a quick rundown of the prerequisites you'll need:

To create a Lambda function that can be triggered from an S3 bucket, you'll need to start by creating the function itself. This can be done in the AWS Lambda console by clicking "Create Function" and naming your function. In the example, the function was named "bbd-s3-trigger-demo" and was using Python 3.9.

You'll also need to give your Lambda function the necessary permissions to access the S3 bucket. This can be done by clicking "Change default execution role" and selecting "Create a new role from AWS policy template." From there, you can choose "Amazon S3 object read-only permissions" to grant your function the necessary access.

If you're creating your function through infrastructure as code, such as CloudFormation or CDK, you'll want to grant your function this permission explicitly. However, in the console, selecting "Amazon S3 object read-only permissions" will suffice.

To verify that your function is working correctly, you can check the "Monitor" tab in the Lambda console. This will display Lambda's CloudWatch metrics, including the Invocations graph, which should equal the number of files you uploaded to Amazon S3.

To modify your Lambda function, you'll need to revisit the Create function page. From there, you can adjust the function code, add triggers, or change the handler. You can also update the environment variables if needed.

The Monitor tab is a great place to verify that your changes are taking effect. It displays Lambda's CloudWatch metrics, and you can see the Invocations graph to ensure your function is running correctly.

To confirm our code is working as anticipated, we need to create a test event and invoke it manually. This will help us ensure our Lambda function is triggered correctly.

After completing the upload, we can head over to the Lambda Monitoring section to view invocation history. Do note that it can take a few minutes after uploading your file for the metrics to be populated.

To see our results faster, we can click on "View logs in CloudWatch" and look at the latest log stream at the top of the page. This will show us the Lambda's execution logs, including the printed contents of the file we uploaded into S3.

The Lambda's execution logs will reveal the details of the file that we uploaded, giving us a clear indication of whether our code ran successfully.

Notification is a crucial aspect of triggering a Lambda function from an S3 bucket. Amazon S3 supports notifications for different types and destinations to publish these events.

You can specify the event types and the destination for the events when configuring event notification triggers. Notification types include s3:TestEvent, s3:ObjectCreated, and s3:ObjectRemoved.

The s3:ObjectCreated notification type can be further specified to listen to specific operations on S3, such as PUT, POST, or COPY. The s3:ObjectRemoved notification type can be specified to listen to Delete or DeleteMarkerCreated events.

To set up S3 notification and Lambda trigger, you can navigate to the Lambda function configuration and add a trigger from the S3 bucket. You can select the event types that you want to trigger the Lambda function, such as all object create events.

Here are the notification types supported by Amazon S3:

You can also generate a test event to confirm your code is working as anticipated. To create a test event, you can select the s3-put template and use it as an example record.

To grant your Lambda function the necessary permissions to access your S3 bucket, navigate to the AWS Management Console and open up Lambda. Select your new Lambda and go to Configurations -> Permissions.

Click on the IAM Role of your Lambda, which is usually named lambda_exec_{lambdaName}. Under permissions, click on Add Permissions and Attach Policies.

Search for S3 and select AmazonS3FullAccess, but note that this is not recommended in production scenarios and you should be more specific. Ideally, you need to add more permissions to your Lambda function.

The Lambda function should have the GetObject read permission on the S3 storage, which can be achieved by editing the Lambda's IAM Role and adding an inline policy. This will allow your Lambda function to read the Object metadata on processing the event notification.

The Lambda Configuration supports two optional properties: Prefix and Suffix.

To deploy your Lambda function, you can use the AWS Toolkit's Publish to AWS Lambda option.

Make sure to install the AWS Toolkit and set up Credentials for your local development environment, as this will help you quickly develop and deploy your .NET application on the AWS infrastructure.

To publish the Lambda function, select the AWS Lambda Lambda Project template and choose the S3 blueprint, which creates an AWS Lambda function template project with a Function.cs class.

You can also set up an automated build/deploy pipeline to publish to your AWS Account for real-world applications.

Here are some social media platforms where you can share your content, but for the purpose of this article, we'll focus on S3 bucket triggers:

You can publish your Lambda function to your AWS account using the Publish to AWS Lambda option provided by the AWS Toolkit. This allows you to directly publish the new lambda function with your development environment connected to your AWS account.

The AWS Toolkit makes it easy to deploy your function, and it's a great option for small projects or development environments. To set up an automated build/deploy pipeline, you can use GitHub Actions, which provides a more robust and scalable solution for real-world applications.

To publish your Lambda function, you can use the Publish to AWS Lambda option. This option allows you to quickly deploy your function to your AWS account.

Here are the platforms you can use to publish your Lambda function:

Using a blueprint like S3 can help you quickly set up a Lambda function and trigger it with an S3 event notification. This can be a great way to get started with Lambda functions and AWS.

Configuring your Lambda function is crucial to ensure it's triggered only when necessary.

Specifying the prefix and suffix in the Lambda trigger configuration allows you to filter the S3 objects that trigger the function. For example, setting the prefix to 'unprocessed/' will only trigger the function when objects are uploaded to the 'unprocessed' folder.

This is particularly useful when reading or writing to the same S3 bucket from the same Lambda function, as it prevents recursive invocations. You can write back the new file to a different folder that doesn't fall under the original filter condition.

Using a prefix like 'unprocessed/' can help you trigger the Lambda function only when a file is changed under a specific filter, like being under the 'unprocessed' folder. This allows you to process images uploaded to the 'unprocessed' folder and write back the processed image to a different folder, like 'processed'.

Now that we've set up our minimal API to upload an image to an S3 Bucket location, it's time to think about what's next. We'll be writing a .NET Lambda to handle the incoming S3 event notification.

Our Lambda will need to fetch the newly uploaded image and process it, so we'll be using the popular ImageSharp package for image conversion. This package will help us create thumbnails of the uploaded images. We'll be using .NET 6 for this Lambda function.

We'll also need to configure our S3 Bucket to receive notifications when an image is uploaded, and set up the necessary configurations to get this working. This will involve some setup and testing to ensure everything is working as expected.