SNS→Lambda Or SNS→SQS→Lambda
Should you be processing messages directly from SNS to Lambda or via an SQS Queue? Learn the disadvantages of directly processing messages from SNS and how you can solve those by introducing an SQS Queue in the middle.
Table of Contents
This article is sponsored by AWS and is part of my AWS Series.
Amazon Simple Notification Service (SNS) is a publish-subscribe service managed by AWS.
It enables asynchronous communication between the publishers and subscribers using messages.
In an earlier blog post, we learned how to use AWS Lambda Functions to process messages coming into SNS. I highly recommend checking out if you are new to SNS and Lambda Triggers.
In this post, we will see some of the disadvantages of processing messages directly from an SNS and learn how introducing an SQS in the middle solves some of them.
We will learn,
- Disadvantages of directly processing messages from SNS → Lambda
- Connecting SNS → SQS → Lambda
- Advantages of Processing Messages From SNS → SQS → Lambda
Disadvantages of directly processing messages from SNS → Lambda
As long as the messages are processed successfully and the Lambda trigger configuration is valid, processing messages directly from an SNS using Lambda Trigger works fine.
Amazon SNS delivers messages asynchronously to Lambda, which means it does not wait for Lambda to return after execution. Lambda is responsible for ensuring the message is processed successfully.
Not everything below is a disadvantage for all scenarios, but they can be if not handled in your application scenario. So, evaluate the pros and cons of subscribing directly from an SNS Topic.
Here are the main disadvantages when processing messages directly from SNS.
- SNS Delivers messages asynchronously, and Lambda is responsible for handling errors/retries.
- For stale SNS subscriptions, SNS retries are lost after a few retries.
- Lambda DLQ needs to be handled separately.
- No Batching is available when processing messages from SNS.
- No control on Lambda Concurrency, messages are processed one by one as soon as they arrive.
In cases of exceptions in processing, the messages get retried depending on the Asynchronous configuration set on the Lambda Function. By default, it retries two times and is discarded. If a DLQ is setup, it needs to be monitored and processed separately to the SNS.
Connecting SNS → SQS → Lambda
Let’s see how we can process messages from SNS by introducing an SQS in the middle. We will then see the advantages of processing messages this way.
SNS supports different protocols for Subscription, and SQS is one of them. Navigate to SNS → Topic → Create Subscription, which prompts a new window to enter the Subscriber details.
Select SQS as the protocol and provide the ARN details in the Endpoint.
If you want the raw message coming into the SNS to be delivered as is to the SQS, make sure to ‘Enable raw message delivery’. If unticked, you will get an SNS Record type inside your SQS message payload and will contain the additional SNS metadata ( including the Subject and other attributes).
Based on your application scenario, you can enable this option. For this post, I have enabled raw message delivery.
Additionally, you can specify a Filter Policy if you want to filter the messages sent to the Queue. You can also set up a DLQ in case this SQS subscription becomes stale (if someone accidentally deletes the original SQS queue).
I cover these configuration details in detail in my post on Amazon SNS and AWS Lambda Triggers.
Once you have created the Subscription, we need to update the Access Policy for the SQS to allow SQS to send messages to the Queue.
For this, navigate to the SQS → Access policy and click Edit.
Add the access policy statement, as shown below, to allow SNS the sqs:SendMessage Action to send messages to the SQS. Make sure to replace the Resource with the SQS ARN details and specify the SNS ARN to allow only one specific SNS to send messages.
{
"Effect":"Allow",
"Principal":{
"Service":"sns.amazonaws.com"
},
"Action":"sqs:SendMessage",
"Resource":"<SQS ARN>",
"Condition":{
"ArnEquals":{
"aws:SourceArn":"<SNS ARN>"
}
}
}
Once added, our subscription is all ready to go. You can test by publishing a new message to the SNS Topic, and it will automatically be sent to the SQS Queue.
AWS Lambda Function can process messages from SQS Queue.
Advantages of Processing Messages From SNS → SQS → Lambda
Amazon SQS invokes the Lambda Function synchronously. This means it waits for a response from the Lambda Function. Only if it’s a successful response is the message removed from the Queue. Otherwise, the message reappears in the Queue based on the VisiblityTimeout, and the Maximum receives setting value.
- On errors in processing, the message is replaced into the Queue. You can move these messages to a DLQ and Re-drive these messages back into the original queue for reprocessing.
- SQS allows Batch delivery of messages, so you can save on the cost of the number of Lambda functions to run. Along with batch delivery, you can also enable batch error processing so that only messages that error out in a batch is retried.
- You can control the Lambda concurrency, and you run fewer Lambda instances with batching.
- SQS supports
BatchWindow, so you can wait for messages to queue up before starting to processing. - Messages stay in the Queue if there are no consumers (for a max of 2 weeks).
If the messages are critical for the application, then processing them through an SQS is more reliable than processing messages directly from SNS. With SQS, you can also have a single flow for processing and error monitoring. You can also optimize the cost of Lambda functions by batching and processing more records at a time.
I hope this helps you understand and choose between different options when processing messages from SNS.
Additional Reading
Rahul Nath Newsletter
Join the newsletter to receive the latest updates in your inbox.
