Concepts
Event source mapping is what directs events from your source (DynamoDB, Kinesis, or SQS) to an AWS Lambda function. When an event occurs in your event source, it triggers the Lambda function to execute. This is particularly useful for creating reactive, event-driven architectures within AWS.
Types of Event Sources
AWS Lambda supports several types of event sources:
- Stream-based sources: Such as Amazon Kinesis and DynamoDB Streams.
- Queue-based sources: Such as Amazon SQS.
- Other: AWS Lambda also natively supports S3 events, SNS notifications, and many more, but the event source mapping is primarily for streams and queues.
Basic Configuration for Event Source Mapping
To set up an event source mapping via the AWS CLI, use the create-event-source-mapping command. The following parameters are essential:
--event-source-arn: The ARN of the event source (stream or queue).--function-name: The name of the Lambda function you want to invoke.--starting-position: Only for stream sources; it describes where to start reading events (e.g.,LATESTorTRIM_HORIZON).
For streams (DynamoDB or Kinesis):
aws lambda create-event-source-mapping \
--event-source-arn arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME \
--function-name FUNCTION_NAME \
--starting-position LATEST
For SQS queues:
aws lambda create-event-source-mapping \
--event-source-arn arn:aws:sqs:REGION:ACCOUNT_ID:queue/QUEUE_NAME \
--function-name FUNCTION_NAME
Best Practices with Event Source Mapping
When using event source mapping, there are some best practices to keep in mind:
- Batch Size Configuration: Configure the batch size according to your workload. For Kinesis and DynamoDB Streams, this can be up to 10,000 records or 10 MB, whichever is smaller. For SQS, it can be up to 10 messages.
- Error Handling: Make sure to handle errors in your Lambda function so that unsuccessful processing doesn’t block the stream or queue.
- Parallelization: With Kinesis, you can set up parallelization by setting the
--parallelization-factor. - DLQ Configuration: For SQS event sources, set up a Dead Letter Queue (DLQ) to capture any messages that couldn’t be processed successfully after multiple attempts.
Limitations and Considerations
Keep in mind there are limits and trade-offs:
- Execution Role Permissions: Your Lambda function’s execution role must have the necessary permissions to poll the event source and invoke your function.
- Concurrency: Stream-based triggers preserve the order of records, queue-based do not necessarily do so.
- Scaling: Stream-based event sources scale the number of Lambda function invocations based on the number of shards in the stream. For SQS, it’s based on the number of messages in the queue.
Monitoring and Troubleshooting
AWS provides monitoring capabilities for your event source mappings through Amazon CloudWatch. You can monitor metrics like:
- Invocation counts
- Error counts
- Throttles
- IteratorAge for streams, which measures the age of the last record for each batch.
When configured correctly, event source mapping is a powerful tool in AWS that allows seamless integration between different AWS services and Lambda, but it’s critical to understand the characteristics of each event source type, their limitations, and how to troubleshoot common issues.
As you prepare for your AWS Certified Developer – Associate exam, ensure you’re comfortable creating, updating, and troubleshooting event source mappings as these are practical, hands-on skills that are likely to be tested.
Answer the Questions in Comment Section
(True/False) Event source mapping is only used with AWS Lambda to poll Kinesis data streams.
- True
- False
Answer: False
Explanation: Event source mapping is used to connect AWS Lambda functions to various data sources such as Amazon Kinesis Data Streams, Amazon DynamoDB Streams, and Amazon SQS queues, not just Kinesis data streams.
(Single Select) Which AWS service is NOT directly integrated with AWS Lambda using event source mapping?
- Amazon Kinesis
- Amazon S3
- Amazon MQ
- Amazon DynamoDB
Answer: Amazon S3
Explanation: Amazon S3 is integrated with AWS Lambda through a different mechanism involving S3 event notifications, not through event source mapping.
(True/False) You can create an event source mapping using the AWS Management Console, AWS CLI, or the AWS SDKs.
- True
- False
Answer: True
Explanation: AWS provides flexibility to create an event source mapping through various interfaces such as the AWS Management Console, AWS CLI, and AWS SDKs.
(Multiple Select) Which types of processing can event source mappings help facilitate with AWS Lambda?
- Batch processing
- Real-time processing
- Transactional processing
- Stream processing
Answer: Batch processing, Real-time processing, Stream processing
Explanation: Event source mappings can facilitate batch processing, real-time processing, and stream processing. It helps in batching records when polling data sources before invoking the Lambda function.
(True/False) Batch size for an event source mapping with Kinesis Data Streams is configurable up to 10,000 records or 6 MB.
- True
- False
Answer: True
Explanation: For Kinesis Data Streams, you can configure the batch size up to 10,000 records, or a total payload size of 6 MB, whichever is reached first.
(Single Select) Event source mapping with DynamoDB streams can trigger a Lambda function:
- Only when new items are added to the table.
- When items are updated, added, or deleted in the table.
- When the table is deleted.
- None of the above.
Answer: When items are updated, added, or deleted in the table.
Explanation: DynamoDB Streams can trigger a Lambda function for various table activities, including item updates, additions, and deletions.
(True/False) Once event source mapping is created, it cannot be updated.
- True
- False
Answer: False
Explanation: Event source mappings can be updated to change configurations such as batch size, starting position, and enabled/disabled status.
(Multiple Select) What are some best practices when using event source mapping with AWS Lambda?
- Setting appropriate batch sizes
- Processing records in the order they are received
- Deleting records from the stream
- Retrying failed batches
Answer: Setting appropriate batch sizes, Processing records in the order they are received, Retrying failed batches
Explanation: Setting appropriate batch sizes and retrying failed batches are best practices to ensure efficient Lambda processing. It is also important to process records in the order they are received, especially with Kinesis and DynamoDB Streams. You should not delete records from the stream, as this is managed by the service itself.
(True/False) AWS ensures that events are delivered exactly once and in the order they are generated when using SQS with event source mapping.
- True
- False
Answer: False
Explanation: AWS SQS queues do not guarantee exactly-once delivery or order of messages. However, FIFO (First-In-First-Out) queues aim to ensure order and exactly-once processing to some extent.
(Single Select) What is the initial position setting used in event source mapping with Kinesis Data Streams and DynamoDB Streams?
- LATEST
- TRIM_HORIZON
- AT_TIMESTAMP
- SHARD_END
Answer: TRIM_HORIZON
Explanation: TRIM_HORIZON is a common initial position setting used in event source mappings that tells the service to start processing from the oldest record available in the stream.
(True/False) You must manually configure an event source mapping’s error handling, including setting up a dead-letter queue.
- True
- False
Answer: True
Explanation: Error handling must be configured manually for event source mappings. You can set up a dead-letter queue (DLQ) to manage Lambda invocation errors.
(True/False) The number of concurrent batches per shard is limited when using event source mapping with AWS Lambda and Amazon Kinesis.
- True
- False
Answer: True
Explanation: When using event source mapping with Amazon Kinesis, there is a limit of 5 concurrent batches per shard. This is to prevent read throttling and potential data loss.
Can anybody explain Event Source Mapping with DynamoDB Streams in a bit more detail?
Appreciate the information presented in this blog post!
Does AWS provide any monitoring tools for Event Source Mapping?
The article was good. Keep up the great work!
Just what I was looking for! Thanks!
Great blog post! I was struggling with Event Source Mapping, and this cleared up a lot of my confusion.
Can someone explain the permissions required for setting up Event Source Mapping with Lambda?
Thanks for the detailed insights!