Integration services (for example, Amazon SQS, Amazon SNS, Amazon EventBridge, Step Functions)

Can you explain the key differences between Amazon SQS and Amazon SNS, and when you would use one over the other?

Amazon SQS (Simple Queue Service) is a message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SNS (Simple Notification Service) is a pub/sub messaging and mobile notifications service for coordinating the delivery of messages to subscribing endpoints or clients. You would use SQS when you need a reliable, highly scalable, hosted queue for storing messages as they travel between computers. On the other hand, you would use SNS when you need to send messages to multiple subscribers or to fan-out messages, such as push notifications to mobile devices, or broadcast email messages.

How would you secure messages in Amazon SQS?

To secure messages in Amazon SQS, you can use server-side encryption (SSE) to encrypt each message body. SQS encrypts messages using the AWS-managed Customer Master Keys (CMKs) or customer-managed CMKs in AWS KMS. Additionally, you can use SQS policies and IAM policies to control access to your queues.

What is Amazon EventBridge and how does it differ from AWS CloudWatch Events?

Amazon EventBridge is a serverless event bus service that enables you to connect your applications with data from a variety of sources. EventBridge is built on the CloudWatch Events infrastructure and offers the same functionality but adds additional features such as schema registry, better integration with SaaS applications, and custom event buses for enterprise-wide event management. EventBridge provides a more powerful and flexible way to create and manage event-driven architectures.

Describe a scenario where you would use AWS Step Functions, and explain how it adds value to that scenario.

AWS Step Functions is a service that facilitates the coordination of components of distributed applications and microservices using visual workflows. One useful scenario to use Step Functions is in the management and automation of complex multi-step processes and business logic across different AWS services. For instance, for orchestrating a series of AWS Lambda functions for data processing or analytics pipelines, Step Functions can simplify error handling, state management, and retries, making the overall workflow more reliable and easier to maintain.

How do you achieve message durability and retention in Amazon SQS?

In Amazon SQS, messages are stored on multiple servers across multiple Availability Zones automatically, providing built-in redundancy and message durability. SQS supports configurable message retention periods from 1 minute to 14 days, with the default being 4 days. This configurable period allows you to specify how long messages should be retained in the event they cannot be processed immediately.

What is the purpose of Amazon SNS message filtering, and how would you set it up?

Amazon SNS message filtering allows subscribers to receive only the messages that interest them, without needing to receive all of the messages published to the topic. This is achieved by assigning attribute-value pairs to messages and setting up filter policies on the subscription level. To set it up, you first publish messages with metadata in the form of message attributes. Then, you create filter policies in JSON format for your subscriptions, specifying which attributes and values must (or must not) be present in the messages they receive.

When would you choose to use an Amazon SNS FIFO (First-In-First-Out) topic, and what advantages does it offer?

An Amazon SNS FIFO topic is ideal when the order of events and exactly-once delivery are crucial for your application. Use cases such as bank transactions, order processing systems, or any workflow that requires strict sequencing benefit from FIFO topics. FIFO offers advantages such as deduplication, where messages with identical content sent within the deduplication interval are treated as duplicates and only delivered once, and ordering, where messages are delivered in the exact order they are sent.

Can you explain the difference between Amazon SQS standard and FIFO queues and why you might choose one type over the other?

Amazon SQS offers two types of queues: standard and FIFO. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery, while FIFO queues guarantee that messages are processed exactly once, in the exact order they are sent. You might choose a standard queue for applications where the order is not critical and performance is the priority. In contrast, you would choose a FIFO queue where the order is critical, such as in banking systems or order processing systems where transactions need to be processed in the specific order they are received.

Describe a situation where you could use Amazon EventBridge to improve an application’s design.

Amazon EventBridge can be used to decouple your application components and respond to changes in your AWS environment or third-party SaaS applications. For instance, if you have an e-commerce application that needs to update several services upon an order being placed, you could use EventBridge to capture the order event and route it to different parts of your application (such as inventory, billing, and shipping services) via EventBridge rules. This improves the design by reducing direct integration and dependencies between services, making the system more scalable and easier to maintain.

Explain how you would use dead-letter queues (DLQ) in Amazon SQS to handle message processing failures.

A dead-letter queue (DLQ) is used to collect messages that were not successfully processed by your Amazon SQS queue consumers after a specified number of attempts. By enabling a DLQ, you can set aside those failed messages to diagnose the problem without blocking the processing of new messages. This helps in debugging and ensures that problematic messages do not continuously recycle through your queue, blocking or delaying valid messages that can be processed successfully.

How do you monitor the performance of your Amazon EventBridge events and ensure they are triggering intended targets as expected?

To monitor the performance and delivery of EventBridge events, you can use Amazon CloudWatch to track metrics such as the number of events matched to a rule, the number of successful and failed invocations of targets, and the latency of event delivery. Additionally, you can set up CloudWatch alarms to alert you when certain thresholds are breached or when delivery failures occur. Furthermore, EventBridge logs the API calls in AWS CloudTrail, which can be used for auditing and troubleshooting purposes.

Contrast the capabilities of Amazon SQS, Amazon SNS, and Amazon EventBridge in terms of subscribers and message throughput.

Amazon SQS is designed for message queuing where each message is processed by a single consumer and can handle high message throughput. SQS standard queues provide a nearly unlimited number of transactions per second (TPS) with at-least-once delivery, while SQS FIFO queues provide lower throughput but ensure order and exactly-once processing. Amazon SNS is designed for publish-subscribe (pub/sub) messaging where a single message can be delivered to many subscribers and also supports high throughput. Finally, Amazon EventBridge is designed for event-driven architectures, it enables event ingestion and delivery to a large number of AWS service targets, and its throughput is generally high but can vary based on the pattern and complexity of the event routing rules.