AWS SQS, SNS, MQ and Kinesis

These are the types of Asynchronous communications solutions in AWS.

AWS SQS (Simple Queuing Service):

In Amazon SQS, you can configure the message retention period to a value from 1 minute to 14 days. The default is 4 days. Once the message retention limit is reached, your messages are automatically deleted.

A single Amazon SQS message queue can contain an unlimited number of messages. However, there is a 120,000 limit for the number of inflight messages for a standard queue and 20,000 for a FIFO queue. Messages are inflight after they have been received from the queue by a consuming component, but have not yet been deleted from the queue.

It is a Queue model and is of 2 types:

  1. Standard:
    • Unlimited throughput (processes unlimited messages per second).
    • no guarantee of ordering (Best-Effort ordering).
    • no guarantee of exactly-once processing, but guarantees of atleast-once processing.
    • Default retention period of the message is 4 days, maximum up to 14 days.
    • Producer send messages and Consumer poll messages.
    • Message is deleted after reading by the Consumer (if it will not delete it will be seen by other consumers)
    • Consumer can be on EC2 instances, on-premises or AWS Lambdas.
    • Message Visibility Timeout: After a message is polled by a consumer it will be invisible for other consumers for certain period of time and that period is called Message Visibility Timeout. By default its 30 seconds. In this period that consumer has to process and delete the message, else after that period it will go back to the queue and will be visible to other consumers and they can process it.
    • Receive Count: It says how many times the message has been polled and received by same or multiple consumers. Suppose a consumer polled a message but do not delete it, after 30 seconds it will be visible to other consumers and if someone poll it the Receive Count would be 2 now.
    • Long Polling: It is nothing but Polling. When a consumer polls for messages it can wait “optionally” if there is no message in the queue, this is called Long Polling. The wait time can be from 1 sec to 20 sec.
    • Dead Letter Queue: If a message could not be processed within a Message Visibility Timeout period it will go back to the queue. But how many time it can go back to the queue is defined by MaximumReceives attribute. After the message has received the MaximumReceives threshold it will go to another queue called Dead Letter Queue.
  2. FIFO:
    • limited throughput,
    • guarantee of ordering,
    • guarantee of exactly-once processing.
    • The naming of this queue should have an extension of “.fifo“

We can use SQS with Auto Scaling Group (ASG) for the scalability requirement.

AWS SNS (Simple Notification Service):

It is like a Topic, a typical Pub-Sub model where it will broadcast the message to everyone who all have subscribed/interested for it.
SNS does not need SQS to function. But you can integrate it with SQS, Lambda, Kinesis Data Firehose, etc.
“Event” Producer will send message to one SNS topic.
“Event” Subscribers has to listen to messages.

SQS + SNS (Fan Out) :

AWS MQ:

It is a managed message broker for Apache ActiveMQ.
It supports traditional JMS APIs.