Amazon Simple Queue Service (SQS) is a fully managed message queuing service that makes it easy to decouple application components, each performing a discrete function which improves scalability and reliability and is best practice design for modern applications. Amazon Simple Queue Service (Amazon SQS) is an important topic to study for the AWS Certification Exams including the AWS Certified Solutions Architect Associate and the AWS Certified SysOps Administrator Certifications.

Amazon Simple Queue Service (Amazon SQS) is a:

Fully managed queuing service that enables you to build solutions designed for high availability and fault tolerance.
Offers a secure, durable, and highly available hosted queue that lets you integrate and decouple distributed software systems and components. You don’t; need to build your own messaging system which comes with overheads such as having to manage software and hardware infrastructure. Being a managed service, this is all taken care of by Amazon
Decoupled components of your application are not depended on each other and can run and fail independently. When building systems in the cloud, very often you will need to design multi-tier architecture for the different components of your application. Using Amazon SQS you can decouple the application components to offer higher levels of resilience, fault tolerance and availability and build a distributed solution.

You can access and configure the Amazon SQS service using the Amazon SQS Console, the AWS Command Line Interface (CLI) and the Amazon Software Development.

Use Case

Standard Architecture

Consider a scenario where you have an order processing system. Following best practices, you design a multi-tier architecture as depicted in the following diagram which would involve implementing load balancing, dynamic scaling across multiple availability zones and a backend database solution using Amazon RDS configured with Multi-AZ.

Amazon SQS

While the above architecture lends itself well to being fault tolerant and enabling services to be highly available, a potential issue can arise if incoming orders are not properly committed to the database due to any bottlenecks such as network issues. Orders that not properly processed could be lost resulting in customer complaints and loss to the business.

Amazon SQS can be introduced in this architecture to improve the ordering process. Using a queue enables orders to be stored as messages, which can then be held temporarily should outages occur due to any bottlenecks. Once systems are back up and running again, messages (orders) can be retrieved from the queue and processed along, thus creating a more resilient environment to ensure no orders are lost.

Depending on the volume of incoming orders, Auto Scaling services in the App Tier can launch or terminate EC2 instances in response to CloudWatch triggers, thus ensuring maximum availability while being cost effective. With this new design, you are now able to scale the web application frontend independently from the processing nodes in the backend. The frontend application can continue to scale based on metrics such as CPU usage, or the number of requests hitting the load balancer. Processing nodes can scale based on the number of orders in the queue.

Furthermore, if there are any transaction failures, you also have a mechanism to move them to another queue; the Deal Letter Queue for further analysis and debugging while still ensuring that your normal order processes are allowed to continue.

Adapting to Change

With Amazon SQS in the mix, you can now easily adapt to changes in business requirements. For example, you can use Amazon SQS to create multiple queues to enable prioritization of your ordering process. Imagine if you wish to implement a priority system where orders placed over $1000 are to be handled with higher priority, you could introduce a new “priority” queue. This will not disrupt the standard workflow process between the Web Tier and the App Tier other than ensuring that orders that enter the high priority queue get processed before those entering the standard queue.

Using Amazon SNS with Amazon SQS

Amazon Simple Notification Service (Amazon SNS) is a push-based messaging system that can be not only used to send out alerts such as email notification when components fail but also used to send the message to different SQS queues. For our example, this can enable parallel asynchronous processing of customer orders which can then be extended to other related applications to support multiple related business use cases. This process of extending a message out to multiple parallel queues is also known as a “fanout” scenario.

As shown in the diagram below, customer orders coming in from the frontend web servers can be sent as SNS messages to a topic and then replicated and pushed to multiple SQS queues and Lambda functions for processing.

Amazon SQS – Key Points

Amazon SQS allows you to decouple different components of an application so that they can work independently of each other.
An SQS queue can enable one application component to send the message to another application component, effectively acting as a buffer to ensure components don’t get overloaded. For example, web servers sending too many requests to backend application servers can lead to system failures. A queue system will allow you to hold messages for a short duration of time, so that the application servers and retrieve and process them when there is capacity
Amazon SQS is designed to deliver messages ‘at least once’
Amazon SQS offers Standard and FIFO Queues. We’ll discuss the difference in a few slides
Amazon SQS supports multiple writers and readers interacting with the same queue
Amazon SQS is highly scalable and designed to be always available due to the fact that it is distributed across multiple nodes and data centre facilities
Messages can be held in the queue for a maximum of 14 days; the default setting is four days.

Asynchronous Linking

Amazon SQS enables you to decouple systems by using queues to exchange messages containing task between components of the system. This also enables you to use services like CloudWatch and SNS to further design highly cost-effective solutions. When there are a large number of messages, you can increase the number of EC2 instances to retrieve and process those messages using Auto Scaling and then when there are no messages to process, you can Auto Scaling to terminate the excess server capacity saving on costs.

At Least Once Delivery

Amazon SQS will store copies of messages across multiple servers and multiple facilities to offer redundancy and high availability. It is possible that a server hosting a copy of the message might be unavailable when you receive or delete a message. This can result in an application component retrieving a message that was already processed again. To ensure that this does not affect your application adversely, ensure that you design your solution to be idempotent.

Short Polling Vs Long Polling

Short Polling involves sampling a subset of servers from which messages are retrieved. Thus some messages may still remain in the queue (on different servers that we not polled in the initial request). Subsequent polling will then retrieve all the remaining messages. However, the downside to this is that Short Polling can be CPU intensive for your application
Long Polling helps to reduce the cost of using Amazon SQS by eliminating the number of empty responses and allowing Amazon SQS to wait until a message is available in the queue before sending a response. You need to set the ReceiveMessage wait time to a value large than 0 and unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage Note the maximum long poll timeout is 20 seconds

Visibility Timeout Period

This is the period when a message is received by an application component, till when it is processed and deleted from the queue. During this period, the message is in a locked state (made invisible) to the rest of the application so that no two consumers try to receive and process the same message. Amazon SQS needs to have a visibility timeout period due to its nature of being highly distributed as it cannot guarantee that the consume actually receives the message (for example due to connectivity or component failure issues). If a component fails to process the message after receiving it, the visibility timeout will expire and another component can receive and process the message

The default visibility timeout for a message is 30 seconds. The maximum is 12 hours. Note: when an application needs more time to process a message, the visibility timeout can be changed using the ChangeMessageVisibility operation dynamically.

Example – For example, you have a message with a visibility timeout of 5 minutes. After 3 minutes, you call ChangeMessageVisibility with a timeout of 10 minutes. You can continue to call ChangeMessageVisibility to extend the visibility timeout to a maximum of 12 hours. If you try to extend the visibility timeout beyond 12 hours, your request is rejected.

Types of Queues

Standard Queue

Amazon SQS standard queue is the default queue used with SQS. It supports an unlimited number of transactions (TPS) per action and ‘at-least-once’ message delivery. However, with Standard queues it is possible that more than one copy of a message can be delivered. Furthermore, Standard queues do not guarantee the ordering of messages and offer best-effort ordering which in most cases means that the messages are delivered in the order they are sent. However, your application has to be intelligent enough to handle the occasional out of sequence message. This can be done by adding sequencing information in each message so you can reorder the messages when they’re received. Use Cases

Decouple user’s interaction with frontend web servers from backend intensive workloads
Allocate tasks to multiple worker nodes, for example, process a high number of credit card transactions
Batch messages for future processing, for example, schedule multiple records to be added to a database

FIFO Queue

Offering the set of services as SQS Standard queues, FIFO queues also ensure the order of message delivery is preserved and are the queue of choice when out of order messages cannot be tolerated. For example, you need to ensure that customers create an account before placing an order.

FIFO queues also offer exactly-once processing meaning that no duplicates of messages are introduced. However, FIFO queues have certain limitations including:

Support for up to 300 messages per second without batching
Support for up to 3,000 messages per second with batching

Typical Use Cases include:

Ensure that user entered commands are executed in the right order
Display correct price information by sending any price modification in the right order
Prevent a student from enrolling in a course before registering for an account

Dead Letter Queue

A Dead Letter Queue is a special queue designed to help you isolate messages that could not be processed. Source Queues can then target and send messages that can’t be processed to the Dead Letter Queue.

A redrive policy specifies the source queue, the dead-letter queue, and the conditions under which Amazon SQS moves messages if the consumer of the source queue fails to process a message a specified number of times. For example, if the source queue has a redrive policy with maxReceiveCount set to 5, and the consumer of the source queue receives a message 5 times without ever deleting it, Amazon SQS moves the message to the dead-letter queue.

Key Points;

The dead-letter queue of a FIFO queue must also be a FIFO queue
The dead-letter queue of a Standard queue must also be a Standard queue.
Dead-letter queues must be created in the same region as the other queues that use the dead-letter queue
The expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp remains unchanged. – For example, if a message spends 2 days in the original queue before being moved to a dead-letter queue, and the retention period of the dead-letter queue is set to 4 days, the message is deleted from the dead-letter queue after 2 days. So you need to ensure that the retention period of a dead-letter queue is set to be longer than the retention period of the original queue.

Shared Queue

You can also associate an access policy statement with specific permissions to be granted with a message queue that you wish to share. Amazon SQS enables you to issue access policy statements like

AddPermission
RemovePermission
SetQueueAttributes
GetQueueAttributes

Note that SQS in each region is totally independent in message stores and queue names and you cannot share queues between regions. Within a region, you can share a queue with other AWS Accounts or even anonymously. Also, note that the queue owner pays for the shared message queue access.

740 Practice Exam Questions – Get Prepared for your Exam Day!

Our Exam Simulator with 740 practice exam questions comes with comprehensive explanations that will help you prepare for one of the most sought-after IT Certifications of the year. Register Today and start preparing for your AWS Certified Solutions Architect – Associate Exam.

Amazon Simple Queue Service (Amazon SQS) Exam Tips – IaaS Academy