Job Queue
Resource Overview
A Job Queue represents an Amazon SQS queue. Amazon Simple Queue Service (SQS) allows you to host a pull-based message queue for services in your distributed systems to poll for messages. These queues are used in a serverless architecture to provide resilient, scalable messaging.
Key Features of Amazon SQS include:
- Fully managed by AWS
- Cost-effective scaling on demand
- Uses server side encryption (SSE) to encrypt all your data
A Job Queue differs from a Topic, in the way that the consuming resource receives a message. A Job Queue requires their consumer resources to poll or "pull" messages from a queue, where as a Topic "pushes" messages instantaneously to subscribed consumer resources.
Event Subscription
Event subscription wires (solid line) visualize and configure event subscription integrations between two resources.
The following resources can be subscribed to a Job Queue:
- Function
Service Discovery
Service discovery wires (dashed line) provide compute resources (Function, Edge Function, Docker Task) with the permissions and environment variables required to perform actions using cloud resources within the stack. This resource is can be on the receiving end of a service discovery wire originating from compute resources.
The following compute resources can use a service discovery wire to access a Job Queue resource:
- Function
Configurable Properties
Logical ID
The unique identifier used to reference this resource in the stack template. Defining a custom Logical ID is recommended, as it allows you to quickly identify a resource and any associated sub-resources when working with your stack in AWS, or anywhere outside of the Stackery Dashboard. As a project grows, it becomes useful in quickly spotting this resource in template.yaml or while viewing a stack in Template View mode.
The Logical ID of all sub-resources associated with this Queue will be prefixed with this value.
The identifier you provide must only contain alphanumeric characters (A-Za-z0-9) and be unique within the stack.
Default Logical ID Example: Queue2
IMPORTANT : AWS uses the Logical ID of each resource to coordinate and apply updates to the stack when deployed. On any update of a resource's logical ID (or any modification that results in one), CloudFormation will delete the currently deployed resource and create a new one in it's place when the updated stack is deployed.
FIFO Queue
When enabled, this Job Queue treats messages on a first-in, first-out basis, meaning messages are delivered to subscribers in the same order they are received.
Message Retention Period
The amount of time (in seconds) that a message is retained in the Job Queue before expiring.
Use Existing SQS Queue
When enabled, this feature provides you with a field to specify the Amazon Resource Name (ARN) of an existing SQS Queue to reference in your application.
You may reference an environment parameter in order to conditionally reference existing infrastructure based on environment.
Subscription
Subscribe a Function resource to receive messages stored in the Job Queue.
Batch Size
The maximum number of messages to include when triggering a subscribed Function resource.
DLQ
Subscribe another Job Queue resource to act as a dead-letter queue to handle message failures. Note that the types of a queue and its dead-letter queue must match (standard or FIFO).
Maximum Attempts
The maximum number of attempts to deliver a message before sending it to a dead-letter queue.
IAM Permissions
When connected by a service discovery wire (dashed wire), a Function or Docker Task will add the following IAM statement/policy to its role and gain permission to access this resource.
SQSSendMessagePolicy
Grants a Function or Docker Task permission to send messages to your Job Queue in the stack, or an existing SQS Queue outside of the current stack.
Environment Variables
When connected by a service discovery wire (dashed wire), a Function or Docker Task will automatically populate and reference the following environment variables in order to interact with this resource.
QUEUE_NAME
The Logical ID of the Queue resource.
Example: JobQueue2
QUEUE_ARN
The Amazon Resource Name of your Queue.
Example: arn:aws:sqs:us-east-2:123456789012:JobQueue2
QUEUE_URL
The URL of your Queue.
Example: https://sqs.us-east-2.amazonaws.com/123456789012/JobQueue2
AWS SDK Code Example
Language-specific examples of AWS SDK calls using the environment variables discussed above.
Add a new job to a Queue
// Load AWS SDK and create a new SQS object
const AWS = require("aws-sdk");
const sqs = new AWS.SQS();
const url = process.env.QUEUE_URL; // supplied by Function service-discovery wire
const queueName = process.env.QUEUE_NAME;
exports.handler = async message => {
// Construct parameters for the sendMessage call
const params = {
MessageBody: 'New Job',
QueueUrl: url
};
await sqs.sendMessage(params).promise();
console.log('Job sent to queue: ' + queueName);
}
import boto3
import os
# Create an SQS client
sqs = boto3.client('sqs')
queue_url = os.environ['QUEUE_URL'] # Supplied by Function service-discovery wire
def handler(message, context):
notification = "Hello All! \n We've made quite a few changes to our product...";
# Send a new message/job to the specified SQS Queue
response = sqs.send_message(
QueueUrl=queue_url,
MessageBody='New Job'
)
return response
Related AWS Documentation
AWS Documentation: AWS::SQS::Queue