Questions tagged [amazon-sqs]
Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers.
4,195
questions
0
votes
1
answer
16
views
Configure AWS SQS to take runtime exceptions into Dead Letter Queue
I am trying to integrate an Async Design with API Gateway -> SQS -> Lambda.
The gateway receives a request and via the SQS integration at gateway we are passing the messages to SQS. The SQS then ...
0
votes
0
answers
23
views
Can't able to resolve dependencies of the SqsService when i tried to use it in Custom Provider
I'm aiming to make the code less dependent on specific parts. My idea is to create a provider, which would allow me to easily switch between using a queueService or bypassing the queue during local ...
1
vote
0
answers
33
views
Connecting to Amazon SQS key (SSE-SQS) Encrypted SQS Queue via Django Celery [sqs]
I am unable to get my Celery working running while connecting to an SQS Encrypted Queue
This is my settings.py for my django project
SQS_AWS_ACCESS_KEY_ID = 'xxxx'
SQS_AWS_SECRET_ACCESS_KEY = 'xxxx'
...
-1
votes
1
answer
23
views
Is there a pricing for using multiple threads to send data to a AWS SQS queue at the exact same moment?
Let's say an HTTP request posts data to a AWS SQS queue. In case if there are like 300 of these requests at the same time or maybe 3000 requests at the same time. Is this chargeable like for more than ...
0
votes
0
answers
13
views
AWS API Gateway Mapping Template for form submissions dies with empty form values
I have an API Gateway, with the following mapping template:
application/x-www-form-urlencoded
#set($form=$input.path("$").split("&")) Action=SendMessage&MessageBody={#...
0
votes
2
answers
56
views
AWS: Only invoke lambda during a specific timeframe each day
I currently have a service that publishes a message to an SNS topic which has a subscription with name filter to add it to an SQS queue which ultimately triggers a lambda function.
I have a use-case ...
0
votes
1
answer
49
views
How to map request headers from ApiGatewayV2 to SQS event
I'm using dotnet framework 6.0, CDK v2 and ApiGatewayV2.Alpha
I'm trying to send messages from the API Gateway directly to an SQS queue. An integration for this does not yet exist in this version, so ...
1
vote
1
answer
39
views
AWS SNS Filtering based on the Message Body
Sample message looks like below:
{
"Type" : "Notification",
"MessageId" : "23456f-9cca-8f1a-5746363",
"Message" : "{\"Records\":[{...
0
votes
0
answers
15
views
AWS Webhook custom template results in 403 when going to sqs
I am running into a strange problem whilst trying to leveraging a custom mapping template in my webhook which has an integration with an SQS service.
As I understand it, the purpose of the mapping ...
0
votes
0
answers
16
views
SQS Min Batch size when triggering Lambda [duplicate]
I have an SQS queue that gets messages into it not very frequently. And I want to run a lambda when there are enough items in the queue to process (let's say 10).
Is there any way to trigger a lambda ...
-1
votes
1
answer
43
views
How to efficiently consume messages from an Amazon Simple Queue Service (SQS) in Golang? [closed]
I want to create an consumer in Golang but theres a few articles and info about it.
So i'm accepting any suggestion about how to do in a reliable way!
I tried using channels and some work arounds but ...
-1
votes
0
answers
17
views
What alternatives can provide the exact same features as AWS SQS?
I am currently using AWS SQS fifo queues with the message group id functionality to enforce ordering within subgroups. However, I have to find a replacement due to the limitation of Sqs where message ...
0
votes
0
answers
26
views
All the messages from SQS goes into DLQ even after successful execution of Lambda
Type: AWS::SQS::Queue
Properties:
QueueName: "notification-service-queue.fifo"
FifoQueue: true
ContentBasedDeduplication: true
VisibilityTimeout: 180
FifoThroughputLimit: ...
0
votes
0
answers
6
views
Optimizing Real-time IoT Data Processing: Harnessing the Power of SQS with Kinesis Data Streams
I'm in the midst of architecting a robust solution for real-time processing of IoT streaming data and would like to seek guidance on whether incorporating Amazon Simple Queue Service (SQS) in ...
0
votes
1
answer
36
views
retain message in sqs even after read
I've a Airflow Dag, which runs for multiple tables processing, but for one table at a time. Dag instances run in parallel if need to run more than one table processing.
My problem is, I need to send ...
0
votes
1
answer
42
views
Why should the SQS visibility timeout for Lambda bet set to 6?
The AWS doc states
To allow your function time to process each batch of records, set the
source queue's visibility timeout to at least six times the timeout
that you configure on your function. The ...
0
votes
0
answers
38
views
Spring boot SQS listener makes message visible if error/exception in processing
As per documentation , The visibility timeout begins when Amazon SQS returns a message. During this time, the consumer processes and deletes the message. However, if the consumer fails before deleting ...
0
votes
0
answers
19
views
Localstack throws The security token included in the request is invalid when it tries to send sqs message
Localstack throws The security token included in the request is invalid when it tries to send a message.
It seems like it tries to connect to real AWS.
Below is my configuration:
companion ...
0
votes
2
answers
40
views
Buffer of 30 mins between Lambda invocations
I want to invoke lambda multiple times but not concurrent with invocation having 30mins(some minutes) having event custom information & buffer in between 2 lambda triggers. So basically, 1 lambda ...
-1
votes
1
answer
23
views
How to allow anonymous access for SendMessage to Amazon SQS?
The AWS SQS docs states that anonymous access is supported: https://aws.amazon.com/sqs/faqs/#:~:text=Q%3A%20Does%20Amazon%20SQS%20support%20anonymous%20access%3F
It also provides an example policy ...
0
votes
0
answers
24
views
What is the correct syntax for Swagger 2.0 AWS requestParameters for MessageAttributes ? to map them from body to header
I'm not familiar enough with AWS to verify whether the syntax for mapping requestParameters is correct for AWS Swagger 2.0 config.
The message will be send to SQS with a body like this
{
"...
0
votes
1
answer
178
views
Getting error while using @aws-sdk/client-sqs in node.js
While using the sendMessage method of SQS getting the following error
error SyntaxError: Unexpected token T in JSON at position 0 Deserialization error: to see the raw response, inspect the hidden ...
1
vote
1
answer
52
views
SQS - Step function pipe failed
I am new to step functions and have a Pipe to poll SQS queue and initiate the step function execution per message received. The intention is to process one message at a time. However, the Step ...
-1
votes
1
answer
32
views
What is the purpose of an sqs in a stepfunction?
So I'm trying to wrap my head around what the purpose of having an sqs in a stepfunction is.
Lets say there's a lambda that sends messages to an queue, and then another lambda that consumes the ...
0
votes
1
answer
38
views
How to configure celery with SQS as backend?
I'm trying to setup a SQS broker with a celery app, configured in a django project.
Here's my setup:
celery.py:
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE&...
1
vote
2
answers
76
views
SQS Messages not deleted after DLQ Implementation
I recently implemented DLQ(Dead Letter Queue) for SQS. I have done the following three configurations.
Default visibility timeout in SQS.
In the Dead Letter Queue configuration, enable and Maximum ...
0
votes
1
answer
28
views
SQS subscribed to multiple SNS with different data models
Context
I have 2 SNS both publishing messages with different data model.
I want a lambda to get these messages and do some processing.
Creating a SQS between the SNS and Lambda is mandatory.
What i ...
0
votes
1
answer
48
views
How can I trigger a job after 15 minutes in Laravel?
I need to create a job that is going to be trigger after 15 minutes a post is created, I can set 15 minutes of delay, but imagine that the job is going to take 1 minute to process, in total will be 16 ...
1
vote
0
answers
14
views
AWS Textract Asynchronous Queue Messaging
I'm getting the following error when calling DeleteMessageCommand for my pdf processing.
error InvalidAddress: The address arn:aws:sns:us-west-1:**** is not valid for this endpoint.
Here's where the ...
0
votes
0
answers
35
views
How to remove some jobs from running in Laravel
I have one worker currently running, utilizing SQS with Laravel Jobs.
Within my jobs, I invoke third-party services to perform actions on my products. Each job corresponds to a specific product (e.g., ...
0
votes
0
answers
34
views
AWS SQS message push taking 50 secs
For some of the requests, sqsClient send message taking 50+ seconds to push message in SQS queue.
How to pass CLientConfiguration so that timeout happens quickly and accordingly retry happens.
Tried ...
0
votes
0
answers
44
views
Customize Request Timeout while publish message to SQS using java sdk - Reducing SQS Message Processing Time: Minimizing Retries after 10 Seconds
We are using SQS in our Java Spring Boot Project. While publishing messages to SQS, most of the messages are processed in <=300ms. However, some requests took more than 50 seconds. Upon initial ...
0
votes
0
answers
26
views
How to use AWS SQSExtendedClient library to make it work asynchrously using JMSListener java
Am trying to achieve exactly what the below link is trying to do.
Amazon SQS Async Listener for the Extended Client Library using Spring JmsListener
I couldn't find AmazonS3Client and AmazonSQSAsync ...
1
vote
0
answers
29
views
Job completion tracking across kafka
I have a distributed pipeline that reads jobs from a Queue (SQS). Each job then produces millions of messages and sent over to Kafka topic TopicA to analyze. After the analysis is done, it’s then sent ...
0
votes
1
answer
19
views
Will reducing retention period in AWS SQS boost the performance?
I have an application that has a Consumer and a Producer.
The Producer will push the messages to the SQS.
The Consumer is reading from the queue but reading the last message is taking too much time.
I ...
0
votes
2
answers
60
views
Serialize consumers of SQS fifo queue in node app
I am trying to implement serialization in the consumers of a SQS queue, which runs in multiple nodeJs instances. i.e. what I am trying to do is ensure that stuff is done in the same order they came in,...
0
votes
1
answer
51
views
How to send large number of mails (more than sending rate limit) at a time in AWS SES
I have an AWS SES service with a sending rate of 100 mails per second. Our application sends emails to all students within educational institutes. There's a need to distribute a substantial volume of ...
0
votes
1
answer
189
views
AWS SQS FIFO in Python - deleted messages are available even after correct VisbilityTimeout
I created an FIFO queue.
I send 10 msgs, receive them and delete them. But they are still available in queue and received again. I receive 10 msgs initially, then 5 , then 2 and then 0. "...
1
vote
0
answers
25
views
Overlook AWS credentials from env and instead use IAM role attached to node_group of EKS to push msgs to SQS
Problem: I have some downstream services which needs to push msgs to sqs using my python package but boto3 uses the credentials defined in their respect env instead of using IAM role of node_group.
I ...
0
votes
1
answer
93
views
Backend stack with microservice architecture
Currently I am migrating my CRM to microservice architecture. Is this arch suitable to be called solid and robust? (High loads are expected in future)
Main point of this architecture is one service - ...
0
votes
0
answers
21
views
Difference between 0 delay schedule and white true loop in SQS long polling Java
I'm writing a custom SQS long polling service.
Basically
Option-1
while(True) {
//long poll SQS for 20 seconds
// process messages
}
Option-2
using ScheduledExecutorService with scheduling of 0 delay....
0
votes
0
answers
25
views
How to get message payload on @SqsListener exception handling
I have a scenario where I want send notification in case of validation failure
public void onMessage(@Valid @Payload String jsonPayload,
@Headers Map<String, String> ...
0
votes
0
answers
19
views
How apache camel ensures idempotent messaging with SQS
Standard SQS queues guarantee that a message is delivered at least once and duplicates can be introduced into the queue. So we need to handle the scenario where in we receive same message multiple ...
1
vote
1
answer
26
views
SQSEvent object needed for SQS triggered lambda function in postman
I am trying to test my SQS triggered lambda function in my local machine using postman. I am able to call the function using postman but it requires SQSEvent type object as input. How do I provide ...
0
votes
0
answers
23
views
What is the use of endpoint in sns.subscribe in case of sns fan-out events
I'm working on a service that uses an SNS-SQS fanout model in Python. I understand that in order to send a notification to a specific SQS, I need to provide a filter policy. However, I'm confused ...
1
vote
1
answer
47
views
Auto-scaling group based on SQS depth
I am writing Terraform code that does the following: sets up S3 bucket and S3 event notifications, also Lambda that is being triggered by S3 notifications. Then it sends the messages to SQS. And then ...
0
votes
0
answers
38
views
Java SQS client throughput doesn't scale up in parallel
I want to poll and process SQS messages with high throughput. I have a number of threads, each with a SQS poller in a Spring @Async. At the moment the code is simply printing the message. My hope is ...
0
votes
1
answer
37
views
Some events are not available when the EventBridge rule triggers the push to SQS
I have an EventBridge rule that gets triggered on the insert and update of a specific item type. The data that satisfies the rule is then pushed to an SQS queue, which in turn triggers a Lambda ...
0
votes
0
answers
45
views
Celery worker is not starting for my django project hosted on elastic beanstalk
error in the eb consoleGood day by co-developers.
I have a project working on aws elasticbeanstalk and i want to integerate celery to it but the celery worker is not starting and everything is ...
0
votes
1
answer
6
views
qbo3 Import Running Multiple Times
I have queued a qbo3 API call importfile/import to bulk import a file with 80M records. The job completes successfully, but continues to run. The logging shows:
Queued @ 2:39:42 AM
Processing @ 2:39:...