Questions tagged [amazon-kinesis]
Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale.
1,813
questions
0
votes
1
answer
16
views
Localstack (AWS) credential failure on java lambda
The purpose of this lambda is insert a bucket content on a kinesis data stream, here is the code:
package productor;
import software.amazon.awssdk.services.s3.*;
import software.amazon.awssdk....
0
votes
1
answer
28
views
Unable to trigger Lambda from Kinesis Consumer
I have a lambda which consumes near-real time updates from a DynamoDB table through a Kinesis Data Stream. I also created a stream consumer from which the lambda consumes data. When I try to deploy ...
0
votes
0
answers
10
views
Is there a way to tell how far behind a Lambda event mapping for Kinesis is?
I have a Lambda subscribing to a Kinesis stream through an event source mapping.
My understanding is that the event source mapping stores the last processed record id (per shard) as a part of its ...
0
votes
0
answers
7
views
KCL 2.x is polling Kinesis stream at slower rate than configured
I have a KCL application where I am not setting any value for idleTimeBetweenReadsInMillis of PollingConfig class. So, it should be using default value set i.e. 1000ms (1sec).
This should poll 60 ...
0
votes
2
answers
42
views
Java 17 runtime kinesis lambda code doesnt work localstack (easy code)
Its very simple. This code works as jar and lambda (maven project):
package test2;
import java.util.Map;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda....
0
votes
1
answer
24
views
Amazon Kinesis, Lambda and DLQ - how to properly get payload contents from the DLQ in a large traffic scenario
Kinesis DLQ does not contain the payload, only the metadata. You're supposed to read the payload like in this SO issue by calling GetRecords.
My setup is as follows:
Apps --> Kinesis Data Stream --...
0
votes
0
answers
24
views
Integrating Spring with AWS Binder (Kinesis)
I'm trying to implement an integration solution with Kinesis (AWS), using "Stream Binder Kinesis". I need to send a json(String) to a queue (producer). I'm based on the example https://www....
0
votes
1
answer
18
views
AWS Kinesis long startup waiting
I'm using AWS Kinesis with spring cloud to read data from a data event hub. The deploy end with the following log:
2023-09-26 14:07:30.476 INFO [,,] 9331 --- [esis-consumer-1] a.i.k....
0
votes
1
answer
38
views
Creating a Kinesis Stream on existing DynamoDB table
I already have an existing DynamoDB table when regularly gets updated with data from an upstream source. How can I create a new Kinesis stream to consume updates from the table in AWS CDK?
From other ...
1
vote
1
answer
46
views
Spring Cloud Stream Kinesis Binder 4.0 doesn't support Multi Stream with KCL 2 Enabled
KCL 2 allows multi stream support with the same consumer group in the same application.
With cloud stream kinesis binder 4.0, kpl-kcl-enabled: true A single stream in the destination works.
bindings:
...
0
votes
1
answer
23
views
Adding Kinesis Streams to a DynamoDB Global Table
I have a DynamoDB global table with replicas in two regions, and I want to enable Kinesis Data Streaming. For any write to either table I want only one Kinesis event. What is the suggested approach to ...
0
votes
0
answers
15
views
Push events data to AWS Kinesis Firehose via web client
I have a very simple use case of pushing events (click events) from a web browser and storing the data to S3 using AWS kinesis. I'm a bit new to streaming data ingestion so I don't understand the ...
0
votes
1
answer
21
views
Do not see data in AWS Kinesis
I created a stream in AWS Kinesis, straightforward:
I put records into the stream (I took the script from AWS documentation):
import datetime
import json
import random
import boto3
STREAM_NAME = &...
0
votes
0
answers
12
views
Lambda Max Concurrency in Monitoring does not match the configured concurrency in kinesis trigger
I have a Lambda function with Kinesis as the triggers. The kinesis stream only have one shard, and the trigger is configured with Concurrent batches per shard: 3.
But, when I checked the Lambda ...
0
votes
0
answers
22
views
How to write data to Kinesis Data Stream with Apache Beam using Python
I am trying to build Beam pipeline in Python that reads from and writes to AWS Kinesis Data Streams.
I am using the FlinkRunner, with Flink 1.15.4 running locally on my machine.
Python 3.8.17
Apache-...
0
votes
0
answers
19
views
AWS Glue crawler for Athena query is not showing all the records that i have in s3 bucket
I have a c# application that logs audit records to Kinesis Firehose so those get PUT into a S3 bucket. That is working perfectly because I can see the files in the S3 bucket.
I'm using Dynamic ...
0
votes
0
answers
18
views
Unable to install Amazon Kinesis Video Streams Producer SDK Java
I've followed this tutorial but I'm not able to build this sdk, I'm getting the following error:
Exception in thread "main" 01:34:40.783 [main] ERROR com.amazonaws.kinesisvideo.java.client....
0
votes
1
answer
33
views
Kinesis KCL not pulling records from all shards
I've configured the Kinesis stream with 4 shards and implemented the KCL library version-1 as a consumer. The consumer pulling records from one shard only (shardId-000000000000). I've also noticed if ...
0
votes
0
answers
41
views
Fluent Bit configuration for separating and routing application logs to distinct locations on AWS EKS Fargate
I'm currently working with AWS EKS Fargate and facing challenges with effectively sending application logs to an AWS S3 bucket. I'm utilizing Fluent Bit as a log router, which routes logs to a Kinesis ...
0
votes
0
answers
28
views
Data output costs in Kinesis Data Stream when connecting many eventbridge pipes
If we connect two EventBridge Pipes to a kinesis Data Stream (Customer Orders) then the cost of data output from the kinesis doubles? and so on if we add more Eventbridge Pipes to Kinesis?
Link to ...
0
votes
0
answers
49
views
Can Amazon Kinesis Firehose deliver to HTTP endpoint configured with a self-signed SSL certificate?
In an Amazon Connect instance, I set up Data Streaming such that Contact Trace Records and Agent Events are published to the same Kinesis Stream.
Then I created a delivery mechanism of said events by ...
0
votes
1
answer
81
views
Why doesn't Kinesis Streams Data Viewer show any Agent Events after call through Amazon Connect contact Flow?
QUESTION: Why don't I see Agent Events in the Data Viewer for the Kinesis Stream I attached to an instance of Amazon Connect?
Here is what I did:
Created a Kinesis Stream. Call it amzn-conn-agent-...
0
votes
0
answers
11
views
AWS KCL MultiLanguageDaemon - override Dynamo table name used for checkpointing
According to the following article it's possible to override the the DynamoDB table used for checkpointing when using the KCL directly.
Here's an example of how to set up a table name on KCL version ...
0
votes
2
answers
54
views
Why is name of stream for Kinesis Video Stream (triggered from Amazon Connect) not recognized?
In Amazon Connect, I create a Flow with Start Streaming Data block. I have verified that calls trigger live audio streams through Kinesis Video Streams. See the ARNs below (captured by calling AWS ...
0
votes
1
answer
26
views
Amazon.IonDotnet.Internals.Text.InvalidTokenException error while reading stream data from AWS Kinesis by QLDB stream
I am not sure if I can find a solution for this, but if is there anybody doing development with AWS QLDB please ping me.
Anyway, my problem is:
I have a running solution doing some operation on AWS ...
1
vote
1
answer
51
views
Log more data from Kafka Source Connector
I installed the Kinesis Connector plugin in AWS, to connect my Kinesis Stream with my MSK cluster. It is working, but the logging that I see in Cloudwatch is not very helpful:
[Worker-...
3
votes
1
answer
159
views
Cloudwatch Subscription Filter does not ingest to Kinesis Data Stream
I am trying to forward Cloudwatch logs to Kinesis Data Stream using the Subscription filter.
Even with broad permissions, for some reason, I am not able to see any records in Kinesis.
I could see logs ...
0
votes
0
answers
23
views
Kinesis Confluent Connector not reaching MSK Cluster
I am trying to add a Kinesis connector to connect my Kinesis Stream with my AWS MSK Cluster, so that I can send messages from the Kinesis to a certain topic in the Kafka cluster.
This is the connector ...
0
votes
0
answers
35
views
How to stream RTSP video stream using AWS KVS to browser
I have a source which generates RTSP stream. I am sending the RTP packets to Amazon Kinesis Video Stream. My client is a web browser were I want to play the video stream. How can I relay the stream ...
0
votes
1
answer
41
views
Are Kinesis sequence numbers unique across the stream or just per shard and per partition key?
The documentation is conflicting in this regard.
The PutRecords documentation says:
The SequenceNumber parameter is an identifier assigned to the put record, unique to all records in the stream.
...
1
vote
1
answer
185
views
No data on AWS Kinesis after putRecords
Disclaimer: This is the first time I am using AWS Kinesis, so I might be expecting something that is not correct.
I have a very simple AWS Lambda function that inserts data into Kinesis. The API ...
0
votes
1
answer
63
views
Best way to process and store real time MQTT data in AWS Cloud
I need to receive the real-time MQTT data, get them stored in AWS Cloud, and provide the insights using these data in a separate web-based application. In this requirement, would that be better to ...
0
votes
1
answer
26
views
How to add multiple consumer of same data stream in Flink
I am struggling with implementing solution where the problem is as below.
There is one Kinesis stream named sushistream, event comes from the stream would contains two things, i.e. broadcastId and ...
1
vote
1
answer
10
views
KPL amazon Library only works when debugging
I'm having an awkward behaviour when trying to send data to amazon kinesis data streams.
what happens is that i'm just sending one record to test but it doesn't work and when i normally debug it or ...
0
votes
0
answers
23
views
High CPU Consumption using Cassandra Sink in Kinesis Analytics
I'm trying to set up a Kinesis Analytics application to write data from a Kinesis Stream to Keyspaces. To do this, I'm using flink's cassandra connector.
The application receives messages from a ...
0
votes
1
answer
28
views
Error while using spring-cloud-bus with aws kinesis stream binder
I want to use the actuator/bus-refresh for signaling my config change across my containers and for that I use:
spring.cloud.bus.enabled=true
management.endpoints.web.exposure.include=bus-refresh
and ...
0
votes
0
answers
29
views
kinesis-client-library is not reading from all the shards
I've amazon kinesis data streams with 16 shards with shardId starting from shardId-000000000012 to shardId-000000000027.
I also have kinesis client library with same region, app-name, stream-name and ...
2
votes
2
answers
194
views
Spark streaming forEachBatch giving inconsistent/unordered result while writing to database
Problem:
I am receiving multiple table/schema data in a single stream.
Now after segregating the data I am opening a parallel write stream for each table.
The function I used in forEachBatch is:
def ...
0
votes
0
answers
45
views
How do I provide a session token when using FlinkKinesisConsumer in Java?
I'm trying to use user A to assume role R1, and using R1 assume R2 in a different account. I have gone as far as creating an StsClient to get the accessKeyId, secretKeyId, and sessionToken that I need,...
1
vote
0
answers
14
views
kinesis connected to multiple shards - wont fully disconnect again
i have an issue where, if i connect to more than one shard, and call client.destroy(), the script keeps running, where i expect it to stop running, as no more connections should stay open.
im using @...
0
votes
1
answer
55
views
Add tags to s3 objects dynamically to apply s3 life cycle rules
I have below s3 lifecycle rule created for existing s3 bucket. It should not be changed.
I am sending the logs from kinesis firehose to s3 bucket as parquet formatted objects. Is there any way or ...
0
votes
0
answers
16
views
How to write to Kinesis stream from Spark3?
How do I write to a Kinesis stream from Spark3? We were previously doing this in Spark 2.4.0 while using the DataSourceV2 class and related classes/APIs as a way to connect - wondering what the ...
0
votes
0
answers
12
views
What impacts aggregation in KPL for a Spark job?
I run Spark jobs to write to Kinesis via KPL, and I recently upgraded my KPL from 1.8.10 to 1.14.x. Previously, my job resulted in 1 Kinesis record with 20 user records. After the upgrade, there are ...
0
votes
0
answers
26
views
Equivalent of aws.sqs.approximate_number_of_messages_visible in Kinesis
I know that the aws.sqs.approximate_number_of_messages_visible is the 'The number of messages available for retrieval from the queue'. What would the equivalent be if I am to use Kinesis and wanting ...
0
votes
0
answers
21
views
KPL upgrade and batching differences
I upgraded KPL from version 1.8.10 to 1.14.x and noticed that the data being produced by my Spark job results in smaller Kinesis records (less user records per batch). There is no data quality issue (...
0
votes
0
answers
117
views
AWS DMS: Replication Instance Fails to Connect to Kinesis Target Endpoint
I am working on setting up an AWS CloudFormation stack to deploy a database using Aurora Postgres Serverless V2, some network resources, a Jumpbox, and some AWS DMS resources to use real-time CDC as ...
0
votes
0
answers
73
views
Apache Flink on python UDFs are not working. Despite following AWS repo on pyflink UDFs
I see AWS has a Kinesis-DataAnalytics stream with Apache Flink. And on the pyflink library for python there is a way to create UDF. In fact, AWS seems to have a repo with examples for pyflink UDF.
I ...
2
votes
1
answer
66
views
RDS MySQL to Kinesis data stream pipeline using AWS Lambda
I am trying to extract data from RDS MySQL instance and load into kinesis data stream using put_record boto3 API. The connection using pymysql is working and I am able to print the table but, I cannot ...
0
votes
0
answers
166
views
AWS Cloudwatch -> Kinesis -> OpenSearch
I setup a Cloudwatch subscription filter to a Kinesis data firehose, with a destination of an OpenSearch instance and I'm using the "Process CloudWatch logs sent to Kinesis Firehose" ...
0
votes
1
answer
72
views
AWS remote video not loading (viewer)
amazon-kinesis-video-streams-webrtc-sdk-js
video stream works on local (ie. viewer and master on same machine or browser with different tabs ) but when viewer and master is on different machine (ie. ...