Questions tagged [amazon-kinesis]
Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale.
1,628
questions
0
votes
1
answer
24
views
How to consume temporary AWS credentials in spring boot?
So I am working on spring boot application from which I am expected to access AWS resources, I know how to access the AWS resources via IAM credentials and STS credentials, I am looking for an example ...
0
votes
0
answers
12
views
How to reprocess failed events from Kinesis?
I have a lambda producer which putRecords to a kinesis steam. Sometimes while writing to kinesis I get Internal service failure. What is the best way to handle such cases where lambda fails to write ...
0
votes
0
answers
15
views
No downtime switching DynamoDB CDC KinesisStreamingDestinations
Ok so, I have an event-based distributed system with a lot of tables with streams connected for CDC.
What I need is a way of switching the streams those tables are connected to.
Suppose I have:
...
0
votes
0
answers
19
views
Error writing single record to Kinesis Data Stream
An error occurred (ResourceNotFoundException) when calling the PutRecord operation: Stream StreamOne under account not found.
The Stream is Active, so I am confused by this error?
The snippet I am ...
2
votes
1
answer
28
views
Run Athena every 15 minutes vs Kinesis Data Analytics
I am going to be using Athena for report generation on data available in S3. A lot of it is time series data coming from IoT devices.
Users can request reports over years and years' worth of data but ...
0
votes
0
answers
8
views
Aggregates on streaming data for reporting
I am streaming data from a Kinesis stream that provides measurements from IoT devices in a multi-tenant application. For the selected devices, we need to report aggregates such as:
Min
Max
Average
...
0
votes
0
answers
16
views
Kinesis - Random guid partition keys only using a subset of shards
The problem I am trying to solve is a relatively even distribution of records across shards on a kinesis stream.
I keep finding documentation and answers to other questions saying that if you don't ...
0
votes
1
answer
20
views
Connect AWS Kinesis Data Stream with Azure Function App
How to connect Kinesis Data Stream with Azure Function App as consumer, without using intermediary services?
1
vote
0
answers
33
views
How to show live video stream from thousands of cameras with some deep learning processing on AWS?
I want to run some machine learning models (like object detection) on a live video stream from multiple cameras ( thousands of cameras) and show all cameras' playback in a web application where the ...
0
votes
0
answers
35
views
How to properly decode AWS Kinesis stream data aside from utf-8 and latin-1
I'm currently working with a Kinesis stream that I am trying to parse data from. I have no knowledge of the data being streamed or the format of it, so I am trying to find that out. I need to know the ...
1
vote
0
answers
14
views
How to achieve dynamic fair processing between batch tasks?
Our use case is that our system supports scheduling multiple multi-channel send jobs at any time. Multi-channel send meaning send emails, send notifications, send sms etc.
How it currently works is we ...
0
votes
1
answer
13
views
Kinesis input stream into Logstash dynamodb creds
I m looking to get logstash input from kinesis but getting dynamodb errors
https://github.com/logstash-plugins/logstash-input-kinesis#authentication
auth is defined in ~/.aws/credentials and user ...
0
votes
0
answers
19
views
Can't connect EMR Pyspark to Kinesis Stream - NoClassDefFoundError
I'm using Pyspark on EMR and trying to connect to Kinesis as per the guide here:
https://spark.apache.org/docs/latest/streaming-kinesis-integration.html
I've been running:
spark-submit --jars '/usr/...
2
votes
0
answers
24
views
Flink job can not produce record to kinesis stream after changing FlinkKinesisProducer to KinesisStreamsSink
I have one Flink job integration test that will produce some records to a localstack kinesis stream. When I use Flink 1.14.4, I have a FlinkKinesisProduer writing to a kinesis stream that localstack ...
0
votes
1
answer
57
views
Streaming data from Kinesis to S3 fails with Illegal Character that KPL itself writes
I have a relatively straightforward use case:
Read Avro data from a Kafka topic
Use KPL (v0.14.12) to send this data to Kinesis Data Streams
Use Kinesis Firehose to transform this data into Parquet ...
0
votes
1
answer
32
views
Why Kinesis stream calls my Lambda function more than one time?
I am consuming Amazon Connect CTRs through Amazon Kinesis and inserting my data into Postgres. I am facing very unexpected behavior from Kinesis and Lambda function. Whenever a CTR record comes ...
0
votes
0
answers
13
views
How to handle update,delete and insert in a single window for apache hudi
I am new to pyspark streaming.So any help would be appreciated.
I have a cdc data coming from kinesis data streams.I was able to store update, delete and insert
events into apache hudi, but when i ...
0
votes
1
answer
23
views
Flink KPL doesn't seem to deliver to second shard
So I have the following code where I am trying to get the KPL to set the partition key, so I can start sharding my stream.
def createSinkFromStaticConfig(stream: Option[String], region: Option[...
0
votes
2
answers
55
views
How do I export aws lambda logs(Prints) to Kinesis Data Streams?
I have been Stocked on how to send Lambda logs(Prints) directly to Amazon Kinesis Data Stream. I have Found the way to send Logs from Cloud watch but I would like to send every single prints to ...
0
votes
1
answer
19
views
What should the value be of a Kinesis Partition key
So, I thought this would be easy to answer, but my google skills are coming up dry.
I have a flink producer that is currently setting the default Paritition to "0". Now that I want to ...
-1
votes
0
answers
45
views
AWS Kinesis Python: Determine if a stream exists and if not create new stream
I have a Python application which writes data to a Kinesis stream.
client.put_record(
StreamName = args.stream_name, #stream name provided as positional argument
Data = json....
0
votes
0
answers
21
views
Sending messages from a file to Kinesis using Spark
We've files containing millions of lines. I need to read each line from the file & send it to Kinesis. I am trying following code:
KinesisAsyncClient kinesisClient = KinesisClientUtil....
0
votes
0
answers
20
views
Flink Kinesis Producer failing without proper error
EMR Release: emr-5.35.0
Java: 1.8
Flink: 1.14.2
I have been trying to deploy an application to the flink in the EMR. This application runs fine locally, reading data from a kinesis stream, aggregating ...
1
vote
1
answer
22
views
Kinesis triggered lambda runs below expected number of times
I have a lambda configured to be triggered by a Kinesis stream at a batch size of 100 and batch window of 60s. The Kinesis stream has about 2 shards and 2K+ total records per minute. I'd imagine this ...
0
votes
0
answers
14
views
Can the AWS Kinesis Data Generator be run from a CLI/API?
I'm using the AWS Kinesis Data Generator (https://awslabs.github.io/amazon-kinesis-data-generator/web/help.html) to populate a Kinesis stream with test data, but was wondering if anyone is aware of an ...
0
votes
1
answer
27
views
filter kinesis event by lambda
i have looked at : https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html
to filter events before invoking lambda but i can't achieve that, can you help please?
{'Records': [{'...
0
votes
1
answer
45
views
Kinesis data stream skipping records with exception at downstream
I have an application with the following set up.
Kinesis data stream (retention period: 1day) ->
StreamExecutionEnvironment
.getExecutionEnvironment()
.addSource(new FlinkKinesisConsumer)
.map(new ...
0
votes
0
answers
18
views
Synchronize Data Via Local Debezium Server From MariaDB to AWS RDS in Real Time
We have hundreds of clients that each have either a MySQL or MariaDB database running on a Windows machine on their local networks. I need to relay updates from these databases to our multi-tenant ...
0
votes
0
answers
54
views
Local AWS Lambda + Kinesis - setting up event source
I have Kinesis streams set up locally with Localstack, and Lambda (in Python) set up locally with Serverless Offline. I cannot set up event source between them due to 404 and 500 errors.
Kinesis is ...
2
votes
0
answers
78
views
Why does the kinesis shard iterator falls behind when using BoundedOutOfOrdernessTimestampExtractor
I'm using KDA with a flink job which should analyse messages emitted by a different IOT device sources. There is a kinesis stream with 4 shards with each of them contains more or less the same amount ...
0
votes
0
answers
20
views
ETL product that get data from streaming services and send to application
I'm looking for library/framework/product recommendation. So the use case is to get data from streaming services, such as AWS Kinesis/Confluent Cloud/AWS Managed Kafka/Azure Eventhubs, and do some ...
1
vote
1
answer
33
views
Is there a way to recover Kinesis Data if Lambda function fails?
Context: I'm scraping data from a third party source using a lambda function (this lambda function is invoked my a cloudwatch event bridge so its asynchronous) then writing that data to kinesis ...
0
votes
1
answer
51
views
DynamoDB table was created for storing checkpoints
I'm trying to run a trivial steaming application on AWS EMR.
In my local machine, everything works as expected.
However, when I'm running my example in AWS EMR cluster, I'm getting empty RDDs.
...
0
votes
0
answers
22
views
Send S3 avro files to Kinesis stream
I'm very new to this so I ask. Every hour I receive .avro files in my S3 bucket. I need to push these files to Kinesis stream in order to build Flink Application on top.
I was trying to use DMS where ...
0
votes
0
answers
25
views
Best way to send data to an SNS topic from Dyanmodb for 2 cases - existing data and any new incoming data with low latency
Trying to design a workflow that can send data from a dynamodb to an sns topic.
I need to make sure that latency is as low as possible.
Currently looking into if kinesis or dynamodb streams can help ...
0
votes
0
answers
37
views
Repository returning null when autowiring in Kinesis KCL Consumer
I'm currently working with AWS Kinesis using the KCL Library. I can consume the records and print them with println. But when I try to call a repository class, it returns null (I'm autowiring it).
...
0
votes
1
answer
42
views
SpringBoot actuator/health/ to reflect Kinesis streams not present
I am trying to build a custom KinesisBinderHealthIndicator for a SpringBoot application.
My custom implementation should reveal a health DOWN status under /actuator/health endpoint when the actual ...
0
votes
0
answers
39
views
Spark Structured Streaming with Kinesis on Localstack: Error while Fetching Shard Iterator
I am trying to write integration tests with scalatest, testcontainers (0.39.5) and localstack. Spark version 2.4.5, spark library for Kinesis: com.qubole.spark spark-sql-kinesis 1.2.0_spark-2.4.
I am ...
0
votes
0
answers
29
views
right tech choice to work with Json data?
We are a data team working with our data producers to store and process infrastructure log data. Workers running at our client systems generate log data which are primarily in json format.
There is no ...
0
votes
0
answers
27
views
kinesis data stream performance testing with partition key
I am using the Kinesis Data Generator tool and I was wondering how to define the partition key in the test data so that the data is distributed to all the shard evenly.
https://awslabs.github.io/...
0
votes
1
answer
46
views
Kinesis Video Streaming is Not Working for Specific Cameras
I am using Kinesis Video Streaming for a solution and I am facing some problems with few RTSP sources. To all rtsp sources I am also validating on VLC and it is working well, except for a huge delay ...
0
votes
1
answer
36
views
Flink - Can't convert Table to DataStream
I've managed to use the Pyflink table API to connect to Kinesis and process a stream of data. I'm now trying to convert this table to a DataStream as I need more low level processing.
I've tried ...
0
votes
1
answer
26
views
UnicodeDecodeError in Python for CLI event, AWS Kinesis + Lambda
I have AWS Kinesis stream (with server-side encryption) set up, with Lambda function for event handling. I am trying to test this from CLI and get the error.
CLI for inserting event:
aws kinesis put-...
0
votes
1
answer
55
views
How to feed json from file to aws kinesis put-record
How to supply content of the file containing json to aws kinesis put-record?
I'd tried aws kinesis put-record --stream-name test_event --partition-key 1 --cli-input-json < file.jsonbut got ...
0
votes
0
answers
23
views
Apache Flink: RuntimeContextInitializationContextAdapters
I get an java.lang.ClassNotFoundException: org.apache.flink.api.common.serialization.RuntimeContextInitializationContextAdapters
when running my Apache Flink code locally.
It works fine when I use a ...
0
votes
0
answers
25
views
Can Kinesis back up the logs that existed before the Subscription Filter is set?
I want to setup a SubscriptionFilter to an existing CloudWatch Logs log group and backup the log to S3 with Kinesis.
Can Kinesis back up the logs that existed before the Subscription Filter is set?
Is ...
0
votes
1
answer
26
views
What are the units for Sum & SampleCount stats for ReadProvisionedThroughputExceeded metric?
What does it mean to have different values for Sum() vs SampleCount() stats over ReadProvisonedThroughputExceeded metric for kinesis stream? Shouldn't it be the same?
Period = 5 mins
0
votes
0
answers
17
views
AWS Kinesis 500 Error when using AT_TIMESTAMP
I am facing a bug when using Amazon Kinesis while building locally with localstack. I'm not sure if this is just a localstack problem or if it is a problem with my implementation/usage. If anyone ...
0
votes
1
answer
17
views
How to guarantee to process Kinesis event stream serially when using paralleization factor?
Kinesis stream has only 1 shard and when creating Lambda, concurrent batches per shard for Kinesis stream source has been set as 10. When there is a spike in stream data, it will increase the ...
0
votes
0
answers
51
views
Unable to enable Kinesis Data Stream as destination stream for DynamoDB in local
I have a project in which I have to capture the DynamoDB table change events using the Kinesis Data Streams.
Here are the sequence of operations that I am performing on my local:
Start the DDB ...