Questions tagged [amazon-kinesis]

Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale.

Filter by
Sorted by
Tagged with
0 votes
1 answer
16 views

Localstack (AWS) credential failure on java lambda

The purpose of this lambda is insert a bucket content on a kinesis data stream, here is the code: package productor; import software.amazon.awssdk.services.s3.*; import software.amazon.awssdk....
Angel Sanchez's user avatar
0 votes
1 answer
28 views

Unable to trigger Lambda from Kinesis Consumer

I have a lambda which consumes near-real time updates from a DynamoDB table through a Kinesis Data Stream. I also created a stream consumer from which the lambda consumes data. When I try to deploy ...
Sudha N's user avatar
  • 25
0 votes
0 answers
10 views

Is there a way to tell how far behind a Lambda event mapping for Kinesis is?

I have a Lambda subscribing to a Kinesis stream through an event source mapping. My understanding is that the event source mapping stores the last processed record id (per shard) as a part of its ...
Quassnoi's user avatar
  • 414k
0 votes
0 answers
7 views

KCL 2.x is polling Kinesis stream at slower rate than configured

I have a KCL application where I am not setting any value for idleTimeBetweenReadsInMillis of PollingConfig class. So, it should be using default value set i.e. 1000ms (1sec). This should poll 60 ...
Niranjan Rathod's user avatar
0 votes
2 answers
42 views

Java 17 runtime kinesis lambda code doesnt work localstack (easy code)

Its very simple. This code works as jar and lambda (maven project): package test2; import java.util.Map; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda....
Angel Sanchez's user avatar
0 votes
1 answer
24 views

Amazon Kinesis, Lambda and DLQ - how to properly get payload contents from the DLQ in a large traffic scenario

Kinesis DLQ does not contain the payload, only the metadata. You're supposed to read the payload like in this SO issue by calling GetRecords. My setup is as follows: Apps --> Kinesis Data Stream --...
blahblah's user avatar
  • 1,171
0 votes
0 answers
24 views

Integrating Spring with AWS Binder (Kinesis)

I'm trying to implement an integration solution with Kinesis (AWS), using "Stream Binder Kinesis". I need to send a json(String) to a queue (producer). I'm based on the example https://www....
Anderson's user avatar
0 votes
1 answer
18 views

AWS Kinesis long startup waiting

I'm using AWS Kinesis with spring cloud to read data from a data event hub. The deploy end with the following log: 2023-09-26 14:07:30.476 INFO [,,] 9331 --- [esis-consumer-1] a.i.k....
AM13's user avatar
  • 671
0 votes
1 answer
38 views

Creating a Kinesis Stream on existing DynamoDB table

I already have an existing DynamoDB table when regularly gets updated with data from an upstream source. How can I create a new Kinesis stream to consume updates from the table in AWS CDK? From other ...
Sudha N's user avatar
  • 25
1 vote
1 answer
46 views

Spring Cloud Stream Kinesis Binder 4.0 doesn't support Multi Stream with KCL 2 Enabled

KCL 2 allows multi stream support with the same consumer group in the same application. With cloud stream kinesis binder 4.0, kpl-kcl-enabled: true A single stream in the destination works. bindings: ...
siddharth Jain's user avatar
0 votes
1 answer
23 views

Adding Kinesis Streams to a DynamoDB Global Table

I have a DynamoDB global table with replicas in two regions, and I want to enable Kinesis Data Streaming. For any write to either table I want only one Kinesis event. What is the suggested approach to ...
Stephen Elmendorf's user avatar
0 votes
0 answers
15 views

Push events data to AWS Kinesis Firehose via web client

I have a very simple use case of pushing events (click events) from a web browser and storing the data to S3 using AWS kinesis. I'm a bit new to streaming data ingestion so I don't understand the ...
muazfaiz's user avatar
  • 4,631
0 votes
1 answer
21 views

Do not see data in AWS Kinesis

I created a stream in AWS Kinesis, straightforward: I put records into the stream (I took the script from AWS documentation): import datetime import json import random import boto3 STREAM_NAME = &...
Edward Igushev's user avatar
0 votes
0 answers
12 views

Lambda Max Concurrency in Monitoring does not match the configured concurrency in kinesis trigger

I have a Lambda function with Kinesis as the triggers. The kinesis stream only have one shard, and the trigger is configured with Concurrent batches per shard: 3. But, when I checked the Lambda ...
Silly Sally's user avatar
  • 1,037
0 votes
0 answers
22 views

How to write data to Kinesis Data Stream with Apache Beam using Python

I am trying to build Beam pipeline in Python that reads from and writes to AWS Kinesis Data Streams. I am using the FlinkRunner, with Flink 1.15.4 running locally on my machine. Python 3.8.17 Apache-...
Charl's user avatar
  • 111
0 votes
0 answers
19 views

AWS Glue crawler for Athena query is not showing all the records that i have in s3 bucket

I have a c# application that logs audit records to Kinesis Firehose so those get PUT into a S3 bucket. That is working perfectly because I can see the files in the S3 bucket. I'm using Dynamic ...
VAAA's user avatar
  • 14.6k
0 votes
0 answers
18 views

Unable to install Amazon Kinesis Video Streams Producer SDK Java

I've followed this tutorial but I'm not able to build this sdk, I'm getting the following error: Exception in thread "main" 01:34:40.783 [main] ERROR com.amazonaws.kinesisvideo.java.client....
Manish Taneja's user avatar
0 votes
1 answer
33 views

Kinesis KCL not pulling records from all shards

I've configured the Kinesis stream with 4 shards and implemented the KCL library version-1 as a consumer. The consumer pulling records from one shard only (shardId-000000000000). I've also noticed if ...
ankit.vishen's user avatar
  • 1,110
0 votes
0 answers
41 views

Fluent Bit configuration for separating and routing application logs to distinct locations on AWS EKS Fargate

I'm currently working with AWS EKS Fargate and facing challenges with effectively sending application logs to an AWS S3 bucket. I'm utilizing Fluent Bit as a log router, which routes logs to a Kinesis ...
Vinasirajan Vilakshan's user avatar
0 votes
0 answers
28 views

Data output costs in Kinesis Data Stream when connecting many eventbridge pipes

If we connect two EventBridge Pipes to a kinesis Data Stream (Customer Orders) then the cost of data output from the kinesis doubles? and so on if we add more Eventbridge Pipes to Kinesis? Link to ...
Carlos Freire's user avatar
0 votes
0 answers
49 views

Can Amazon Kinesis Firehose deliver to HTTP endpoint configured with a self-signed SSL certificate?

In an Amazon Connect instance, I set up Data Streaming such that Contact Trace Records and Agent Events are published to the same Kinesis Stream. Then I created a delivery mechanism of said events by ...
Kode Charlie's user avatar
  • 1,295
0 votes
1 answer
81 views

Why doesn't Kinesis Streams Data Viewer show any Agent Events after call through Amazon Connect contact Flow?

QUESTION: Why don't I see Agent Events in the Data Viewer for the Kinesis Stream I attached to an instance of Amazon Connect? Here is what I did: Created a Kinesis Stream. Call it amzn-conn-agent-...
Kode Charlie's user avatar
  • 1,295
0 votes
0 answers
11 views

AWS KCL MultiLanguageDaemon - override Dynamo table name used for checkpointing

According to the following article it's possible to override the the DynamoDB table used for checkpointing when using the KCL directly. Here's an example of how to set up a table name on KCL version ...
Charles's user avatar
  • 21.7k
0 votes
2 answers
54 views

Why is name of stream for Kinesis Video Stream (triggered from Amazon Connect) not recognized?

In Amazon Connect, I create a Flow with Start Streaming Data block. I have verified that calls trigger live audio streams through Kinesis Video Streams. See the ARNs below (captured by calling AWS ...
Kode Charlie's user avatar
  • 1,295
0 votes
1 answer
26 views

Amazon.IonDotnet.Internals.Text.InvalidTokenException error while reading stream data from AWS Kinesis by QLDB stream

I am not sure if I can find a solution for this, but if is there anybody doing development with AWS QLDB please ping me. Anyway, my problem is: I have a running solution doing some operation on AWS ...
Arda's user avatar
  • 407
1 vote
1 answer
51 views

Log more data from Kafka Source Connector

I installed the Kinesis Connector plugin in AWS, to connect my Kinesis Stream with my MSK cluster. It is working, but the logging that I see in Cloudwatch is not very helpful: [Worker-...
Homer's user avatar
  • 85
3 votes
1 answer
159 views

Cloudwatch Subscription Filter does not ingest to Kinesis Data Stream

I am trying to forward Cloudwatch logs to Kinesis Data Stream using the Subscription filter. Even with broad permissions, for some reason, I am not able to see any records in Kinesis. I could see logs ...
user203687's user avatar
  • 6,905
0 votes
0 answers
23 views

Kinesis Confluent Connector not reaching MSK Cluster

I am trying to add a Kinesis connector to connect my Kinesis Stream with my AWS MSK Cluster, so that I can send messages from the Kinesis to a certain topic in the Kafka cluster. This is the connector ...
Homer's user avatar
  • 85
0 votes
0 answers
35 views

How to stream RTSP video stream using AWS KVS to browser

I have a source which generates RTSP stream. I am sending the RTP packets to Amazon Kinesis Video Stream. My client is a web browser were I want to play the video stream. How can I relay the stream ...
User7723337's user avatar
  • 11.9k
0 votes
1 answer
41 views

Are Kinesis sequence numbers unique across the stream or just per shard and per partition key?

The documentation is conflicting in this regard. The PutRecords documentation says: The SequenceNumber parameter is an identifier assigned to the put record, unique to all records in the stream. ...
piousinquisitor's user avatar
1 vote
1 answer
185 views

No data on AWS Kinesis after putRecords

Disclaimer: This is the first time I am using AWS Kinesis, so I might be expecting something that is not correct. I have a very simple AWS Lambda function that inserts data into Kinesis. The API ...
Rodrigo's user avatar
  • 257
0 votes
1 answer
63 views

Best way to process and store real time MQTT data in AWS Cloud

I need to receive the real-time MQTT data, get them stored in AWS Cloud, and provide the insights using these data in a separate web-based application. In this requirement, would that be better to ...
user22299710's user avatar
0 votes
1 answer
26 views

How to add multiple consumer of same data stream in Flink

I am struggling with implementing solution where the problem is as below. There is one Kinesis stream named sushistream, event comes from the stream would contains two things, i.e. broadcastId and ...
Dipesh Darji's user avatar
1 vote
1 answer
10 views

KPL amazon Library only works when debugging

I'm having an awkward behaviour when trying to send data to amazon kinesis data streams. what happens is that i'm just sending one record to test but it doesn't work and when i normally debug it or ...
Erick Jhorman Romero's user avatar
0 votes
0 answers
23 views

High CPU Consumption using Cassandra Sink in Kinesis Analytics

I'm trying to set up a Kinesis Analytics application to write data from a Kinesis Stream to Keyspaces. To do this, I'm using flink's cassandra connector. The application receives messages from a ...
maxwellray's user avatar
0 votes
1 answer
28 views

Error while using spring-cloud-bus with aws kinesis stream binder

I want to use the actuator/bus-refresh for signaling my config change across my containers and for that I use: spring.cloud.bus.enabled=true management.endpoints.web.exposure.include=bus-refresh and ...
JITHIN_PATHROSE's user avatar
0 votes
0 answers
29 views

kinesis-client-library is not reading from all the shards

I've amazon kinesis data streams with 16 shards with shardId starting from shardId-000000000012 to shardId-000000000027. I also have kinesis client library with same region, app-name, stream-name and ...
user51's user avatar
  • 8,927
2 votes
2 answers
194 views

Spark streaming forEachBatch giving inconsistent/unordered result while writing to database

Problem: I am receiving multiple table/schema data in a single stream. Now after segregating the data I am opening a parallel write stream for each table. The function I used in forEachBatch is: def ...
Shubham Jain's user avatar
  • 5,327
0 votes
0 answers
45 views

How do I provide a session token when using FlinkKinesisConsumer in Java?

I'm trying to use user A to assume role R1, and using R1 assume R2 in a different account. I have gone as far as creating an StsClient to get the accessKeyId, secretKeyId, and sessionToken that I need,...
squirrel_coder's user avatar
1 vote
0 answers
14 views

kinesis connected to multiple shards - wont fully disconnect again

i have an issue where, if i connect to more than one shard, and call client.destroy(), the script keeps running, where i expect it to stop running, as no more connections should stay open. im using @...
keja's user avatar
  • 1,333
0 votes
1 answer
55 views

Add tags to s3 objects dynamically to apply s3 life cycle rules

I have below s3 lifecycle rule created for existing s3 bucket. It should not be changed. I am sending the logs from kinesis firehose to s3 bucket as parquet formatted objects. Is there any way or ...
Divya's user avatar
  • 23
0 votes
0 answers
16 views

How to write to Kinesis stream from Spark3?

How do I write to a Kinesis stream from Spark3? We were previously doing this in Spark 2.4.0 while using the DataSourceV2 class and related classes/APIs as a way to connect - wondering what the ...
Sahil Dogra's user avatar
0 votes
0 answers
12 views

What impacts aggregation in KPL for a Spark job?

I run Spark jobs to write to Kinesis via KPL, and I recently upgraded my KPL from 1.8.10 to 1.14.x. Previously, my job resulted in 1 Kinesis record with 20 user records. After the upgrade, there are ...
stackoverflowflowflwofjlw's user avatar
0 votes
0 answers
26 views

Equivalent of aws.sqs.approximate_number_of_messages_visible in Kinesis

I know that the aws.sqs.approximate_number_of_messages_visible is the 'The number of messages available for retrieval from the queue'. What would the equivalent be if I am to use Kinesis and wanting ...
Nik's user avatar
  • 1
0 votes
0 answers
21 views

KPL upgrade and batching differences

I upgraded KPL from version 1.8.10 to 1.14.x and noticed that the data being produced by my Spark job results in smaller Kinesis records (less user records per batch). There is no data quality issue (...
stackoverflowflowflwofjlw's user avatar
0 votes
0 answers
117 views

AWS DMS: Replication Instance Fails to Connect to Kinesis Target Endpoint

I am working on setting up an AWS CloudFormation stack to deploy a database using Aurora Postgres Serverless V2, some network resources, a Jumpbox, and some AWS DMS resources to use real-time CDC as ...
E. Faslo's user avatar
  • 325
0 votes
0 answers
73 views

Apache Flink on python UDFs are not working. Despite following AWS repo on pyflink UDFs

I see AWS has a Kinesis-DataAnalytics stream with Apache Flink. And on the pyflink library for python there is a way to create UDF. In fact, AWS seems to have a repo with examples for pyflink UDF. I ...
Eugenio.Gastelum96's user avatar
2 votes
1 answer
66 views

RDS MySQL to Kinesis data stream pipeline using AWS Lambda

I am trying to extract data from RDS MySQL instance and load into kinesis data stream using put_record boto3 API. The connection using pymysql is working and I am able to print the table but, I cannot ...
talha babar's user avatar
0 votes
0 answers
166 views

AWS Cloudwatch -> Kinesis -> OpenSearch

I setup a Cloudwatch subscription filter to a Kinesis data firehose, with a destination of an OpenSearch instance and I'm using the "Process CloudWatch logs sent to Kinesis Firehose" ...
Enrique Avina's user avatar
0 votes
1 answer
72 views

AWS remote video not loading (viewer)

amazon-kinesis-video-streams-webrtc-sdk-js video stream works on local (ie. viewer and master on same machine or browser with different tabs ) but when viewer and master is on different machine (ie. ...
razer's user avatar
  • 118

1
2 3 4 5
37