Questions tagged [avro]
Apache Avro is a data serialization framework primarily used in Apache Hadoop.
3,655
questions
0
votes
0
answers
7
views
HybridSource with AvroInputFormat in Flink
How do I use a Hybridsource with the AvroInputFormat?
HybridSource requires a Source but AvroInputFormat extends FileInputFormat which is not accepted by FileSource. It looks like those are two ...
0
votes
2
answers
42
views
Unable to deserialise kafka topic in a stream using avro serialisation
I have a Quarkus project and a kafka producer which has
SpecificAvroSerializer<MyObject>
which writes successfully to topic and I can see the message ok
Then I attach a stream to this topic ...
0
votes
0
answers
19
views
Kafka Mirror to Apache Kafka from AWS MKS Cluster
I'm confused about the architecture of mirroring a topic among an AWS MKS cluster (source) to an Apache Kafka 2.13 cluster (target).
The source cluster also uses AWS Glue Schema registry, so I need to ...
0
votes
0
answers
20
views
How to override avro namespace and name to deserialize to a specific class in KafkaAvroDeserializer
I am trying to deserialize records in Kafka that are serialized using Avro. Unfortunately, the producer (who is not me, and has been producing for other consumers for a long time) has set a schema ...
0
votes
1
answer
22
views
How to extract schema id from avro message in Spark Scala
I have spark Scala dataframe with column contains the value of avro message(Array[Byte]).
I know that the 0 byte is the magic byte and the bytes in positions 1-4 included is the schema id.
how can i ...
0
votes
0
answers
18
views
AWS EMR Master node is missing avro files
I am following the AWS workshop to create AWS EMR Cluster and play with HUDI - https://catalog.us-east-1.prod.workshops.aws/workshops/c86bd131-f6bf-4e8f-b798-58fd450d3c44/en-US
I created only the ...
0
votes
0
answers
12
views
Upload Avro to BigQuery Provided Schema does not match Table x. Field big_numeric_col has changed type from BIGNUMERIC(40, 2) to NUMERIC
Attempting to load data to BigQuery using an Avro file. The column is defined in BigQuery as BIGNUMERIC(40,2).The Avro schema doesn't have a way to disambiguate between NUMERIC and BIGNUMERIC. I'm ...
-1
votes
0
answers
17
views
Apache Flink: Handling Compatibility Issues and Schema-Based Deserialization for Kafka Topic to AWS S3 Sink
I am currently working on a project where I need to use Apache Flink to consume data from a Kafka topic and sink it into an AWS S3 bucket. In the past, when I worked with Kafka Streams, I used Spring ...
0
votes
0
answers
33
views
Apache Avro C++ serialization framework (native type flexibility)
Apache Avro usage for C++ stub generation, native types to c++ (e.g. char[max], bool, etc).
As an example I would like to produce given header stub file:
struct CommonStruct
{
int32_t SeqNum;
...
0
votes
0
answers
22
views
Created Stream Does not Populate Event from the Topic
I am trying to do a simple stream POC to show a stream can extract event from a topic. But the stream does not populate any data at all.
My steps:
Manually create a topic in Control Center console, ...
0
votes
1
answer
44
views
Is there a python library to convert schema from pyspark dataframe to avro schema?
I am building an application in which I am querying SQL Server table using pyspark. My plan is to push this to kafka, which will be consumed by Google Cloud Storage Sink Connector & saved in ...
0
votes
0
answers
16
views
How to append parquet file with old schema to new parquet file having new schema
Our current project create individual parquet file to save record on hdfs file system with avro schema.
There is another process which run every four hours, it took such individual parquet files and ...
0
votes
0
answers
30
views
Avro "long" type's maximum value
I have an type which is "long" in my avro schema file (avsc) like the below snippet
{
"name" : "TransactionDateTime",
"type" : {
"...
0
votes
0
answers
17
views
Use Avro schema generated during the process in writing operation to Avro file in Apache Beam pipeline/Java
I use Apache Beam 2.49. I have to get data from Oracle database and save it in Avro file on Google Cloud Storage. I know how to get this data, schema and generate GenericRecord from Oracle but I have ...
0
votes
0
answers
19
views
Apache beam write PCollection<string,pojo> to GCS bucket
I am trying to put a PCollection<KV<String,POJO>> , I need to write this Pojo to GCS bucket in the avro format, where the string(key) for the map is the GCS folder path that I need to ...
0
votes
0
answers
22
views
How to use Avro to decode an InputStream with JsonDecoder?
I am trying to process a potentially very large input, not sourced from a File, through an InputStream and deserialize Java POJO (records) out of it, using ReflectDatumReader.
While BinaryDecoder is ...
0
votes
0
answers
8
views
Avro rpc only import records not rpc methods
I am working with legacy avdl definitions for rpc. I am creating a new one for purpose and importing records from another avdl definition. Is there a way to only import records not the rpc service ...
0
votes
1
answer
44
views
How to disable automatic registration of schema on the Azure Schema Registry in Kafka Connect scenario?
Does anyone know whether it is possible to setup the source connector in Kafka Connect scenarios so that it does not automatically register schemas on the Azure Schema Registry?
I ask since I'm ...
0
votes
1
answer
27
views
Avro schema automatic publishing in spring
How to configure spring/kafka to automatically publish avro schemas to schema registry without need to publish records to corresponding topic?
0
votes
1
answer
38
views
Can different order of values in enum between producer schema and kafka schema return 40403
I am using kafka and spring boot application. I had some issues with the producer not being able to send messages to kafka, was getting
Caused by: io.confluent.kafka.schemaregistry.client.rest....
0
votes
1
answer
42
views
spark scala kafka avro deserializer [duplicate]
I am consuming a streaming dataframe from kafka. The data in value column is in avro. I want to deserialize the data into struct type.
Currently I have written a udf function which calls the method ...
0
votes
0
answers
16
views
How to convert external data with Decimal into Value in Avro
I'm working on iceberg-rs and iceberg support avro format metadata. When writting metadata, I use append_ser provided by writer in avro. It works fine because I only need to implement Serialize trait ...
0
votes
0
answers
26
views
In Kotlin creating a parquet file with AvroParquetWriter only first record has value
I'm trying to write several records to a parqeut file using the following writer:
private fun createAvroWriter(schema: Schema, outputFile: HadoopOutputFile): ParquetWriter<GenericRecord>? {
...
-1
votes
1
answer
64
views
How to solve "avro.AvroRowDeserializationSchema does not exist in the JVM" when trying to get data using pyflink [closed]
Pyflink getting data from kafka producer. When producer sends data in json-fomat, JsonRowDeserializationSchema works fine, but when I'm sending data in avro format, AvroRowDeserializationSchema falls ...
0
votes
0
answers
17
views
Finding out the intended deserialization type of an Avro byte array before trying to deserialize
I want to use Avro for network packets in my TCP program.
I would also like to use structs for all my messages, or at least, used pooled objects.
But either approach would need me to know the intended ...
0
votes
0
answers
59
views
Pubsub to BigQuery Subscription : Schema revisions leads to null values in BigQuery
I am facing issues with pubsub schema evolution and when I add new fields then their values end up being null in BigQuery. My project structure is as follows.
PubSub Schema
{
"namespace": &...
0
votes
0
answers
23
views
Capturing deserialization exceptions in KafkaIO
I have a typical KafkaIO based source for reading Avro formatted key and value from a Kafka topic.
PCollection<KafkaRecord<GenericRecord, GenericRecord>> records =
pipeline.apply(
...
0
votes
1
answer
41
views
How could Kafka Connect help persisting Avro to a database and why is this a good design approach?
According to this stack Kafka Connect would help prevent performing a "2-step commit" when publishing to Kafka and persisting the avro message to a database, and that it follows a "read ...
0
votes
0
answers
19
views
Avro - sample input
I have an Avro schema as follows
{
"doc": "Sample schema to help you get started.",
"fields": [
{
"name": "specversion",
"...
0
votes
0
answers
20
views
Confluent Kafka S3 source connector - not a valid AVRO format
I am trying to setup a Confluent kafka s3 source connector to read from S3 and add messages to Kafka topic. I am struggling with the formats - it asks for input and output format. What those mean? I ...
0
votes
1
answer
37
views
Kafka Avro Int to date conversion using apache camel/groovy
There is Avro int date formate like
"name":"BirthDate","type":["null","int"],"default":null,"logicalType":"date"}
A ...
0
votes
0
answers
11
views
Use Avro model from different package in different repository
I have not common problem
I have repository X which contains avro model called Person. In my repository Y, I would like to create a new model with property of type Person from repository X. Is it even ...
0
votes
0
answers
66
views
Azure Function KafkaTrigger (.NET) consuming messages with Avro array schema as root
I have an Azure Function KafkaTrigger (using azure functions kafka extension: https://github.com/Azure/azure-functions-kafka-extension) which is set to consume messages from Kafka with Avro schema ...
0
votes
0
answers
15
views
Avro class is not generated
I have a simple avro schema in the path src/main/resources/avro/Task.avsc:
{
"namespace": "com.minompi.java_kafka_example.kafka.avro.model",
"type": "record"...
0
votes
0
answers
17
views
Kotlin Quarkus Avro NoSuchFileException
I have created an application that reads data from a Kafka topic using Quarkus and Kotlin. The data on the topic are in Avro format, so I have to use io.quarkus:quarkus-avro dependency.
Everything ...
0
votes
1
answer
18
views
Parsing Nested array within a nested array in MemSQL
I am trying to extract data from Avro schema and insert the data to db in MemSQL. I have the below Avro Schema
{
"doc": "Sample schema to help you get started.",
"...
0
votes
0
answers
33
views
Kafka connect transform ValueToKey not recognising value type
In my Cassandra connector configuration, I'm trying to create a composite primary key for Cassandra but I'm facing some issues.
The producer is serialising the message using Avro.
Connector ...
0
votes
0
answers
43
views
Debezium source connector database migrations breaking AVRO schema compatibility
Sorry for the stupid title.
We're using Debezium source connector to stream changes from PostgreSQL to kafka. Along the way the connector is registering schemas to confluent schema-registry.
Streaming ...
0
votes
0
answers
47
views
Evolve AVRO with optional field I have "Error retrieving Avro schema"
I want to evolve my AVRO schema with an optional field.
I add this on my microservice file (.avsc)
{
"name": "optionalFieldExample",
"type": [
"null",
...
0
votes
1
answer
36
views
Does message in Kafka serialized with AVRO contain also version of schema?
I've found: Apache Kafka with Avro and Schema Repo - where in the message does the schema Id go?
It's clear to me the schema ID is part of every message produced to Kafka. I can't find it being said ...
0
votes
2
answers
44
views
How to create Avro with union types in Python
I try to convert a JSON file to Avro in Python, the caveat is that the schema has a union type. More specifically, in the minimal non-working example the JSON file is just {"field1": {"...
0
votes
1
answer
41
views
Failed to deserialize data for topic "testtopic" to Avro for sink connector
Here is the sink connector properties
"name": "sink-testtopic-crdb",
"config": {
"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector",
&...
1
vote
0
answers
28
views
Missing type conversion trying to write Avro messages to Minio from Kafka Connect using camel-minio-sink-kafka-connector
I want to use the camel-minio-sink-kafka-connector from the Apache Camel project to write messages from a Kafka topic to a Minio storage bucket. The problem is that the messages are in the AVRO format,...
0
votes
0
answers
34
views
Apache Avro vs. DFDL for data conversion
I am planning a service which will be used to convert data between many different structure formats. For example, some data may be simple and written in csv format, while other can be much more ...
0
votes
0
answers
14
views
PySpark Structured Streaming compatibility issues with Kafka Schema Registry [duplicate]
I am trying to read a structured stream from kafka from pyspark. Everything works wonders until there is no schema registry involved. I recently wanted to integrated a schema registry on my cluster ...
0
votes
1
answer
28
views
How can a kafka producer persist the published avro messages to a database?
I want to be able to persist the published messages in avro format to the database after I have I published them.
How is it possible from a kafka publisher to get access to the published avro message ...
0
votes
0
answers
49
views
Expected start-union. Got VALUE_STRING while converting JsonObject to Avro Generic Record
I'm currently trying to convert a JsonObject into an Avro Generic Record where I stumbled upon this issue.
My schema toString() looks like:
{
"type":"record",
"name"...
1
vote
1
answer
84
views
How to convert a JsonObject into an Avro Generic Record?
I'm still a newbie in this area, and currently I'm trying to deserialise data from a parquet file using the ParDo and DoFn in Apache Beam, make some modifications to the record and then update the ...
0
votes
0
answers
17
views
Transforming an array to a string containing that array using jolt
This is my source input json
{
"payload": {
"keys": {
"sce0_client_num": 101,
"sce0_application_num": -5493929203645992000,
"...
0
votes
1
answer
26
views
Kafka Schema Registry - when we use this do we need to keep all the historical avro files in the codebase?
We have an application - which has many components mainly microservices and
we have kafka and schema registry getting installed as part of application package.
In the code package of each application -...