Questions tagged [avro]

Apache Avro is a data serialization framework primarily used in Apache Hadoop.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

HybridSource with AvroInputFormat in Flink

How do I use a Hybridsource with the AvroInputFormat? HybridSource requires a Source but AvroInputFormat extends FileInputFormat which is not accepted by FileSource. It looks like those are two ...
Clemens Valiente's user avatar
0 votes
2 answers
42 views

Unable to deserialise kafka topic in a stream using avro serialisation

I have a Quarkus project and a kafka producer which has SpecificAvroSerializer<MyObject> which writes successfully to topic and I can see the message ok Then I attach a stream to this topic ...
simonC's user avatar
  • 4,105
0 votes
0 answers
19 views

Kafka Mirror to Apache Kafka from AWS MKS Cluster

I'm confused about the architecture of mirroring a topic among an AWS MKS cluster (source) to an Apache Kafka 2.13 cluster (target). The source cluster also uses AWS Glue Schema registry, so I need to ...
Siraxis's user avatar
  • 15
0 votes
0 answers
20 views

How to override avro namespace and name to deserialize to a specific class in KafkaAvroDeserializer

I am trying to deserialize records in Kafka that are serialized using Avro. Unfortunately, the producer (who is not me, and has been producing for other consumers for a long time) has set a schema ...
campbellc.dev's user avatar
0 votes
1 answer
22 views

How to extract schema id from avro message in Spark Scala

I have spark Scala dataframe with column contains the value of avro message(Array[Byte]). I know that the 0 byte is the magic byte and the bytes in positions 1-4 included is the schema id. how can i ...
Tal's user avatar
  • 71
0 votes
0 answers
18 views

AWS EMR Master node is missing avro files

I am following the AWS workshop to create AWS EMR Cluster and play with HUDI - https://catalog.us-east-1.prod.workshops.aws/workshops/c86bd131-f6bf-4e8f-b798-58fd450d3c44/en-US I created only the ...
Judi's user avatar
  • 730
0 votes
0 answers
12 views

Upload Avro to BigQuery Provided Schema does not match Table x. Field big_numeric_col has changed type from BIGNUMERIC(40, 2) to NUMERIC

Attempting to load data to BigQuery using an Avro file. The column is defined in BigQuery as BIGNUMERIC(40,2).The Avro schema doesn't have a way to disambiguate between NUMERIC and BIGNUMERIC. I'm ...
Art Kaufmann's user avatar
-1 votes
0 answers
17 views

Apache Flink: Handling Compatibility Issues and Schema-Based Deserialization for Kafka Topic to AWS S3 Sink

I am currently working on a project where I need to use Apache Flink to consume data from a Kafka topic and sink it into an AWS S3 bucket. In the past, when I worked with Kafka Streams, I used Spring ...
engicode123's user avatar
0 votes
0 answers
33 views

Apache Avro C++ serialization framework (native type flexibility)

Apache Avro usage for C++ stub generation, native types to c++ (e.g. char[max], bool, etc). As an example I would like to produce given header stub file: struct CommonStruct { int32_t SeqNum; ...
Skuch's user avatar
  • 33
0 votes
0 answers
22 views

Created Stream Does not Populate Event from the Topic

I am trying to do a simple stream POC to show a stream can extract event from a topic. But the stream does not populate any data at all. My steps: Manually create a topic in Control Center console, ...
Brian N.'s user avatar
0 votes
1 answer
44 views

Is there a python library to convert schema from pyspark dataframe to avro schema?

I am building an application in which I am querying SQL Server table using pyspark. My plan is to push this to kafka, which will be consumed by Google Cloud Storage Sink Connector & saved in ...
downloaderfan's user avatar
0 votes
0 answers
16 views

How to append parquet file with old schema to new parquet file having new schema

Our current project create individual parquet file to save record on hdfs file system with avro schema. There is another process which run every four hours, it took such individual parquet files and ...
anna's user avatar
  • 265
0 votes
0 answers
30 views

Avro "long" type's maximum value

I have an type which is "long" in my avro schema file (avsc) like the below snippet { "name" : "TransactionDateTime", "type" : { "...
asb's user avatar
  • 781
0 votes
0 answers
17 views

Use Avro schema generated during the process in writing operation to Avro file in Apache Beam pipeline/Java

I use Apache Beam 2.49. I have to get data from Oracle database and save it in Avro file on Google Cloud Storage. I know how to get this data, schema and generate GenericRecord from Oracle but I have ...
pijars's user avatar
  • 1
0 votes
0 answers
19 views

Apache beam write PCollection<string,pojo> to GCS bucket

I am trying to put a PCollection<KV<String,POJO>> , I need to write this Pojo to GCS bucket in the avro format, where the string(key) for the map is the GCS folder path that I need to ...
Surya Srivastava's user avatar
0 votes
0 answers
22 views

How to use Avro to decode an InputStream with JsonDecoder?

I am trying to process a potentially very large input, not sourced from a File, through an InputStream and deserialize Java POJO (records) out of it, using ReflectDatumReader. While BinaryDecoder is ...
RonVe's user avatar
  • 39
0 votes
0 answers
8 views

Avro rpc only import records not rpc methods

I am working with legacy avdl definitions for rpc. I am creating a new one for purpose and importing records from another avdl definition. Is there a way to only import records not the rpc service ...
Manoj Ramakrishnan's user avatar
0 votes
1 answer
44 views

How to disable automatic registration of schema on the Azure Schema Registry in Kafka Connect scenario?

Does anyone know whether it is possible to setup the source connector in Kafka Connect scenarios so that it does not automatically register schemas on the Azure Schema Registry? I ask since I'm ...
Longclaw's user avatar
0 votes
1 answer
27 views

Avro schema automatic publishing in spring

How to configure spring/kafka to automatically publish avro schemas to schema registry without need to publish records to corresponding topic?
Tix's user avatar
  • 614
0 votes
1 answer
38 views

Can different order of values in enum between producer schema and kafka schema return 40403

I am using kafka and spring boot application. I had some issues with the producer not being able to send messages to kafka, was getting Caused by: io.confluent.kafka.schemaregistry.client.rest....
f.trajkovski's user avatar
0 votes
1 answer
42 views

spark scala kafka avro deserializer [duplicate]

I am consuming a streaming dataframe from kafka. The data in value column is in avro. I want to deserialize the data into struct type. Currently I have written a udf function which calls the method ...
Mohammad Shahwez's user avatar
0 votes
0 answers
16 views

How to convert external data with Decimal into Value in Avro

I'm working on iceberg-rs and iceberg support avro format metadata. When writting metadata, I use append_ser provided by writer in avro. It works fine because I only need to implement Serialize trait ...
NOTME ZE's user avatar
0 votes
0 answers
26 views

In Kotlin creating a parquet file with AvroParquetWriter only first record has value

I'm trying to write several records to a parqeut file using the following writer: private fun createAvroWriter(schema: Schema, outputFile: HadoopOutputFile): ParquetWriter<GenericRecord>? { ...
Pearce Jackson's user avatar
-1 votes
1 answer
64 views

How to solve "avro.AvroRowDeserializationSchema does not exist in the JVM" when trying to get data using pyflink [closed]

Pyflink getting data from kafka producer. When producer sends data in json-fomat, JsonRowDeserializationSchema works fine, but when I'm sending data in avro format, AvroRowDeserializationSchema falls ...
dusha32741's user avatar
0 votes
0 answers
17 views

Finding out the intended deserialization type of an Avro byte array before trying to deserialize

I want to use Avro for network packets in my TCP program. I would also like to use structs for all my messages, or at least, used pooled objects. But either approach would need me to know the intended ...
Saturn's user avatar
  • 17.9k
0 votes
0 answers
59 views

Pubsub to BigQuery Subscription : Schema revisions leads to null values in BigQuery

I am facing issues with pubsub schema evolution and when I add new fields then their values end up being null in BigQuery. My project structure is as follows. PubSub Schema { "namespace": &...
SRJ's user avatar
  • 2,122
0 votes
0 answers
23 views

Capturing deserialization exceptions in KafkaIO

I have a typical KafkaIO based source for reading Avro formatted key and value from a Kafka topic. PCollection<KafkaRecord<GenericRecord, GenericRecord>> records = pipeline.apply( ...
pravish's user avatar
  • 33
0 votes
1 answer
41 views

How could Kafka Connect help persisting Avro to a database and why is this a good design approach?

According to this stack Kafka Connect would help prevent performing a "2-step commit" when publishing to Kafka and persisting the avro message to a database, and that it follows a "read ...
Hercules Konsoulas's user avatar
0 votes
0 answers
19 views

Avro - sample input

I have an Avro schema as follows { "doc": "Sample schema to help you get started.", "fields": [ { "name": "specversion", "...
Disha Giriyan's user avatar
0 votes
0 answers
20 views

Confluent Kafka S3 source connector - not a valid AVRO format

I am trying to setup a Confluent kafka s3 source connector to read from S3 and add messages to Kafka topic. I am struggling with the formats - it asks for input and output format. What those mean? I ...
manpreet's user avatar
  • 636
0 votes
1 answer
37 views

Kafka Avro Int to date conversion using apache camel/groovy

There is Avro int date formate like "name":"BirthDate","type":["null","int"],"default":null,"logicalType":"date"} A ...
DummyGuy's user avatar
  • 425
0 votes
0 answers
11 views

Use Avro model from different package in different repository

I have not common problem I have repository X which contains avro model called Person. In my repository Y, I would like to create a new model with property of type Person from repository X. Is it even ...
Mati 's user avatar
0 votes
0 answers
66 views

Azure Function KafkaTrigger (.NET) consuming messages with Avro array schema as root

I have an Azure Function KafkaTrigger (using azure functions kafka extension: https://github.com/Azure/azure-functions-kafka-extension) which is set to consume messages from Kafka with Avro schema ...
michn's user avatar
  • 246
0 votes
0 answers
15 views

Avro class is not generated

I have a simple avro schema in the path src/main/resources/avro/Task.avsc: { "namespace": "com.minompi.java_kafka_example.kafka.avro.model", "type": "record"...
Alessandro Minoccheri's user avatar
0 votes
0 answers
17 views

Kotlin Quarkus Avro NoSuchFileException

I have created an application that reads data from a Kafka topic using Quarkus and Kotlin. The data on the topic are in Avro format, so I have to use io.quarkus:quarkus-avro dependency. Everything ...
Stefano Castoldi's user avatar
0 votes
1 answer
18 views

Parsing Nested array within a nested array in MemSQL

I am trying to extract data from Avro schema and insert the data to db in MemSQL. I have the below Avro Schema { "doc": "Sample schema to help you get started.", "...
Disha Giriyan's user avatar
0 votes
0 answers
33 views

Kafka connect transform ValueToKey not recognising value type

In my Cassandra connector configuration, I'm trying to create a composite primary key for Cassandra but I'm facing some issues. The producer is serialising the message using Avro. Connector ...
Rafael Simões's user avatar
0 votes
0 answers
43 views

Debezium source connector database migrations breaking AVRO schema compatibility

Sorry for the stupid title. We're using Debezium source connector to stream changes from PostgreSQL to kafka. Along the way the connector is registering schemas to confluent schema-registry. Streaming ...
welcomeboredom's user avatar
0 votes
0 answers
47 views

Evolve AVRO with optional field I have "Error retrieving Avro schema"

I want to evolve my AVRO schema with an optional field. I add this on my microservice file (.avsc) { "name": "optionalFieldExample", "type": [ "null", ...
davidleongz's user avatar
0 votes
1 answer
36 views

Does message in Kafka serialized with AVRO contain also version of schema?

I've found: Apache Kafka with Avro and Schema Repo - where in the message does the schema Id go? It's clear to me the schema ID is part of every message produced to Kafka. I can't find it being said ...
welcomeboredom's user avatar
0 votes
2 answers
44 views

How to create Avro with union types in Python

I try to convert a JSON file to Avro in Python, the caveat is that the schema has a union type. More specifically, in the minimal non-working example the JSON file is just {"field1": {"...
Piotr Śniady's user avatar
0 votes
1 answer
41 views

Failed to deserialize data for topic "testtopic" to Avro for sink connector

Here is the sink connector properties "name": "sink-testtopic-crdb", "config": { "connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector", &...
Sivapandian Jayakumar's user avatar
1 vote
0 answers
28 views

Missing type conversion trying to write Avro messages to Minio from Kafka Connect using camel-minio-sink-kafka-connector

I want to use the camel-minio-sink-kafka-connector from the Apache Camel project to write messages from a Kafka topic to a Minio storage bucket. The problem is that the messages are in the AVRO format,...
Andreas Baus's user avatar
0 votes
0 answers
34 views

Apache Avro vs. DFDL for data conversion

I am planning a service which will be used to convert data between many different structure formats. For example, some data may be simple and written in csv format, while other can be much more ...
Frank's user avatar
  • 621
0 votes
0 answers
14 views

PySpark Structured Streaming compatibility issues with Kafka Schema Registry [duplicate]

I am trying to read a structured stream from kafka from pyspark. Everything works wonders until there is no schema registry involved. I recently wanted to integrated a schema registry on my cluster ...
ETisREAL's user avatar
0 votes
1 answer
28 views

How can a kafka producer persist the published avro messages to a database?

I want to be able to persist the published messages in avro format to the database after I have I published them. How is it possible from a kafka publisher to get access to the published avro message ...
Marinos An's user avatar
  • 9,565
0 votes
0 answers
49 views

Expected start-union. Got VALUE_STRING while converting JsonObject to Avro Generic Record

I'm currently trying to convert a JsonObject into an Avro Generic Record where I stumbled upon this issue. My schema toString() looks like: { "type":"record", "name"...
Kulasangar's user avatar
  • 9,086
1 vote
1 answer
84 views

How to convert a JsonObject into an Avro Generic Record?

I'm still a newbie in this area, and currently I'm trying to deserialise data from a parquet file using the ParDo and DoFn in Apache Beam, make some modifications to the record and then update the ...
Kulasangar's user avatar
  • 9,086
0 votes
0 answers
17 views

Transforming an array to a string containing that array using jolt

This is my source input json { "payload": { "keys": { "sce0_client_num": 101, "sce0_application_num": -5493929203645992000, "...
Ankit Punamiya's user avatar
0 votes
1 answer
26 views

Kafka Schema Registry - when we use this do we need to keep all the historical avro files in the codebase?

We have an application - which has many components mainly microservices and we have kafka and schema registry getting installed as part of application package. In the code package of each application -...
Lucia's user avatar
  • 791

1
2 3 4 5
74