Skip to main content
Filter by
Sorted by
Tagged with
0 votes
1 answer
33 views

PySpark connection to Cassandra returns "Py4JJavaError: An error occurred while calling o54.start"

I'm try to make connection from Pyspark to Cassandra in virtual environment and the services is installed by docker. I've been using the --packages method to solve the dependencies but it seems doesn'...
ramadhani nugraha's user avatar
1 vote
0 answers
24 views

Pyspark connector slow during joins

I'm building an application that allows me to use pyspark to combine an oracle and a cassandra table. The cassandra table's count is in a scale of 100s of millions with the oracle one at a few 1000. ...
GrImPeAper's user avatar
0 votes
0 answers
57 views

Spark is not inserting data into Cassandra when using ```writeStream```

I'm trying to create a pipeline in streaming that is calling an API using Airflow, then processing it with Kafka and inserting the data into Cassandra using Spark. I'm struggling when inserting data ...
francollado99's user avatar
1 vote
0 answers
26 views

Ignoring codec because it collides with previously generated codec

i am trying to register custom code(for map) like below val session: CqlSession = CassandraConnector.apply(spark.sparkContext).openSession() val codecRegistry: MutableCodecRegistry = session....
Shivam Sajwan's user avatar
2 votes
0 answers
49 views

register a custom codec in cassandra connector

I am using spark-cassandra-connector_2.11 and version 2.5.2 in my scala application and want to register a custom map codec, but facing issues. Is there any way to register it. I did the same thing on ...
Shivam Sajwan's user avatar
0 votes
1 answer
52 views

How To Use Spark Submit Operator With Cassandra Remote Server In Apache Airflow

I'm working airflow into the Docker container on WindowsPC. I have some problems with apache airflow spark submit operator. I want to write data to a remote Cassandra server. When I was using df.write....
zehra özdemir's user avatar
0 votes
1 answer
117 views

PySpark app returns "NoClassDefFoundError: com/datastax/spark/connector/util/Logging"

I had this error: py4j.protocol.Py4JJavaError: An error occurred while calling o59.start. : java.lang.NoClassDefFoundError: com/datastax/spark/connector/util/Logging at java.base/java.lang....
francollado99's user avatar
0 votes
1 answer
45 views

Pyspark cassandra connector generates tombstones during writing

I understand that when inserting data, tombstones might be created because of existing null values in the columns of the dataframe. To mitigate this issue and minimize tombstones, insertion queries ...
danmo41's user avatar
0 votes
3 answers
268 views

Simple Python app failed to load, getting "ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra"

Context : Sorry to bother you with I am struggling to install a Cassandra-Spark connector. My goal is to install it to use Spark-SQL since Cassandra has strong limitations to do requests. I have : ...
laurentg13's user avatar
0 votes
0 answers
36 views

Cassandra db restore database issue

i'm trying to restore cassandra database keyspace but in restore time face some issues ERROR ---------- INFO [CompactionExecutor:1] 2023-12-19 06:32:07,415 CompactionTask.java:253 - Compacted (...
Ajay's user avatar
  • 1
0 votes
0 answers
94 views

Spark-Cassandra read cassandra table returning NoSuchElementException

I have been trying to copy a table from one keyspace to another using Apache spark. I have been getting the java.util.NoSuchElementException on null partition when iterating via the .foreachPartition()...
Candy Ho's user avatar
0 votes
1 answer
228 views

Unable to connect to spark

Running the Python code does not connect to Spark and does not create a database in Cassandra either. I have confirmed the services are up on docker and accessible from the PC. I placed the .jar files ...
Gregory Morris's user avatar
0 votes
1 answer
54 views

Filtering erroneous record from spark structured streaming dataframe while writing to cassandra

I know that there is some issue in nth row of my spark Scala dataframe(let's say data type is not proper). When I try to write this dataframe in cassandra using spark structured streaming, it fails ...
Aman Kumar's user avatar
0 votes
1 answer
61 views

Spark cassandra connection not pushing down the predicates/conditions. Instead filtering it in spark

I am trying to perform migration from one cassandra cluster (v3) to another cassandra cluster (v4). The table and schema is same in source and destination cluster as follows: CREATE KEYSPACE ...
Prashant Kalkar's user avatar
0 votes
1 answer
276 views

Can't connect Spark with Cassandra as docker containers

I'm running cassandra and spark docker containers as follows in my compose.yaml : spark: image: docker.io/bitnami/spark:3.4 container_name: spark_master environment: - SPARK_MODE=...
Amine SAIHI's user avatar

15 30 50 per page
1
2 3 4 5
67