1,004
questions
0
votes
1
answer
33
views
PySpark connection to Cassandra returns "Py4JJavaError: An error occurred while calling o54.start"
I'm try to make connection from Pyspark to Cassandra in virtual environment and the services is installed by docker. I've been using the --packages method to solve the dependencies but it seems doesn'...
1
vote
0
answers
24
views
Pyspark connector slow during joins
I'm building an application that allows me to use pyspark to combine an oracle and a cassandra table. The cassandra table's count is in a scale of 100s of millions with the oracle one at a few 1000.
...
0
votes
0
answers
57
views
Spark is not inserting data into Cassandra when using ```writeStream```
I'm trying to create a pipeline in streaming that is calling an API using Airflow, then processing it with Kafka and inserting the data into Cassandra using Spark. I'm struggling when inserting data ...
1
vote
0
answers
26
views
Ignoring codec because it collides with previously generated codec
i am trying to register custom code(for map) like below
val session: CqlSession = CassandraConnector.apply(spark.sparkContext).openSession()
val codecRegistry: MutableCodecRegistry = session....
2
votes
0
answers
49
views
register a custom codec in cassandra connector
I am using spark-cassandra-connector_2.11 and version 2.5.2 in my scala application and want to register a custom map codec, but facing issues.
Is there any way to register it. I did the same thing on ...
0
votes
1
answer
52
views
How To Use Spark Submit Operator With Cassandra Remote Server In Apache Airflow
I'm working airflow into the Docker container on WindowsPC. I have some problems with apache airflow spark submit operator. I want to write data to a remote Cassandra server.
When I was using df.write....
0
votes
1
answer
117
views
PySpark app returns "NoClassDefFoundError: com/datastax/spark/connector/util/Logging"
I had this error:
py4j.protocol.Py4JJavaError: An error occurred while calling o59.start.
: java.lang.NoClassDefFoundError: com/datastax/spark/connector/util/Logging
at java.base/java.lang....
0
votes
1
answer
45
views
Pyspark cassandra connector generates tombstones during writing
I understand that when inserting data, tombstones might be created because of existing null values in the columns of the dataframe. To mitigate this issue and minimize tombstones, insertion queries ...
0
votes
3
answers
268
views
Simple Python app failed to load, getting "ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra"
Context :
Sorry to bother you with I am struggling to install a Cassandra-Spark connector. My goal is to install it to use Spark-SQL since Cassandra has strong limitations to do requests.
I have :
...
0
votes
0
answers
36
views
Cassandra db restore database issue
i'm trying to restore cassandra database keyspace but in restore time face some issues
ERROR ----------
INFO [CompactionExecutor:1] 2023-12-19 06:32:07,415 CompactionTask.java:253 - Compacted (...
0
votes
0
answers
94
views
Spark-Cassandra read cassandra table returning NoSuchElementException
I have been trying to copy a table from one keyspace to another using Apache spark. I have been getting the java.util.NoSuchElementException on null partition when iterating via the .foreachPartition()...
0
votes
1
answer
228
views
Unable to connect to spark
Running the Python code does not connect to Spark and does not create a database in Cassandra either. I have confirmed the services are up on docker and accessible from the PC. I placed the .jar files ...
0
votes
1
answer
54
views
Filtering erroneous record from spark structured streaming dataframe while writing to cassandra
I know that there is some issue in nth row of my spark Scala dataframe(let's say data type is not proper). When I try to write this dataframe in cassandra using spark structured streaming, it fails ...
0
votes
1
answer
61
views
Spark cassandra connection not pushing down the predicates/conditions. Instead filtering it in spark
I am trying to perform migration from one cassandra cluster (v3) to another cassandra cluster (v4).
The table and schema is same in source and destination cluster as follows:
CREATE KEYSPACE ...
0
votes
1
answer
276
views
Can't connect Spark with Cassandra as docker containers
I'm running cassandra and spark docker containers as follows in my compose.yaml :
spark:
image: docker.io/bitnami/spark:3.4
container_name: spark_master
environment:
- SPARK_MODE=...