I have created Kafka Standalone.properties file to built a connection. The file is placed at home/kafka/config/connect-standalone.properties as given below:
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/home/kafka-connect-cassandra-sink-1.4.0/kafka-connect-cassandra-sink-1.4.0.jar
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
#listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
At second step I have added kafka-connect-cassandra-sink-1.4.0 file. The file is placed at the path home/kafka-connect-cassandra-sink-1.4.0 , the file is given below:
name=users-sink
connector.class=com.datastax.oss.kafka.sink.CassandraSinkConnector
tasks.max=10
loadBalancing.localDc=datacenter1
contactPoints=localhost
port=9042
username=...
password=...
topics=demo
topic.demo.demo.users.mapping=lastname=value.lastname, firstname=value.firstname, email=value.email
On my PC Zookeeper is already running and I have also started the Kafka with the following command bin/kafka-server-start.sh config/connect-standalone.properties
Now for connecting kafka to sink connector I used the following command sudo kafka/config/connect-standalone.properties kafka-connect-cassandra-sink-1.4.0/conf/cassandra-sink-standalone.properties &> standalone-mode.log &
My standalone-mode.log file is empty(I am assuming that means no error because in previous attempts there were errors in this log file which i have resolved ).
Now for uploading the text file through kafka sink connector in cassandra I have used the following command cat Desktop/users.txt | kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo --property "parse.key=true" --property "key.separator=:"; sleep 10;
Again there is no error on terminal. But Now the problem is when I am writing a query to view the data in cassandra database the table is empty:
user.txt file is given below:
Pruitt:{"lastname":"Pruitt", "firstname":"Allie", "email":"allie@example.com"}
Krause:{"lastname":"Krause", "firstname":"Duncan", "email":"duncan@example.com"}
Chase:{"lastname":"Chase", "firstname":"Juana", "email":"juana@example.com"}
Estrada:{"lastname":"Estrada", "firstname":"Edward", "email":"edward@example.com"}
Singleton:{"lastname":"Singleton", "firstname":"Marie", "email":"Marie@example.com"}
Poole:{"lastname":"Poole", "firstname":"Olivia", "email":"olivia@example.com"}
Marks:{"lastname":"Marks", "firstname":"Timothy", "email":"timothy@example.com"}
Suarez:{"lastname":"Suarez", "firstname":"Claud", "email":"claud@example.com"}
Sloan:{"lastname":"Sloan", "firstname":"Eloy", "email":"eloy@example.com"}
Rodriguez:{"lastname":"Rodriguez", "firstname":"Gale", "email":"gale@example.com"}
Bautista:{"lastname":"Bautista", "firstname":"Constance", "email":"Constance@example.com"}
Mcintyre:{"lastname":"Mcintyre", "firstname":"Donte", "email":"donte@example.com"}
Lang:{"lastname":"Lang", "firstname":"Willa", "email":"willa@example.com"}
Richmond:{"lastname":"Richmond", "firstname":"Dionne", "email":"dionne@example.com"}
&> standalone-mode.log &
while debugging so you can immediately see logs