0

sorry for my English in advance.

I am a beginner with Cassandra and his data model. I am trying to insert one million rows in a cassandra database in local on one node. Each row has 10 columns and I insert those only in one column family.

With one thread, that operation took around 3 min. But I would like do the same operation with 2 millions rows, and keeping a good time. Then I tried with 2 threads to insert 2 millions rows, expecting a similar result around 3-4min. bUT i gor a result like 7min...twice the first result. As I check on differents forums, multithreading is recommended to improve performance. That is why I am asking that question : is it useful to use multithreading to insert data in local node (client and server are in the same computer), in only one column family?

Some informations : - I use pycassa - I have separated commitlog repertory and data repertory on differents disks - I use batch insert for each thread - Consistency Level : ONE - Replicator factor : 1

1
  • Did you ever find a solution to this? I have a similar problem and I can't find any solution.
    – bwight
    Mar 25, 2013 at 16:40

4 Answers 4

0

It's possible you're hitting the python GIL but more likely you're doing something wrong.

For instance, putting 2M rows in a single batch would be Doing It Wrong.

2
  • Thanks for your answer. I am not putting 2M rows in a single batch. Each thread connects on database, and insert 1M rows with a batch configured with a queue_size of 1000.
    – C. Oran
    May 10, 2011 at 20:44
  • You should start with contrib/stress because then you know the load-generation part isn't the problem.
    – jbellis
    May 17, 2011 at 21:40
0

Try running multiple clients in multiple processes, NOT threads.

Then experiment with different insert sizes.

1M inserts in 3 mins is about 5500 inserts/sec, which is pretty good for a single local client. On a multi-core machine you should be able to get several times this amount provided that you use multiple clients, probably inserting small batches of rows, or individual rows.

1
  • Thank you for your help. I am just change my program with multiple processes (one fork) but no improvement. I am thinking that would be my program is too complex : it reads on csv file, insert data in batchs, then send those to database. I used the time command and user time was around 2min... For a single local client (with 4cores), it is probably to much to schedule my program and the database in a same time
    – C. Oran
    May 11, 2011 at 8:30
0

You might consider Redis. Its single-node throughput is supposed to be faster. It's different from Cassandra though, so whether or not it's an appropriate option would depend on your use case.

1
  • Yes I heard about Redis, but it is not a good match for my data model, unfortunatly...
    – C. Oran
    May 22, 2011 at 19:01
0

The time taken doubled because you inserted twice as much data. Is it possible that you are I/O bound?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.