0

this program similar to word count.

I have a large file each line contains a key and 2 number

I need to sum the delta values by each key.

All keys are given, the text file won't have any key that is not in the list.

Problem is each time I run with the same input I get different sums.

public static ConcurrentDictionary<string, ulong> entries = new ConcurrentDictionary<string, ulong>();
//sequentially load keys into entries 

               ParallelOptions op = new ParallelOptions();
                op.MaxDegreeOfParallelism = nThread;
                Parallel.ForEach(lines, op, (one) =>
                    {
                        string[] parts = one.Split('\t');

                        string key = parts[1];

                        //get length
                        ulong len = Convert.ToUInt64(parts[4]) - Convert.ToUInt64(parts[3]);


                        if (entries.ContainsKey(key))
                        {
                                entries[key] += len;

                        }
                        else
                        {
                            Console.WriteLine("key not found: " + key);
                        }


                    });
1
  • 1
    Anytime you are using ContainsKey in a ConcurrentDictionary take a step back. It is almost always the wrong thing to do. The fact they key is there now doesn't mean it will be there in half a second when I try to read from it.
    – mjwills
    Jan 15, 2019 at 3:38

1 Answer 1

2

Accessing the value by its indexer isnt thread safe. Use one of the other methods instead which insure thread safety like AddOrUpdate through Func. However which approach you choose will depend entirely on your needs

entries.AddOrUpdate(key, len, (s, value) => value + len);

AddOrUpdate(TKey, Func, Func, TArg)

Uses the specified functions and argument to add a key/value pair to the ConcurrentDictionary if the key does not already exist, or to update a key/value pair in the ConcurrentDictionary if the key already exists.

It should be noted that, sometimes the ConcurrentDictionary may enter the lock, see that the value has changed since it read it, and try the delegate again. So it may have unexpected side effects in other certain situations

1
  • Thanks this works. But real problem is this locking method is even slower than the sequential method
    – Benny Ae
    Jan 18, 2019 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.