0

I've been working on a multithreaded file archiver for a week now, it works exclusively on clean threads. Synchronization is achieved by monitors and AutoResetEvent. I allocated the number of threads to the number of cores like that:

public static int GetCoreCount()
        {
            int coreCount = 0;
            foreach (var item in new System.Management.ManagementObjectSearcher("Select * from Win32_Processor").Get())
            {
                coreCount += int.Parse(item["NumberOfCores"].ToString());
            }
            return coreCount;
        }

But that load my CPU max ~65%. And this load is far from uniform, it constantly falls and rises. Tell me. Does anyone have any idea how to use 100% processor capability? This is my Run() code :

 public void Run()
        {
            var readingThread = new Thread(new ThreadStart(ReadInFile));
            var compressingThreads = new List<Thread>();
            for (var i = 0; i < CoreManager.GetCoreCount(); i++)
            {
                var j = i;
                ProcessEvents[j] = new AutoResetEvent(false);
                compressingThreads.Add(new Thread(() => Process(j)));
            }
            var writingThread = new Thread(new ThreadStart(WriteOutFile));

            readingThread.Start();

            foreach (var compressThread in compressingThreads)
            {
                compressThread.Start();
            }

            writingThread.Start();

            WaitHandle.WaitAll(ProcessEvents);
            OutputDictionary.SetCompleted();

            writingThread.Join();
8
  • 1
    Are you sure your file system isn't the bottleneck? You could have all the threads you want, but if you're reading files from a slow disk/over a network connection there's a real limit there.
    – Wai Ha Lee
    Mar 3, 2019 at 9:28
  • 1
    time to use a profiler, don't try such activity with random tries.
    – OznOg
    Mar 3, 2019 at 9:30
  • @WaiHaLee yep. You are right. I understand that but if we talk about WinRar or 7Zip. They use maximum system resources or I don't understand anything about that? :D Mar 3, 2019 at 9:45
  • @OznOg but what exactly i should check? Proccesor load? I use that but the results +- are the same Mar 3, 2019 at 9:47
  • 1
    process load is not profiling: use a tool that tells you where the cpu is taken in your code, and where parallelization is effective
    – OznOg
    Mar 3, 2019 at 10:21

1 Answer 1

0

It's not possible to tell what is limiting your core usage without profiling, and also knowing how much data you are compressing in your test.

However I can say that in order to get good efficiency, which includes both full core utilization and close to a factor of n speedup for n threads over one thread, in pigz I have to create pools of threads that are always there, either running or waiting for more work. It is a huge impact to create and destroy threads for every chunk of data to be processed. I also have pools of pre-allocated blocks of memory for the same reason.

The source code at the link, in C, may be of help.

1
  • Yep, I saw that lib. Unfurtanaly I can't use ThreadPool but maybe I should create custom ThreadPool :D thank you ) Mar 4, 2019 at 6:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.