I have a C++ application I'm in the process of optimizing. What method or tool can I use to pinpoint where exactly my code is running slowly?
We started with Q&A. Technical documentation is next, and we need your help.
Whether you're a beginner or an experienced developer, you can contribute.
closed as off-topic by Andrew Medico, lpapp, Reto Koradi, EdChum, Raul Rene Jul 22 '14 at 7:38This question appears to be off-topic. The users who voted to close gave this specific reason:
|
|
|
If your goal is to use a profiler, use one of the suggested ones. However, if you're in a hurry and you can manually interrupt your program under the debugger while it's being subjectively slow, there's a simple way to find performance problems. Just halt it several times, and each time look at the call stack. If there is some code that is wasting some percentage of the time, 20% or 50% or whatever, that is the probability that you will catch it in the act on each sample. So that is roughly the percentage of samples on which you will see it. There is no educated guesswork required. If you do have a guess as to what the problem is, this will prove or disprove it. You may have multiple performance problems of different sizes. If you clean out any one of them, the remaining ones will take a larger percentage, and be easier to spot, on subsequent passes. This magnification effect, when compounded over multiple problems, can lead to truly massive speedup factors. Caveat: Programmers tend to be skeptical of this technique unless they've used it themselves. They will say that profilers give you this information, but that is only true if they sample the entire call stack, and then let you examine a random set of samples. (The summaries are where the insight is lost.) Call graphs don't give you the same information, because
They will also say it only works on toy programs, when actually it works on any program, and it seems to work better on bigger programs, because they tend to have more problems to find. They will say it sometimes finds things that aren't problems, but that is only true if you see something once. If you see a problem on more than one sample, it is real. P.S. This can also be done on multi-thread programs if there is a way to collect call-stack samples of the thread pool at a point in time, as there is in Java. P.P.S As a rough generality, the more layers of abstraction you have in your software, the more likely you are to find that that is the cause of performance problems (and the opportunity to get speedup). Added: It might not be obvious, but the stack sampling technique works equally well in the presence of recursion. The reason is that the time that would be saved by removal of an instruction is approximated by the fraction of samples containing it, regardless of the number of times it may occur within a sample. Another objection I often hear is: "It will stop someplace random, and it will miss the real problem". This comes from having a prior concept of what the real problem is. A key property of performance problems is that they defy expectations. Sampling tells you something is a problem, and your first reaction is disbelief. That is natural, but you can be sure if it finds a problem it is real, and vice-versa. ADDED: Let me make a Bayesian explanation of how it works. Suppose there is some instruction Then suppose we take just 2 stack samples, and we see instruction
The last column says that, for example, the probability that Suppose the prior assumptions are different. Suppose we assume P(f=0.1) is .991 (nearly certain), and all the other possibilities are almost impossible (0.001). In other words, our prior certainty is that
Now it says P(f >= 0.5) is 26%, up from the prior assumption of 0.6%. So Bayes allows us to update our estimate of the probable cost of Yet another way to look at it is called the Rule Of Succession. If you flip a coin 2 times, and it comes up heads both times, what does that tell you about the probable weighting of the coin? The respected way to answer is to say that it's a Beta distribution, with average value (number of hits + 1) / (number of tries + 2) = (2+1)/(2+2) = 75%. (The key is that we see So, even a very small number of samples can tell us a lot about the cost of instructions that it sees. (And it will see them with a frequency, on average, proportional to their cost. If |
|||||||||||||||||||||
|
|
You can use Valgrind with the following options
It will generate a file called |
|||||||||||||||||
|
|
I assume you're using GCC. The standard solution would be to profile with gprof. Be sure to add
I haven't tried it yet but I've heard good things about google-perftools. It is definitely worth a try. Related question here. A few other buzzwords if |
|||||||||||||||||||||
|
|
Newer kernels (e.g. the latest Ubuntu kernels) come with the new 'perf' tools ( These come with classic sampling profilers (man-page) as well as the awesome timechart! The important thing is that these tools can be system profiling and not just process profiling - they can show the interaction between threads, processes and the kernel and let you understand the scheduling and I/O dependencies between processes.
|
|||||||||||||
|
|
OProfile is good, because it makes it much easier than Gprof to profile multiple programs at once. You also can run it on your release build (if it has symbols), instead of having to build a special profiling build. If you don't care about taking a massive performance hit (50x), Valgrind (Cachegrind) is good. |
|||||||||||||||||||||
|
|
You can use callgrind. Together with KCacheGrind, it gives a pretty nice profiler. Besides that, Intel VTune is free for educational use on Linux. It is probably the best profiler out there. If you have an AMD CPU, use AMD Codeanalyst (succeeded by AMD’s CodeXL), which is also available for Linux; this one is only decent, but free. |
||||
|
I would use Valgrind and Callgrind as a base for my profiling tool suite. What is important to know is that Valgrind is basically a Virtual Machine:
Callgrind is a profiler build upon that. Main benefit is that you don't have to run your aplication for hours to get reliable result. Even one second run is sufficient to get rock-solid, reliable results, because Callgrind is a non-probing profiler. Another tool build upon Valgrind is Massif. I use it to profile heap memory usage. It works great. What it does is that it gives you snapshots of memory usage -- detailed information WHAT holds WHAT percentage of memory, and WHO had put it there. Such information is available at different points of time of application run. |
|||
|
|
|
This is a response to Nazgob's Gprof answer. I've been using Gprof the last couple of days and have already found three significant limitations, one of which I've not seen documented anywhere else (yet):
This was GNU Gprof (GNU Binutils for Debian) 2.18.0.20080103 running under 64-bit Debian Lenny, if that helps anyone. |
|||||||||||||
|
|
Oprofile is a decent free option, but I've had better luck using Zoom. It's a commercial (free eval) profiler for Linux that has a sweeet GUI for seeing hotspots in your source code. |
|||||||||
|
|
google-perftools is the only reasonable alternative to Gprof I've found. It's quite usable, familiar, and I believe it's time sampling, so that I/O bottlenecks are revealed, in addition to the usual CPU bottlenecks that Gprof discovers. It's also significantly less invasive. |
|||||||||||||
|
|
Use Valgrind I am pretty sure you can use Cachegrind or some similar plugin to do profiling. |
|||||
|
|
The answer to run So this is what I recommend. Run program first:
Now when it works and we want to start profiling we should run in another window:
This turns profiling on. To turn it off and stop whole task we might use:
Now we have some files named callgrind.out.* in current directory. To see profiling results use:
I recommend in next window to click on "Self" column header, otherwise it shows that "main()" is most time consuming task. "Self" shows how much each function itself took time, not together with dependents. |
|||||||||||||||||
|
|
You may have a look at Gprof, the GNU profiler. Another interesting tool may be IBM Rational Quantify, but it's not free. |
||||
|
|
|
In addition to Intel Vtune / AMD CodeAnalyst, perfmon2 is an OSS alternative, that requires a patched kernel to open up the CPU performance counter, and that would give you various performance figures that you can gather. perfmon2 is still implementation-specific, i.e. L2 cache misses are called different things on Intel Pentium III compared to AMD64, and they're beginning work on perfmon3, which should unify the API. But generally, Gprof would work well enough for you to detect slow code. |
||||
|
|
|
You can use Sun Studio's collect/analyzer. Using collect you can also profile memory usage, threads, MPI, etc. You also get a nice timeline view of your program. If you use these tools in Solaris you can also get hardware performance counter information like in vtune or oprofile. You can get this (and other very useful tools from Sun) at Oracle Solaris Studio 12.3. |
|||||
|
|
I've had very good luck with Rational Quantify (expensive but very good) and OProfile. Be aware when using OProfile that it's a statistical profiler, not a full-on tracing profiler like Quantify. OProfile uses a kernel module to poke into the call stack of every running process on every interval so certain behaviors may not be caught. Using multiple profilers is good, especially since different profilers tend to give you different data all of which is useful. As for using Gprof, it's OK. I would get one of the graphical front-ends, since the data can be rather difficult to get through just on the command line. I would avoid Valgrind, until you require memory checking. |
||||
|
|
|
There is also LTTng. I've never used that one though, so I cannot tell how well it works. But one advantage it has is an userspace tracer. In some situations that could be rather nice to have. |
|||||
|
|
Don't forget the poor man's profiler which is basically a wrapper for GDB so it can do sampling of where the threads "are" every so often. Basically it uses GDB to get the stacktraces for sampling, instead of lsstack, etc. A little unrelated, but Gprof itself can do sampling profiling, as well: Use callgrind as a sampling profiler? ("Flat profile is based mainly on sampling") And you can also use a USR1 signal et al to dump thread backtraces. |
|||||||||
|
|
You can check out Gprof from the GNU Project. Here are some quick start guides: |
||||
|
|
|
The PCT profiler takes the stack sampling approach advocated by other responders who did not suggest a specific tool. It can do instruction-level or procedure-level profiling. I have used it to profile non-C code in addition to C code (I have used it previously for OCaml with interesting results). |
||||
|
|
|
Use SysProf; It's pretty good in doing quick profiling with minimum hassle. It shows information like how many times a function has been called, time spent in each routine, overall time spent for each function, etc. Just install, do the initial configuration and then run your application/service (no need to recompile, instrument your code). |
|||||||||
|
|
You can try out AMD CodeXL's CPU profiler. It is free and available for Linux. AMD CodeXL's CPU profiler replaces the no longer supported CodeAnalyst tool (which was mentioned in an answer above given by Anteru). CodeXL supports several CPU profiling techniques for AMD CPUs (for non-AMD CPUs, only time based CPU profiling is supported). For more information and download links, visit: AMD CodeXL web page. |
|||
|
|
|
These are the two methods I use for speeding up my code: For CPU bound applications:
For I/O bound applications:
N.B. If you don't have a profiler, use the poor man's profiler. Hit pause while debugging your application. Most developer suites will break into assembly with commented line numbers. You're statistically likely to land in a region that is eating most of your CPU cycles. For CPU, the reason for profiling in DEBUG mode is because if your tried profiling in RELEASE mode, the compiler is going to reduce math, vectorize loops, and inline functions which tends to glob your code into an un-mappable mess when it's assembled. An un-mappable mess means your profiler will not be able to clearly identify what is taking so long because the assembly may not correspond to the source code under optimization. If you need the performance (e.g. timing sensitive) of RELEASE mode, disable debugger features as needed to keep a usable performance. For I/O-bound, the profiler can still identify I/O operations in RELEASE mode because I/O operations are either externally linked to a shared library (most of the time) or in the worst case, will result in a sys-call interrupt vector (which is also easily identifiable by the profiler). |
|||||||||||||
|
|
Use a profiler as most people suggest. I would recommend Very Sleepy. Be aware that Visual Studio has a built-in profiler itself, but it depends on the sampling method; you may profile CPU or I/O operations. |
|||||
|

codeprofilers. However, priority inversion, cache aliasing, resource contention, etc. can all be factors in optimizing and performance. I think that people read information into my slow code. FAQs are referencing this thread. – artless noise Mar 17 '13 at 18:44