6

I recently discovered at work that it is the policy not to use compiler optimizations for hard real time embedded systems because of the risk of compiler bugs (we mainly use gcc but the policy extends to other compilers as well). Apparently this policy started because someone was burnt in the past by a bug with an optimizer. My gut feeling is that this is being overly paranoid so I've started looking for data on this issue but the problem is I can't find any hard data on this.

Does anyone know of a way to actually get this type of data? Can the gcc bugzilla page be used to generate some statistics of bugs vs compiler optimization level? Is it even possible to get unbiased data like this?

3

7 Answers 7

4

You are assuming the compiler is bug free without optimizations and only the optimizations are dangerous. the compilers themselves are programs and very often have bugs with or without using certain features. Sure the features might make it better or they might make it worse.

Llvm was mentioned in another answer, there is a well know llvm optimization bug that they appear to have zero interest in fixing

while(1) continue;

gets optimized out, just goes away...sometimes...and other similar but not completely infinite loops also disappear in the llvm optimizer. Leaving you with a binary that doesnt match your source code. This is one I know there are probably many more in both gcc and llvm's compilers.

gcc is a monster that is barely held together with duct tape and bailing wire. It is like watching one of those faces of death movies or something like that once you have had those images in your head one time, you cant unwatch them, they are burned in there for life. So it is worth finding out for yourself how scary gcc is, by looking behind the curtain. But you might not be able to forget what you had seen. For various targets -O0 -O1 -O2 -O3 can and have all failed miserably with some code at some point in time. Likewise the fix sometime is to optimize more not less.

When you write a program the hope is the compiler does what it says it does, just like you hope your program does what you say it does. But it is not always the case, your debugging does not end when the source code is perfect, it ends when the binary is perfect, and that includes whatever binary and operating system you hope to target (different minor versions of gcc make different binaries, different linux targets react differently to programs).

The most important advise is develop and test using the target optimization level. if you develop and test by always building for a debugger, well you have a created a program that works in a debugger, you get to start over when you want to make it work somewhere else. gcc's -O3 does work often but folks are afraid of it and it doesnt get enough usage to be debugged properly, so it is not as reliable. -O2 and no optimization -O0 get a lot of mileage, lots of bug reports, lots of fixes, choose one of those or as another answer said, go with what Linux uses. Or go with what firefox uses or go with what chrome uses.

Now hard realtime embedded systems. Man mission systems, systems where life or property are directly affected. First why are you using gcc? Second, yes, optimizers are often NOT used in these environments, it creates too much risk and/or greatly increases the testing and validation effort. Normally you want to use a compiler that has been through a lot of testing itself and its warts and traps are well known. Do you want to be the person who turned on the optimizer, and as a result the flight computer crashed the airplane into an elementary school on a school day? There is a lot to be learned from the old timers. yes they have a lot of war stories, and a lot of fear of new fangled things. dont repeat history, learn from it. "They dont build em like they used to" means something it is not just a saying. those legacy systems were stable and reliable and still running for a reason, partly those old timers and what they learned the hard way, and partly because newer stuff is built cheaper and with lower quality components.

For this class of environment you definitely dont stop at the source code, your money and time is poured into validating the BINARY. Each time you change the binary you need to start validation over again. No different than the hardware it runs on, you change one component you, warm up one solder joint, you start validation testing over again from the beginning. One difference perhaps is that in some of these environments each solder joint is only allowed a maximum number of cycles before you scrap the whole unit. But it can be the case in software, only so many burn cycles on the prom before you scrap the prom and only so many rework cycles on the prom pads/holes before you scrap the board/unit. Leave the optimizer off and find a better, more stable, compiler and/or programming language.

Now if this hard real time environment is not going to hurt people or property (other than what it runs on) when it crashes, then that is another story. Maybe its a blue ray player and it skips a frame here and there or displays a few bad pixels, big deal. Turn the optimizer on, the masses dont care about that level of quality anymore, they are content with youtube quality images, compressed video formats, etc. Cars that have to be turned off and on again for the radio or bluetooth to work. Doesnt bother them one bit, turn the optimizer on and claim performance gain over your competitor. If the software is too buggy to tolerate the customers will work around it or just buy someone elses, when that one fails they will come back to you and buy your new model with the newer firmware. They will continue to do this because they want to dancing baloney, they dont want stability nor quality. That stuff costs too much.

You should collect your own data, try the optimizers on the software in your environment and run the product through a full validation suite. If it doesnt break then either the optimizer for that code that day is okay or the test system needs more work. If you cannot do that then you can at least disassemble and analyze what the compiler is doing with your code. I would assume (and know from personal experience) that both gcc and llvm bug systems do have bugs that are tied to optimization levels, does that mean you can sort them based on optimization level? Dont know, these are open source, largely uncontrolled interfaces, so you cant rely on the masses to accurately and completely define the input fields, if there were an optimization field on the bug report form, it is probably always set to the default for the form/web page. You have to examine the problem report itself to see if the user had problems related to the optimizer. If this were a closed system for a corporation where an employees performance review might be negatively reflected for not following procedure like filling out forms correctly, you would have better searchable databases to draw information from.

The optimizer does increase your risk. Lets say 50% of the compiler is used to get an output with no optimization, another 10% to get -O1, you have increased your risk, more compiler code used, more risk of having a bug, more risk in the output being bad. and more code is used to get to -O2 and -O3. Reducing optimization doesnt eliminate the risk completely but does reduce the odds.

5
  • Do you have any reference for LLVM optimizing out infinite loops? And what exactly do you mean by LLVM optimizing out this loop? That it executes code coming after the code? Jul 24, 2012 at 13:46
  • IIRC the LLVM optimization is allowed by C standard as it does not have side effects. May 25, 2013 at 14:31
  • what optimization, the bug I pointed out, that has major side effects, it completely changes the flow of code.
    – old_timer
    May 25, 2013 at 17:48
  • @dwelch: Suppose one had a function void foo(void) { long long exp=3; while(!FermatCounterexampleExistsWithExponent(exp) exp++; someGlobal = exp;} and some other function called foo() and then did some other stuff. If the Fermat... method had no side-effects, a compiler would be free to split off a separate thread to handle foo if it ensured that the primary thread would wait for the secondary thread to complete before the next access to someGlobal. If everything useful the program was going to do could be accomplished without ever actually accessing someGlobal...
    – supercat
    Feb 10, 2014 at 17:52
  • ...the spec would allow the program to do all those things without the loop ever having completed. While I would posit that a compiler should probably squawk if it sees a while loop that has no break, doesn't access any volatile variables, doesn't write to any pointers, and can't modify the looping condition, I have no problem with the spec saying that the time required to execute a piece of code (even if infinite) should not be considered a "side-effect".
    – supercat
    Feb 10, 2014 at 17:58
2

I don't have any data (and haven't heard of anyone that does ...) but ...

I'd choose which compiler I would use before I'd choose to disable optimizations. In other words, I wouldn't use any compiler I couldn't trust the optimizations on.

The linux kernel is compiled with -Os. That's a lot more convincing to me than any bugzilla analysis.

Personally, I'd be okay with any version of gcc linux is okay with.

As another data point, Apple's been converting from gcc to llvm, with and without clang. llvm has traditionally had issues with some C++ and while llvm-gcc is now a lot better, there still seem to be issues with clang++. But that just kind of proves the pattern: while Apple (purportedly) now compiles OS X and iOS with clang, they don't use much if any C++ and Objective C++. So for pure C and Objective C, I'd trust clang, but I still don't yet trust clang++.

4
  • 1
    Unfortunately the argument has been "if it isn't real time it probably isn't critical if it crashes". I've been looking for examples of critical systems that would be compiled with optimizations (if Linux is compiled with -Os something that runs Linux would suffice). Something like a root nameserver or a core router on the internet would be good. Really anything that would be catastrophic if it failed even remotely often, the bigger the impact the better. Unfortunately finding specifics on systems like that have been (understandably) hard to find.
    – nightrain
    Jan 30, 2012 at 14:16
  • Every thing Android is running Linux, plenty of consumer routers, PlayStations. "if it isn't real time it probably isn't critical if it crashes" is a pretty irrational definition. As such, I can see it could be hard to disuade. Arguably, impossible.
    – smparkes
    Jan 30, 2012 at 18:06
  • 5
    And I gotta wonder: if you're not using optimizations, does that mean you're hand-optimizing code? That's gotta be more error prone.
    – smparkes
    Jan 30, 2012 at 19:11
  • 3
    I think @smparkes comment deserves more attention. All programs have a certain acceptable minimum speed. One way to achieve that speed is by turning on optimization settings. Another way is rewriting code to be 'hand-optimized'. If it comes down to one or the other, I would pick turning on optimization settings, personally. Mar 25, 2012 at 18:57
2

Is using a compiler safe ?

A compiler, by design, transforms your code into another form. It normally should transform it correctly, but as all software there may be a bug lurking there. So no it is not safe.

What can make code safe ?

Testing/Usage.

For bugs to manifest, the code that contain them must be run in a particular configuration. For any non-trivial piece of software, it is nigh impossible to prove the absence of bugs, however heavy testing and heavy usage tend to at least clear some paths of execution.

So, how can I be safe ?

Well, by using the same paths that every one else does. This gives you the best chance that the path is bug-free, with all the people that already went through there.

For gcc then ? I would use -O2 or -Os (like Linux does) because those are likely to have received a tremendous amount of scrutiny, either direct or indirect.

Should you turn on optimizations ?

However, introducing optimizations into a tool-chain is disruptive. It requires more than just switching the flip. You need to perform heavy testing to make sure that in your conditions nothing bad happens.

More specifically, compilers rely on undefined behavior to perform a number of optimizations. If your code has never been exposed to optimizations, then it is very likely to rely on such undefined behavior here and there, and turning optimizations may expose those bugs (not introduce them).

It's no more disruptive than switching compilers though.

1

In embedded systems it is frequently necessary to control the hardware by writing to registers. In C this is very easy, just initialise a pointer with the register address and away you go.

If no other part of the program reads or writes the register it is quite likely that the optimiser will remove the assignment. Breaking the code.

This particular problem can be fixed using the "volatile" keyword. But don't forget that the optimiser also changes the sequence of commands. So if your hardware expects registers to be written in a certain order you may be burned.

An optimiser is supposed to produce correct results, but the interim steps may change, this is where you can be hurt by an optimiser.

0

In my understanding, most compiler optimizations are safe for all programs, except the scheduling and rearrangement optimizations. Because This type of optimization could change the original program's behavior.

For the datas about this problem, you can check:

Can compiler optimization introduce bugs?

0

I don't know about gcc bugs but the C programming language is ill fitting for current hardware. Remember it was conceived in the 1970's when it wasn't even clear that 2's complement arithmetic was to be the future. Ok, you add 2 unsigned integers in C. The specifications say you are not allowed to overflow. The compiler can assume that the carry flag is clear after the add and make further optimizations based on that. You assume 2's complements arithmetic (and who wouldn't these days) and ka-boom you just set the carry flag. Things like this are a major source of security problems. I think Java is probably better even for low levelish code because it is much better defined and the current HotSpot just in time compiler produces code that runs as fast a C. You could look at the D programming language as well, likely it is well defined too.

1
  • What's ironic is that compiler writers eagerly exploit "optimization opportunities" posed by cases where the Standard made allowances for weird hardware, than in defining directives that are based upon the actual needs of modern compilers.
    – supercat
    Jun 16, 2017 at 23:06
0

A recent study on compiler optimizer bugs https://www.sciencedirect.com/science/article/abs/pii/S0164121220302740

Yes, compilers have bugs, and the optimizer can add bugs to your code ;-(

A useless solution would be too compile and test with several optimization choices and compare program results.

As programs grow in size and complexity it becomes infeasible to run enough tests to provide assurance that your code does not become buggier because of the compiler.

It is possible to perform a risk estimate, am I working on a small project where performance is key? Or am I working on the kernel or some other application which is mission critical? If the former, then you may be comfortable with all the optimizations the compiler may provide. If the latter, I suggest starting with '-O0' and then '-O1' and then ... measuring your code's key performance metrics. You may find that '-O2' and above provide negligible benefit and are not worth the risk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.