586

Once again I was in a design review, and encountered the claim that the probability of a particular scenario was "less than the risk of cosmic rays" affecting the program, and it occurred to me that I didn't have the faintest idea what that probability is.

"Since 2-128 is 1 out of 340282366920938463463374607431768211456, I think we're justified in taking our chances here, even if these computations are off by a factor of a few billion... We're way more at risk for cosmic rays to screw us up, I believe."

Is this programmer correct? What is the probability of a cosmic ray hitting a computer and affecting the execution of the program?

28
  • 49
    "Winning Lotteries: What is the probability they will affect a program?"
    – kennytm
    Apr 5, 2010 at 20:29
  • 31
    It depends in part on where the program is being executed and how well it's shielded. On Earth, the cosmic ray flux is much lower than in deep space, or even near Earth orbit. The Hubble Space Telescope, for instance, produces raw images that are riddled with cosmic ray traces. Apr 5, 2010 at 20:30
  • 94
    Does this mean that from now on, when someone next asks about finally blocks, we'll have to qualify it with "always executes except if the program exits, or if it gets hit with a cosmic ray"?
    – skaffman
    Apr 5, 2010 at 20:42
  • 77
    Working on a prototype particle detector years ago, I programmed it to print "ouch!" every time it was hit by a cosmic ray. Good times...
    – Beta
    Apr 5, 2010 at 21:42
  • 9
    On of the most interesting questions I've read here in a while. A real eye-opener. Count on me to re-open. Apr 6, 2010 at 6:08

15 Answers 15

322

From Wikipedia:

Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month.[15]

This means a probability of 3.7 × 10-9 per byte per month, or 1.4 × 10-15 per byte per second. If your program runs for 1 minute and occupies 20 MB of RAM, then the failure probability would be

                 60 × 20 × 1024²
1 - (1 - 1.4e-15)                = 1.8e-6 a.k.a. "5 nines"

Error checking can help to reduce the aftermath of failure. Also, because of more compact size of chips as commented by Joe, the failure rate could be different from what it was 20 years ago.

17
  • 15
    More importantly, the chip feature size for CPUs in 1995 was around 0.35 µm or 350nm. It's now 1/10th that size at 35nm.
    – Joe Koberg
    Apr 5, 2010 at 21:02
  • 26
    Is it possible that instead of reducing risk, decreased size would increase risk since it would take less energy to change the state of each bit?
    – Robert
    Apr 5, 2010 at 21:59
  • 71
    Reduced size definitely increases risk. Hardened processors for space vehicles use very large feature sizes to avoid cosmic ray effects.
    – Joe Koberg
    Apr 5, 2010 at 23:27
  • 26
    Not just cosmic rays, radioactive isotopes in the materials used in the chip are a much bigger problem. Makers go to huge lengths to make sure the silicon, solder, encapsulation etc doesn't contain any alpha or beta emitters. Apr 6, 2010 at 3:12
  • 19
    Wow! This means that about 1 byte in my PC gets corrupted every two days. Sep 26, 2010 at 7:10
98

Apparently, not insignificant. From this New Scientist article, a quote from an Intel patent application:

"Cosmic ray induced computer crashes have occurred and are expected to increase with frequency as devices (for example, transistors) decrease in size in chips. This problem is projected to become a major limiter of computer reliability in the next decade. "

You can read the full patent here.

8
  • 9
    Why do they increase with a decrease in size of the chip? Surely a smaller object is less likely to be hit by a ray (i.e. compare throwing a tennis ball at a wall, to throwing it at a stamp)
    – Jonathan.
    May 13, 2015 at 16:28
  • 14
    Because as the size of components shrinks, they become much more sensitive to cosmic ray hits. May 13, 2015 at 17:14
  • 7
    Yes, smaller equals less likely to be hit, but more likely that the hit will affect the state. May 13, 2015 at 18:09
  • 2
    @ire_and_curses [citation needed]
    – Anko
    May 13, 2015 at 20:15
  • 12
    @Anko - It's kind-of obvious. As a given component gets smaller, it needs less voltage and less charge to set a bit. That makes it more sensitive to being blasted with energy from outer space. However, here's a citation for you: As LSI memory devices become smaller, they become more sensitive to nuclear-radiation-induced soft fails. May 13, 2015 at 21:18
70

Note: this answer is not about physics, but about silent memory errors with non-ECC memory modules. Some of errors may come from outer space, and some - from inner space of desktop.

There are several studies of ECC memory failures on large server farms like CERN clusters and Google datacenters. Server-class hardware with ECC can detect and correct all single bit errors, and detect many multi-bit errors.

We can assume that there is lot of non-ECC desktops (and non-ECC mobile smartphones). If we check the papers for ECC-correctable error rates (single bitflips), we can know silent memory corruptions rate on non-ECC memory.

  • Large scale CERN 2007 study "Data integrity": vendors declares "Bit Error Rate of 10-12 for their memory modules", "a observed error rate is 4 orders of magnitude lower than expected". For data-intensive tasks (8 GB/s of memory reading) this means that single bit flip may occur every minute (10-12 vendors BER) or once in two days (10-16 BER).

  • 2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year".

  • 2012 Sandia report "Detection and Correction of Silent Data Corruptionfor Large-Scale High-Performance Computing": "double bit flips were deemed unlikely" but at ORNL's dense Cray XT5 they are "at a rate of one per day for 75,000+ DIMMs" even with ECC. And single-bit errors should be higher.

So, if the program has large dataset (several GB), or has high memory reading or writing rate (GB/s or more), and it runs for several hours, then we can expect up to several silent bit flips on desktop hardware. This rate is not detectable by memtest, and DRAM modules are good.

Long cluster runs on thousands of non-ECC PCs, like BOINC internet-wide grid computing will always have errors from memory bit-flips and also from disk and network silent errors.

And for bigger machines (10 thousands of servers) even with ECC protection from single-bit errors, as we see in Sandia's 2012 report, there can be double-bit flips every day, so you will have no chance to run full-size parallel program for several days (without regular checkpointing and restarting from last good checkpoint in case of double error). The huge machines will also get bit-flips in their caches and cpu registers (both architectural and internal chip's triggers e.g. in ALU datapath), because not all of them are protected by ECC.

PS: Things will be much worse if the DRAM module is bad. For example, I installed new DRAM into laptop, which died several weeks later. It started to give lot of memory errors. What I get: laptop hangs, linux reboots, runs fsck, finds errors on root filesystem and says that it want to do reboot after correcting errors. But at every next reboot (I did around 5-6 of them) there are still errors found on the root filesystem.

5
  • 12
    Additional material from BH 2011: "Bitsquatting. DNS hijacking without exploitation" media.blackhat.com/bh-us-11/Dinaburg/… lists modern multi-GB DRAMs to have around 10000-30000 FIT/Mbit (less than 100 hours between errors for every 128MB). The paper also lists articles which conclude that most soft errors are from radiation, almost all cases - from cosmic rays, and some cases from alpha-emitters inside PC. BH authors did experiment and got 50000 accesses to domains, having 1 bit changed from popular sites
    – osgx
    May 14, 2014 at 10:45
  • Kudos for adding more recent studies here. Given the dynamics of SO voting and how they are accumulated over time, it is alas difficult to have an up-to-date presentation on this topic stand out (here).
    – Fizz
    Jan 12, 2015 at 14:06
  • We had similar problem. We didn't do any exact study, but we had quite some crash dumps with visible bit flips. We checked those bit flips and it turned out they were in code section. We compared with what should be there and it did not look like deliberate modification (i.e. resulting instructions did not have much sense). In the end we had simple application, that compared crash dumps against (archived) released versions and filtered out such cases. Interestingly I think most of such cases were coming from Iran, Arabia and I think one more country from South America (don't remember now).
    – GiM
    Aug 17, 2017 at 6:25
  • 2
    In Google's paper it looks more like a case that some RAM is bad About a third of machines and over 8% of DIMMs in our fleet saw at least one correctable error per year. Our per-DIMM rates of correctable errors translate to an aver- age of 25,000–75,000 FIT (failures in time per billion hours of operation) per Mbit and a median FIT range of 778 – 25,000 per Mbit (median for DIMMs with errors), while pre- vious studies report 200-5,000 FIT per Mbit. The number of correctable errors per DIMM is highly variable, with some DIMMs experiencing a huge number of errors, compared to others.
    – vartec
    Sep 18, 2017 at 15:52
  • OMG! There're already a lot of bit flips in this answer: "one-bit FIT per Mbit", "Corruptionfor"
    – TeaDrinker
    May 4, 2022 at 13:56
31

Wikipedia cites a study by IBM in the 90s suggesting that "computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month." Unfortunately the citation was to an article in Scientific American, which didn't give any further references. Personally, I find that number to be very high, but perhaps most memory errors induced by cosmic rays don't cause any actual or noticable problems.

On the other hand, people talking about probabilities when it comes to software scenarios typically have no clue what they are talking about.

9
  • 7
    The probability of a bit being flipped must be multiplied by the probability of the bit having a noticeable affect on the program. I'm guessing the second probability is a lot lower than you think. Apr 5, 2010 at 21:18
  • 2
    @Mark: Typical computer programs don't have that kind of fault-tolerance built-in. A single-bit error in the program code will more likely than not crash the program, if the broken code is executed. Apr 6, 2010 at 3:05
  • 76
    Yes, but most of the memory contains data, where the flip won't be that visiblp.
    – zoul
    Apr 6, 2010 at 5:01
  • 35
    @zoul. lol at 'visiblp', but if e=1100101 and p=1110000 then you're the unfortunate victim of 3 bit flips!
    – PaulG
    Apr 8, 2010 at 8:47
  • 11
    @Paul: or one finger blip.
    – mpen
    May 9, 2010 at 7:29
31

Well, cosmic rays apparently caused the electronics in Toyota cars to malfunction, so I would say that the probability is very high :)

Are cosmic rays really causing Toyota woes?

9
  • 24
    "Federal regulators are studying whether sudden acceleration in Toyotas is linked to cosmic rays." This is why you should never give federal regulators power over your life.
    – user1228
    Apr 5, 2010 at 20:33
  • 13
    I guess the theory here is that cosmic rays are flipping bits in older brains causing them to malfunction and press the wrong pedal.
    – Knox
    Apr 5, 2010 at 20:43
  • 16
    "Apparently"? I'd say that's a wild guess at this point. My own wild guess is that this phenomenon is a result of that old nightmare of embedded systems (actually most complex computer systems) - the race condition. Apr 5, 2010 at 20:49
  • 8
    @Knox: Get out your old tinfoil hat, it is useful!
    – Roger Pate
    Apr 6, 2010 at 5:31
  • 3
    It may not be a joke. I've seen some seriously weird stuff like that happen before. Not as rare as most people think. Apr 6, 2010 at 17:32
28

With ECC you can correct the 1 bit errors of Cosmic Rays. In order to avoid the 10% of cases where cosmic rays result in 2-bit-errors the ECC cells are typically interleaved over chips so no two cells are next to each other. A cosmic ray event which affects two cells will therefore result in two correctable 1bit errors.

Sun states: (Part No. 816-5053-10 April 2002)

Generally speaking, cosmic ray soft errors occur in DRAM memory at a rate of ~10 to 100 FIT/MB (1 FIT = 1 device fail in 1 billion hours). So a system with 10 GB of memory should show an ECC event every 1,000 to 10,000 hours, and a system with 100 GB would show an event every 100 to 1,000 hours. However, this is a rough estimation that will change as a function of the effects outlined above.

18

Memory errors are real, and ECC memory does help. Correctly implemented ECC memory will correct single bit errors and detect double bit errors (halting the system if such an error is detected.) You can see this from how regularly people complain about what seems to be a software problem that is resolved by running Memtest86 and discovering bad memory. Of course a transient failure caused by a cosmic ray is different to a consistently failing piece of memory, but it is relevant to the broader question of how much you should trust your memory to operate correctly.

An analysis based on a 20 MB resident size might be appropriate for trivial applications, but large systems routinely have multiple servers with large main memories.

Interesting link: http://cr.yp.to/hardware/ecc.html

The Corsair link in the page unfortunately seems to be dead, so view the Corsair link here instead.

2
  • Cosmic ray bitflips may not be uniformly distributed, particularly if we include solar storms under the "cosmic ray events"-umbrella. If you got two or more bitflips within the same byte, the typical ECC won't be able to correct the error.
    – tobixen
    Nov 15, 2017 at 9:48
  • @tobixen Detecting a double bit error is better than continuing to run with bad data. The next step after ECC is Chipkill with DIMM mirroring ...
    – janm
    Nov 16, 2017 at 5:33
16

This is a real issue, and that is why ECC memory is used in servers and embedded systems. And why flying systems are different from ground-based ones.

For example, note that Intel parts destined for "embedded" applications tend to add ECC to the spec sheet. A Bay Trail for a tablet lacks it, since it would make the memory a bit more expensive and possibly slower. And if a tablet crashes a program every once in a blue moon, the user does not care much. The software itself is far less reliable than the HW anyway. But for SKUs intended for use in industrial machinery and automotive, ECC is mandatory. Since here, we expect the SW to be far more reliable, and errors from random upsets would be a real issue.

Systems certified to IEC 61508 and similar standards usually have both boot-up tests that check that all RAM is functional (no bits stuck at zero or one), as well as error handling at runtime that tries to recover from errors detected by ECC, and often also memory scrubber tasks that go through and read and write memory continuously to make sure that any errors that occur get noticed.

But for mainstream PC software? Not a big deal. For a long-lived server? Use ECC and a fault handler. If an uncorrectable error kills the kernel, so be it. Or you go paranoid and use a redundant system with lock-step execution so that if one core gets corrupted, the other one can take over while the first core reboots.

1
  • Cosmic ray bitflips may not be uniformly distributed, particularly if we include solar storms under the "cosmic ray events"-umbrella. A sudden burst may cause several bitflips within a byte, and ECC-algorithms won't be able to correct a failure.
    – tobixen
    Nov 15, 2017 at 9:47
13

If a program is life-critical (it will kill someone if it fails), it needs to be written in such a way that it will either fail-safe, or recover automatically from such a failure. All other programs, YMMV.

Toyotas are a case in point. Say what you will about a throttle cable, but it is not software.

See also http://en.wikipedia.org/wiki/Therac-25

5
  • Nevermind the software for throttles. The sensors and wiring for the throttles is the weak point. My Mitsubishi throttle position sensor failed into a random number generator... No unintended acceleration, but it sure didn't do anything good for the fuel mixture! Apr 6, 2010 at 17:31
  • 3
    @Brian: Good software would have figured out that the data points were discontinuous, and concluded that the data was bad. Apr 6, 2010 at 19:19
  • ..and then what... Good data is required. Knowing it's bad doesn't help any. Not something you can magically work around. Apr 6, 2010 at 19:50
  • 4
    @Brian: Well, for one thing, you can take corrective action based on the knowledge that your data is bad. You can stop accelerating, for instance. Apr 6, 2010 at 20:29
  • Yes you can (and should) cheksum data. Best end-to-end. However this only reduces the chances of corruption. Imagine your "is this valid" instruction gets the bit corrupted in memory or CPU register just when you want to branch to the error handler.
    – eckes
    Sep 26, 2013 at 2:25
12

"cosmic ray events" are considered to have a uniform distribution in many of the answers here, this may not always be true (i.e. supernovas). Although "cosmic rays" by definition (at least according to Wikipedia) comes from outer space, I think it's fair to also include local solar storms (aka coronal mass ejection) under the same umbrella. I believe it could cause several bits to flip within a short timespan, potentially enough to corrupt even ECC-enabled memory.

It's well-known that solar storms can cause quite some havoc with electric systems (like the Quebec power outage in March 1989). It's quite likely that computer systems can also be affected.

Some 10 years ago I was sitting right next to another guy, we were sitting with each our laptops, it was in a period with quite "stormy" solar weather (sitting in the arctic, we could observe this indirectly - lots of aurora borealis to be seen). Suddenly - in the very same instant - both our laptops crashed. He was running OS X, and I was running Linux. Neither of us are used to the laptops crashing - it's a quite rare thing on Linux and OS X. Common software bugs can more or less be ruled out since we were running on different OS'es (and it didn't happen during a leap second). I've come to attribute that event to "cosmic radiation".

Later, "cosmic radiation" has become an internal joke at my workplace. Whenever something happens with our servers and we cannot find any explanation for it, we jokingly attribute the fault to "cosmic radiation". :-)

11

I once programmed devices which were to fly in space, and then you (supposedly, noone ever showed me any paper about it, but it was said to be common knowledge in the business) could expect cosmic rays to induce errors all the time.

2
  • 10
    Above the atmosphere two things happen: 1) the total flux is higher 2) much more of it comes in the form of heavy, very energetic particles (with enough energy to flip a bit packed into a small space). Apr 5, 2010 at 21:22
  • 2
    With regard to references, there are books (e.g., books.google.com/books?hl=en&lr=&id=Er5_rzW0q3MC), conferences (e.g., radecs2015.org , seemapld.org , and others), and papers galore on this topic. Cosmic rays are not a joke in aerospace. They are one of the key reasons that many spacecraft use rad hardened computers, most of which have the processing power of a modern smart toaster oven. Sep 28, 2015 at 9:42
7

More often, noise can corrupt data. Checksums are used to combat this on many levels; in a data cable there is typically a parity bit that travels alongside the data. This greatly reduces the probability of corruption. Then on parsing levels, nonsense data is typically ignored, so even if some corruption did get past the parity bit or other checksums, it would in most cases be ignored.

Also, some components are electrically shielded to block out noise (probably not cosmic rays I guess).

But in the end, as the other answerers have said, there is the occasional bit or byte that gets scrambled, and it's left up to chance whether that's a significant byte or not. Best case scenario, a cosmic ray scrambles one of the empty bits and has absolutely no effect, or crashes the computer (this is a good thing, because the computer is kept from doing harm); but worst case, well, I'm sure you can imagine.

1
  • Cosmic ray bitflips may not be uniformly distributed, particularly if we include solar storms under the "cosmic ray events"-umbrella. If you got two bitflips within the same byte, the parity bit check will fail. Several bitflips, and ECC-algorithms probably won't be able to correct a failure.
    – tobixen
    Nov 15, 2017 at 9:46
6

I have experienced this - It's not rare for cosmic rays to flip one bit, but it's very unlikely that a person observe this.

I was working on a compression tool for an installer in 2004. My test data was some Adobe installation files of about 500 MB or more decompressed.

After a tedious compression run, and a decompression run to test integrity, FC /B showed one byte different.

Within that one byte the MSB had flipped. I also flipped, worrying that I had a crazy bug that would only surface under very specific conditions - I didn't even know where to start looking.

But something told me to run the test again. I ran it and it passed. I set up a script to run the test 5 times overnight. In the morning all 5 had passed.

So that was definitely a cosmic ray bit flip.

4
  • Definitely? Couldn't it have been an uninitialized variable that never got a bad initial value in the subsequent tests?
    – doug65536
    Jan 6, 2016 at 21:34
  • I always compile with W3 or W4 on VS - Also Rational Purify, there were no bugs of that sort.
    – rep_movsd
    Jan 6, 2016 at 21:49
  • Ah, sorry I didn't know that those compiler options and Rational Purify were utterly infallible. =)
    – doug65536
    Jan 25, 2016 at 6:51
  • Considering that the code was then put into production and compressed and uncompressed hundreds of GB properly, there was no sign of a similar bug.
    – rep_movsd
    Jan 25, 2016 at 14:51
5

As a data point, this just happened on our build:

02:13:00,465 WARN  - In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/ostream:133:
02:13:00,465 WARN  - /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/locale:3180:65: error: use of undeclared identifier '_'
02:13:00,465 WARN  - for (unsigned __i = 1; __i < __trailing_sign->size(); ++_^i, ++__b)
02:13:00,465 WARN  - ^
02:13:00,465 WARN  - /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/locale:3180:67: error: use of undeclared identifier 'i'
02:13:00,465 WARN  - for (unsigned __i = 1; __i < __trailing_sign->size(); ++_^i, ++__b)
02:13:00,465 WARN  - ^

That looks very strongly like a bit flip happening during a compile, in a very significant place in a source file by chance.

I'm not necessarily saying this was a "cosmic ray", but the symptom matches.

4

You might want to have a look at Fault Tolerant hardware as well.

For example Stratus Technology builds Wintel servers called ftServer which had 2 or 3 "mainboards" in lock-step, comparing the result of the computations. (this is also done in space vehicles sometimes).

The Stratus servers evolved from custom chipset to lockstep on the backplane.

A very similar (but software) system is the VMWare Fault Tolerance lockstep based on the Hypervisor.

Not the answer you're looking for? Browse other questions tagged or ask your own question.