11

My current configs are:

> cat /proc/sys/vm/panic_on_oom
0
> cat /proc/sys/vm/oom_kill_allocating_task
0
> cat /proc/sys/vm/overcommit_memory
1

but when I run a task, it's killed anyway.

> ./test/mem.sh
Killed
> dmesg | tail -2
[24281.788131] Memory cgroup out of memory: Kill process 10565 (bash) score 1001 or sacrifice child
[24281.788133] Killed process 10565 (bash) total-vm:12601088kB, anon-rss:5242544kB, file-rss:64kB

Update

My tasks are used to scientific computing, which costs many memories, it seems that overcommit_memory=1 may be the best choice.

Update 2

Actually, I'm working on a data analyzation project, which costs memory more than 16G, but I was asked to limit them in about 5G. It might be impossible to implement this requirement via optimizing the program itself, because the project uses many sub-commands, and most of them does not contains options like Xms or Xmx in Java.

Update 3

My project should be an overcommited system. Exacetly as what a3f saying, it seems that my apps prefer to crash by xmalloc when mem allocated failed.

> cat /proc/sys/vm/overcommit_memory
2
> ./test/mem.sh
./test/mem.sh: xmalloc: .././subst.c:3542: cannot allocate 1073741825 bytes (4295237632 bytes allocated)

I don't want to surrender, although so many aweful tests make me exhausted. So please show me a way to the light ; )

2
  • I just found this: "How to Configure the Linux Out-of-Memory Killer" oracle.com/technical-resources/articles/it-infrastructure/… I'm glad the accepted answer links to documentation, but a full tutorial/guide would be more helpful I think.
    – PJ Brunet
    Jul 11, 2022 at 16:34
  • Since you tag docker, the best way to limit memory resources is using docker/compose/k8s. Just check the doc, depending on the docker orchestration mechanism you are using, e.g. in docker-compose it's mem_limit.
    – RodolfoAP
    Mar 7 at 9:46

2 Answers 2

14

The OOM killer won't go away. If there is no memory, someone's got to pay. What you can do is set a limit after which memory allocations fail. That's exactly what setting vm.overcommit_memory to 2 achieves.

From the docs:

The Linux kernel supports the following overcommit handling modes

2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

Normally, the kernel will happily hand out virtual memory (overcommit). Only when you reference a page, the kernel has to map the page to a real physical frame. If it can't service that request, a process needs to be killed by the OOM killer to make space.

Disabling overcommit means that e.g. malloc(3) will return NULL if the kernel couldn't commit the amount of memory requested. This makes things a bit more predictable, albeit limited (many applications allocate more than they would ever need).

9
  • Thanks! My tasks always cost many memories. If set the overcommit_memory=2, will the task be paused? It's not good for scientific computing tasks, maybe.
    – Yang
    Mar 4, 2016 at 9:04
  • @Yang I updated the answer. If you want to avoid OOM situations, you need to buy more RAM (or fix your memory management strategy or keep using overcommit and hope for the best).
    – a3f
    Mar 4, 2016 at 9:23
  • 1
    @Yang Check out the link I posted. You can set an upper limit after which allocations fail. Your application would then have to deal with the memory allocation failure. Many applications just crash (either implicitly by null pointer dereference or explicitly by e.g. xmalloc). I don't know how your application handles it.
    – a3f
    Mar 4, 2016 at 13:50
  • 1
    @Yang Either rewrite the application to consume less or increase the physical memory. The second option is probably cheaper.
    – a3f
    Mar 4, 2016 at 15:27
  • 1
    @workless This depends on the overcommit strategy, check the linked doc for more information. And even with the default overcommit behavior,malloc(-1) will give you NULL.
    – a3f
    May 9, 2016 at 17:04
3

The possible values of oom_adj range from -17 to +15. The higher the score, more likely the associated process is to be killed by OOM-killer. If oom_adj is set to -17, the process is not considered for OOM-killing.

But, increase ram is better choice ,if increasing ram is not possible, then add swap memory.

To increase swap memory try this link,

4
  • 2
    Please do not post an answer that consists essentially of a link. Include the important points in your answer; leave the link for extra information or as a reference.
    – glennsl
    Sep 25, 2017 at 11:32
  • @glennsl Thanks. Sep 25, 2017 at 11:42
  • @glennsl, thanks for information and link you provided. I updated my post. Sep 25, 2017 at 11:59
  • @RavipatiPraveen, Thank you! Disabling the OOM-killing is really what I want, by set the oom_adj to -17 for the task. Increasing swap sounds really cool, and it might solve the problems. I'll give it a try later, Thank you!
    – Yang
    Sep 26, 2017 at 0:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.