1

I've got a very strange problem on our servers: the total used memory is way more than the sum of the used memory of all processes. And 'way more' is 28 gigabytes!

The server has 128Gb memory. It works as a KVM virtualization host. There are several qemu processes with the sum of memory allocations (-m switch) of 116784Mb (116Gb) and almost no other processes.

But at the same time there are 28Gb of used swap space, with rather busy swap in/out operations.

I ran calculation for all processes memory and it was less than 70Gb:

ps aux|awk '{a+=$6}END{print a}'
66432908

VSS is about 180Gb:

ps aux|awk '{a+=$5}END{print a}'
180574360

Here /proc/meminfo:

MemTotal:       131988428 kB
MemFree:         1002468 kB
Buffers:           14172 kB
Cached:           396696 kB
SwapCached:      2566436 kB
Active:         14984848 kB
Inactive:        5634948 kB
Active(anon):   14944552 kB
Inactive(anon):  5592652 kB
Active(file):      40296 kB
Inactive(file):    42296 kB
Unevictable:      428476 kB
Mlocked:          151800 kB
SwapTotal:      41947128 kB
SwapFree:       12291740 kB
Dirty:                16 kB
AnonPages:      18074276 kB
Mapped:            19608 kB
Shmem:             46300 kB
Slab:            4617044 kB
SReclaimable:    3296416 kB
SUnreclaim:      1320628 kB
KernelStack:        6424 kB
PageTables:       209028 kB
CommitLimit:    107941340 kB
Committed_AS:   124320872 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      739544 kB
VmallocChunk:   34358825808 kB
AnonHugePages:   1236992 kB
Hugepagesize:       2048 kB

The system is running on augfs/tmpfs, but tmpfs uses about 650Mb...

And I'm lost: What are those 28GB of memory used for?

This problem happens on all our servers to a certain degree....

7
  • Might get a better answer over at serverfault.com
    – Adam
    Jul 28, 2015 at 14:11
  • I'm afraid this is beyond 'system administrator' level and it needs some digging in the linux internals... Jul 28, 2015 at 14:13
  • This answer might be of help: superuser.com/questions/358049/where-has-all-my-memory-gone/… Jul 28, 2015 at 19:50
  • "needs some digging in the linux internals" — when trying to determine whether the system is broken or your understanding is, it is never the kernel.
    – msw
    Jul 28, 2015 at 22:25
  • Dan Cornilescu, thanks, but it is not that easy. In my case the cache and buffers are almost empty (see /proc/meminfo - all together they are less than 500MB, and 'the problem' is about 28000Mb....) Jul 29, 2015 at 0:15

1 Answer 1

1

Note that your simple addition using the "VSS" is counting alot of memory pages multiple times. When shared libraries are loaded into a process, the memory for those libraries is not duplicated across every process - if 10 processes load the same library only 1 copy of the library is physically in RAM. The VSS figure you're summing though, reports the memory for each process as if it had its own private copy of the library. So if 10 processes have the same library loaded, you are counting the memory for that library 10 times over.

Essentially, it is impossible to sum up total memory usage by counting the 'VSS' figure reported by 'ps'. To get a remotely accurate figure for usage, you need to take into account how much memory used by a process is shared vs private - this data is available in /proc/$PID/smaps, but interpreting it correctly is pretty difficult to do.

WRT to KVM in particular, besides guest RAM itself, QEMU uses memory for quite a few other things. BIOS ROMS, video RAM are two big ones, but also lots of transient allocations for I/O operations on disk/network. If the I/O load of the guest is high, this can add up to some significant memory usage beyond the -m RAM figure. It is hard to give a good answer for exactly what kind of memory overhead there is per guest beyond RAM, so you kind of need to monitor it. If you're seeing VMs pushed out into swap, its simply a sign that you've tried to put too many VMs on the host I'm afraid.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.