7

I have a Linux instance (Amazon Linux Linux ip-xxx 4.9.20-11.31.amzn1.x86_64 #1) which runs Jenkins. It occasionally stops working because of the lack of the memory needed for a job.

Based on my investigation with free command and /proc/meminfo, it seems that Slab is taking up most of the memory available on the instance.

[root@ip-xxx ~]# free -tm
             total       used       free     shared    buffers     cached
Mem:          7985       7205        779          0         19        310
-/+ buffers/cache:       6876       1108
Swap:            0          0          0
Total:        7985       7205        779

[root@ip-xxx ~]# cat /proc/meminfo | grep "Slab\|claim"
Slab:            6719244 kB
SReclaimable:      34288 kB
SUnreclaim:      6684956 kB

I could find the way to purge dentry cache by running echo 3 > /proc/sys/vm/drop_caches, but how can I purge kmalloc-256? Or, is there a way to find which process is using kmalloc-256 memory space?

[root@ip-xxx ~]# slabtop -o | head -n 15
 Active / Total Objects (% used)    : 26805556 / 26816810 (100.0%)
 Active / Total Slabs (% used)      : 837451 / 837451 (100.0%)
 Active / Total Caches (% used)     : 85 / 111 (76.6%)
 Active / Total Size (% used)       : 6696903.08K / 6701323.05K (99.9%)
 Minimum / Average / Maximum Object : 0.01K / 0.25K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
26658528 26658288  99%    0.25K 833079       32   6664632K kmalloc-256            
 21624  21009  97%    0.12K    636       34      2544K kernfs_node_cache      
 20055  20055 100%    0.19K    955       21      3820K dentry                 
 10854  10646  98%    0.58K    402       27      6432K inode_cache            
 10624   9745  91%    0.03K     83      128       332K kmalloc-32             
  7395   7395 100%    0.05K     87       85       348K ftrace_event_field     
  6912   6384  92%    0.02K     27      256       108K kmalloc-16             
  6321   5581  88%    0.19K    301       21      1204K cred_jar  

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.