Is there a way for a Python program to determine how much memory it's currently using? I've seen discussions about memory usage for a single object, but what I need is total memory usage for the process, so that I can determine when it's necessary to start discarding cached data.
12 Answers
Here is a useful solution that works for various operating systems, including Linux, Windows, etc.:
import os, psutil
process = psutil.Process()
print(process.memory_info().rss) # in bytes
Notes:
do
pip install psutil
if it is not installed yethandy one-liner if you quickly want to know how many MiB your process takes:
import os, psutil; print(psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2)
with Python 2.7 and psutil 5.6.3, it was
process.memory_info()[0]
instead (there was a change in the API later).
-
7Why doesn't this number match the one in the process explorer? The number from psutil always seems to be larger by about 10%. May 13, 2015 at 22:42
-
67
-
36For recent versions of
psutil
,psutil.Process()
is equivalent topsutil.Process(os.getpid())
. That's one less thing you need to remember to type.– rnorrisApr 26, 2019 at 3:07 -
9Why do you use rss? In psutil document: rss | resident set size, vms | total program size. So I think last line should be
print(process.memory_info().vms)
Oct 25, 2019 at 13:16 -
4
For Unix based systems (Linux, Mac OS X, Solaris), you can use the getrusage()
function from the standard library module resource
. The resulting object has the attribute ru_maxrss
, which gives the peak memory usage for the calling process:
>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
2656 # peak memory usage (kilobytes on Linux, bytes on OS X)
The Python docs don't make note of the units. Refer to your specific system's man getrusage.2
page to check the unit for the value. On Ubuntu 18.04, the unit is noted as kilobytes. On Mac OS X, it's bytes.
The getrusage()
function can also be given resource.RUSAGE_CHILDREN
to get the usage for child processes, and (on some systems) resource.RUSAGE_BOTH
for total (self and child) process usage.
If you care only about Linux, you can alternatively read the /proc/self/status
or /proc/self/statm
file as described in other answers for this question and this one too.
-
2Okay, will do. I wasn't sure if SO had a process for merging questions or what. The duplicate post was partly to show people there was a standard library solution on both questions... and partly for the rep. ;) Should I delete this answer? Oct 6, 2011 at 3:19
-
8Mac OS definitely returns the RSS in bytes, Linux returns it in kilobytes.– NeilDec 6, 2013 at 23:33
-
20The units are NOT in kilobytes. It is platform dependent, so you have to use resource.getpagesize() to find out. The given Python docs (docs.python.org/2/library/resource.html#resource-usage) is actually very clear about it. It is 4096 in my box.– Ben LinApr 15, 2014 at 16:53
-
6@BenLin Those Python docs are clearly wrong, or there is a bug on the Mac version. The unit used by getrusage and the value returned by getpagesize are definitely different.– AndrewJul 8, 2015 at 17:56
-
18The question asked for current usage. Note that this is maximum usage. (Still a useful answer, just warning people who mistakenly copy-paste it.)– LucJan 23, 2018 at 8:29
On Windows, you can use WMI (home page, cheeseshop):
def memory():
import os
from wmi import WMI
w = WMI('.')
result = w.query("SELECT WorkingSet FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=%d" % os.getpid())
return int(result[0].WorkingSet)
On Linux (from python cookbook http://code.activestate.com/recipes/286222/:
import os
_proc_status = '/proc/%d/status' % os.getpid()
_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0, 'KB': 1024.0, 'MB': 1024.0*1024.0}
def _VmB(VmKey):
'''Private.'''
global _proc_status, _scale
# get pseudo file /proc/<pid>/status
try:
t = open(_proc_status)
v = t.read()
t.close()
except:
return 0.0 # non-Linux?
# get VmKey line e.g. 'VmRSS: 9999 kB\n ...'
i = v.index(VmKey)
v = v[i:].split(None, 3) # whitespace
if len(v) < 3:
return 0.0 # invalid format?
# convert Vm value to bytes
return float(v[1]) * _scale[v[2]]
def memory(since=0.0):
'''Return memory usage in bytes.'''
return _VmB('VmSize:') - since
def resident(since=0.0):
'''Return resident memory usage in bytes.'''
return _VmB('VmRSS:') - since
def stacksize(since=0.0):
'''Return stack size in bytes.'''
return _VmB('VmStk:') - since
-
14The Windows code doesn't work for me. This change does:
return int(result[0].WorkingSet)
Aug 31, 2010 at 0:46 -
1This Windows code doesn't work for me on Windows 7 x64, even after John Fouhy's comment modification.– BasjFeb 7, 2014 at 15:59
-
I have this error: return [ wmi_object (obj, instance_of, fields) for obj in self._raw_query(wql) ] File "C:\Python27\lib\site-packages\win32com\client\util.py", line 84, in next return _get_good_object_(self._iter.next(), resultCLSID = self.resultCLSID) pywintypes.com_error: (-2147217385, 'OLE error 0x80041017', None, None) if anyone can help me? I have win 8 x64 but python on x32 Sep 9, 2014 at 6:06
-
-
1@jedwards: WorkingSet is a very bad number in >90% of the cases. It's just that part of memory which is currently in RAM Jul 16, 2020 at 20:26
On unix, you can use the ps
tool to monitor it:
$ ps u -p 1347 | awk '{sum=sum+$6}; END {print sum/1024}'
where 1347 is some process id. Also, the result is in MB.
Current memory usage of the current process on Linux, for Python 2, Python 3, and pypy, without any imports:
def getCurrentMemoryUsage():
''' Memory usage in kB '''
with open('/proc/self/status') as f:
memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]
return int(memusage.strip())
It reads the status file of the current process, takes everything after VmRSS:
, then takes everything before the first newline (isolating the value of VmRSS), and finally cuts off the last 3 bytes which are a space and the unit (kB).
To return, it strips any whitespace and returns it as a number.
Tested on Linux 4.4 and 4.9, but even an early Linux version should work: looking in man proc
and searching for the info on the /proc/$PID/status
file, it mentions minimum versions for some fields (like Linux 2.6.10 for "VmPTE"), but the "VmRSS" field (which I use here) has no such mention. Therefore I assume it has been in there since an early version.
Below is my function decorator which allows to track how much memory this process consumed before the function call, how much memory it uses after the function call, and how long the function is executed.
import time
import os
import psutil
def elapsed_since(start):
return time.strftime("%H:%M:%S", time.gmtime(time.time() - start))
def get_process_memory():
process = psutil.Process(os.getpid())
return process.memory_info().rss
def track(func):
def wrapper(*args, **kwargs):
mem_before = get_process_memory()
start = time.time()
result = func(*args, **kwargs)
elapsed_time = elapsed_since(start)
mem_after = get_process_memory()
print("{}: memory before: {:,}, after: {:,}, consumed: {:,}; exec time: {}".format(
func.__name__,
mem_before, mem_after, mem_after - mem_before,
elapsed_time))
return result
return wrapper
So, when you have some function decorated with it
from utils import track
@track
def list_create(n):
print("inside list create")
return [1] * n
You will be able to see this output:
inside list create
list_create: memory before: 45,928,448, after: 46,211,072, consumed: 282,624; exec time: 00:00:00
For Python 3.6 and psutil 5.4.5 it is easier to use memory_percent()
function listed here.
import os
import psutil
process = psutil.Process(os.getpid())
print(process.memory_percent())
-
1
Even easier to use than /proc/self/status
: /proc/self/statm
. It's just a space delimited list of several statistics. I haven't been able to tell if both files are always present.
/proc/[pid]/statm
Provides information about memory usage, measured in pages. The columns are:
- size (1) total program size (same as VmSize in /proc/[pid]/status)
- resident (2) resident set size (same as VmRSS in /proc/[pid]/status)
- shared (3) number of resident shared pages (i.e., backed by a file) (same as RssFile+RssShmem in /proc/[pid]/status)
- text (4) text (code)
- lib (5) library (unused since Linux 2.6; always 0)
- data (6) data + stack
- dt (7) dirty pages (unused since Linux 2.6; always 0)
Here's a simple example:
from pathlib import Path
from resource import getpagesize
PAGESIZE = getpagesize()
PATH = Path('/proc/self/statm')
def get_resident_set_size() -> int:
"""Return the current resident set size in bytes."""
# statm columns are: size resident shared text lib data dt
statm = PATH.read_text()
fields = statm.split()
return int(fields[1]) * PAGESIZE
data = []
start_memory = get_resident_set_size()
for _ in range(10):
data.append('X' * 100000)
print(get_resident_set_size() - start_memory)
That produces a list that looks something like this:
0
0
368640
368640
368640
638976
638976
909312
909312
909312
You can see that it jumps by about 300,000 bytes after roughly 3 allocations of 100,000 bytes.
-
For those in need of higher accuracy, documentation suggests using
/proc/[pid]/smaps
, but that's a bit more complicated.– djvgSep 4 at 14:38
I like it, thank you for @bayer. I get a specific process count tool, now.
# Megabyte.
$ ps aux | grep python | awk '{sum=sum+$6}; END {print sum/1024 " MB"}'
87.9492 MB
# Byte.
$ ps aux | grep python | awk '{sum=sum+$6}; END {print sum " KB"}'
90064 KB
Attach my process list.
$ ps aux | grep python
root 943 0.0 0.1 53252 9524 ? Ss Aug19 52:01 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
root 950 0.6 0.4 299680 34220 ? Sl Aug19 568:52 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
root 3803 0.2 0.4 315692 36576 ? S 12:43 0:54 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
jonny 23325 0.0 0.1 47460 9076 pts/0 S+ 17:40 0:00 python
jonny 24651 0.0 0.0 13076 924 pts/4 S+ 18:06 0:00 grep python
Reference
-
just an optimisation of code to avoid multi pipe
ps aux | awk '/python/{sum+=$6}; END {print sum/1024 " MB"}'
Oct 4, 2017 at 5:06
import os, win32api, win32con, win32process
han = win32api.OpenProcess(win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ, 0, os.getpid())
process_memory = int(win32process.GetProcessMemoryInfo(han)['WorkingSetSize'])
-
9This could be improved with some explanation of what it does and how it works. Nov 25, 2014 at 15:39
-
3Based on the large number returned (8 digits) and how I'm not doing much of anything, I'm guessing this has to be bytes? So it's around 28.5 MB for a rather idle interactive instance. (Wow... I didn't even realize the above comment was mine from 4 years ago... that's weird.) Jun 8, 2018 at 18:50
For Unix systems command time
(/usr/bin/time) gives you that info if you pass -v. See Maximum resident set size
below, which is the maximum (peak) real (not virtual) memory that was used during program execution:
$ /usr/bin/time -v ls /
Command being timed: "ls /"
User time (seconds): 0.00
System time (seconds): 0.01
Percent of CPU this job got: 250%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 315
Voluntary context switches: 2
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
-
2Note that this may fail if you just try to use
time
instead of/usr/bin/time
. See: askubuntu.com/questions/434289/…– aboughtApr 9, 2020 at 19:49
Using sh and os to get into python bayer's answer.
float(sh.awk(sh.ps('u','-p',os.getpid()),'{sum=sum+$6}; END {print sum/1024}'))
Answer is in megabytes.
-
5Should be noted that `sh' isn't a stdlib module. It's installable with pip, though. Sep 4, 2013 at 0:00