I wrote a small program to collect data over serial port using MinimalModbus. The data is dumped into a CSV file. I have read several posts on SO and other places. A few things mentioned are:
- Using lazy evaluation wherever possible (xrange instead of range)
- Deleting large unused objects
- Use child processes and upon their death memory is released by OS
The script is on github here. I also use a script to periodically upload these files to a server. Both these scripts are fairly trivial. Also nothing else is running on the system, thus i feel that withing these two systems only memory hogging is taking place. What would be the best way to tackle this issue. I am not the most willing to adopt the subprocess route.
Some more information:
- Data collection is on Raspberry Pi (512 MB RAM)
- Python version: 2.7
- It takes about 3-4 days for RAM to be completely used after which the RaspberryPi freezes
I followed this guide to find out top 20 programs which are eating up RAM.
$ ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
12434 2.2 python
12338 1.2 python
2578 0.8 /usr/sbin/console-kit-daemon
30259 0.7 sshd:
30283 0.7 -bash
1772 0.6 /usr/sbin/rsyslogd
2645 0.6 /usr/lib/policykit-1/polkitd
2146 0.5 dhclient
1911 0.4 /usr/sbin/ntpd
12337 0.3 sudo
12433 0.3 sudo
1981 0.3 sudo
30280 0.3 sshd:
154 0.2 udevd
16994 0.2 /usr/sbin/sshd
17006 0.2 ps
1875 0.2 /usr/bin/dbus-daemon
278 0.2 udevd
290 0.2 udevd
1 0.1 init
So the two Python processes are eating up some RAM, but that is very small when compared to overall RAM consumed. The following is the output of the free command.
pi@raspberrypi ~ $ free -m
total used free shared buffers cached
Mem: 438 414 23 0 45 320
-/+ buffers/cache: 48 389
Swap: 99 0 99
The following is the output of the top command.
Tasks: 69 total, 1 running, 68 sleeping, 0 stopped, 0 zombie
%Cpu(s): 66.9 us, 5.0 sy, 0.0 ni, 18.1 id, 0.0 wa, 0.0 hi, 10.0 si, 0.0 st
KiB Mem: 448776 total, 429160 used, 19616 free, 47016 buffers
KiB Swap: 102396 total, 0 used, 102396 free, 332288 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12338 root 20 0 10156 5644 2384 S 69.8 1.3 3059:31 python
26039 root 20 0 0 0 0 S 1.6 0.0 0:02.71 kworker/0:1
26863 pi 20 0 4664 1356 1028 R 1.3 0.3 0:00.12 top
1982 root 20 0 1752 516 448 S 0.3 0.1 1:08.36 sh
1985 root 20 0 1692 552 460 S 0.3 0.1 5:15.16 startpar
1 root 20 0 2144 728 620 S 0.0 0.2 0:17.43 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.14 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:13.20 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/u:0H
8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
12 root 20 0 0 0 0 S 0.0 0.0 0:00.06 bdi-default
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
EDIT 2
As suggested in the first answer, i decided to look into log files. I had a look at syslog and the following is the result of tail on it.
May 19 10:03:26 raspberrypi wpa_supplicant[7065]: wlan0: Failed to initialize driver interface
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: nl80211: 'nl80211' generic netlink not found
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: Failed to initialize driver 'nl80211'
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: rfkill: Cannot open RFKILL control device
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: Could not read interface wlan0 flags: No such device
These messages are filling up the log files and are coming every second. The interesting part is that i am using Ethernet and not WiFi.
Thus, now it is unclear where the RAM has gone?