For reference, the Scenarios described below were executed using the following:
- Ubuntu 18.04.4 LTS (Bionic Beaver)
- top "procps-ng 3.3.12"
- Lua 5.3.3 Copyright (C) 1994-2016 Lua.org, PUC-Rio
Each "Scenario" below demonstrates results of the following steps:
- Capture "lua5.3" initial process resource usage from "top"
- In "lua5.3", allocate ~16GB from the heap
- Capture "lua5.3" new process resource usage from "top" to show that 16GB was allocated
- In "lua5.3", mark root object for deletion and then call "collectgarbage()"
- Capture "lua5.3" final process resource usage from "top" to show effect of "collectgarbage()"
For each scenario, compare the initial process resource usage, "allocated" process resource usage, and the final process resource usage after "collectgarbage(). Scenario 1 shows that the "allocated" 16GB is NOT freed after garbage collection. Scenario 2 and 3 show that some of the "allocated" 16GB is freed after garbage collection. Scenario 4 shows that all of the "allocated" 16GB is freed after garbage collection. Scenarios 1 through 4 differ only in the number of objects (i.e. tables) used during allocation, where Scenario 1 uses the most tables (128*1024), and Scenario 4 uses the least (1). Hence, it appears that the Lua 5.3.3 garbage collection has a limitation on the number of objects it can truly manage, or there is a defect in Lua's memory management.
In short, Lua's garbage collection is not showing deterministic behavior with memory management. If a large number of tables are created, the memory usage associated with those tables may never be returned to the heap as managed by the operating system, when the memory is no longer needed in Lua. However, if a small number of tables are created, the memory usage associated with those tables seem to ALWAYS be returned to the heap as managed by the operating system, when the memory is no longer needed in Lua.
Why is the memory management behavior in Lua not consistent/deterministic?
NOTE: For each scenario, calling collectgarbage("count") after collectgarbage() shows that lua5.3 has properly cleaned up all garbage, whereas using "top" shows that is not always the case. Hence, "top" is used here to show the true behavior of Lua's garbage collection.
Scenario 1: Lua collectgarbage() does not free memory used
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22866 user 20 0 18152 3140 2840 S 0.0 0.0 0:00.00 lua5.3
> collectgarbage("count")
22.8759765625
> a = {} for i=0,256*1024 do a[i] = {} for j=0,4*1024*1024 do a[i][j] = i*j end end
> collectgarbage("count")
16803927.791016
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22866 user 20 0 16.053g 0.016t 2868 S 0.0 51.3 0:38.97 lua5.3
> a = nil collectgarbage()
> collectgarbage("count")
25.29296875
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22866 user 20 0 16.049g 0.016t 2868 S 0.0 51.3 0:39.08 lua5.3
Scenario 2: Lua collectgarbage() frees half of memory used
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22958 user 20 0 18152 3200 2916 S 0.0 0.0 0:00.00 lua5.3
> collectgarbage("count")
22.8759765625
> a = {} for i=0,128*1024 do a[i] = {} for j=0,8*1024*1024 do a[i][j] = i*j end end
> collectgarbage("count")
16790679.791016
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22958 user 20 0 16.284g 0.016t 2944 S 0.0 52.0 0:39.79 lua5.3
> a = nil collectgarbage()
> collectgarbage("count")
23.1826171875
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22958 user 20 0 8422324 8.018g 2944 S 0.0 25.6 0:40.50 lua5.3
Scenario 3: Lua collectgarbage() frees almost all memory used
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23127 user 20 0 18152 3192 2904 S 0.0 0.0 0:00.00 lua5.3
> collectgarbage("count")
22.8759765625
> a = {} for i=0,64*1024 do a[i] = {} for j=0,16*1024*1024 do a[i][j] = i*j end end
> collectgarbage("count")
16784151.791016
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23127 user 20 0 16.275g 0.016t 2932 S 0.0 52.0 0:41.22 lua5.3
> a = nil collectgarbage()
> collectgarbage("count")
23.1826171875
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23127 user 20 0 25900 10992 2932 S 0.0 0.0 0:41.81 lua5.3
Scenario 4: Lua collectgarbage() frees all memory used
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23771 user 20 0 18152 3204 2920 S 0.0 0.0 0:00.00 lua5.3
> collectgarbage("count")
22.8759765625
> a = {} for i=0,0 do a[i] = {} for j=0,1024*1024*1024 do a[i][j] = i*j end end
> collectgarbage("count")
16777241.969727
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23771 user 20 0 16.017g 0.016t 2948 S 0.0 51.2 0:38.97 lua5.3
> a = nil collectgarbage()
> collectgarbage("count")
23.17578125
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23771 user 20 0 18152 3364 2948 S 0.0 0.0 0:40.80 lua5.3
Update
After additional testing, it does appear that the number of objects (i.e. tables) managed in Lua affects the behavior of Lua's garbage collection. Specifically, if the number of tables being managed by Lua is less than 256 (this seems to be the threshold), the memory will be returned to the operating system's heap after garbage collection. As the number of objects grows above 256 (inclusive), Lua's memory management begins to exhibit different behavior as it holds on to more memory for internal reuse after its garbage collection. Scenario 1 above shows the most dramatic affect of Lua holding onto memory after garbage collection, and Scenario 4 shows the least affect.
Note: The Example results are not deterministic; that is, other execution in Lua may affect the behavior of Lua's garbage collection. For the most deterministic results, launch a new Lua process for each test.
For illustration, the following examples continuously allocate, populate, and deallocate a table of tables, printing out the following from the Lua process: iteration, preMemoryUsage (after allocation), preMemoryUsagePercent, postMemoryUsage (after deallocation), postMemoryUsagePercent.
Example Program: Change numberOfTables as Desired
numberOfTables = 255 -- Example 1: 255; Example 2: 256
function getMemoryPercent()
handle = io.popen("v=`ps -p $PPID -o pmem= | tr -d '\n' | sed 's/ *//g'`; echo -n $v")
memoryPercent = handle:read("*a")
handle:close()
return memoryPercent
end
socket = require("socket")
print("#", "preMemUsage", "%", "postMemUsage", "%")
iteration = 0
while true
do
a = {}
for i = 1,numberOfTables
do
a[i] = {}
for j = 1,math.floor(256*1024*1024/numberOfTables)
do
a[i][j] = i*j
end
end
preMemoryUsage = collectgarbage("count")
preMemoryUsagePercent = getMemoryPercent()
a = nil
collectgarbage()
postMemoryUsage = collectgarbage("count")
postMemoryUsagePercent = getMemoryPercent()
iteration = iteration + 1
print(iteration, preMemoryUsage, preMemoryUsagePercent, postMemoryUsage, postMemoryUsagePercent)
end
Example 1: Results for numberOfTables=255
# preMemUsage % postMemUsage %
1 8355905.0869141 25.4 47.240234375 0.0
2 8355905.1972656 25.4 47.322265625 0.0
3 8355905.1972656 25.4 47.322265625 0.0
4 8355905.1972656 25.4 47.322265625 0.0
5 8355905.1972656 25.4 47.322265625 0.0
6 8355905.1972656 25.4 47.322265625 0.0
7 8355905.1972656 25.4 47.322265625 0.0
8 8355905.1972656 25.4 47.322265625 0.0
9 8355905.1972656 25.4 47.322265625 0.0
10 8355905.1972656 25.4 47.322265625 0.0
Example 2: Results for numberOfTables=256
# preMemUsage % postMemUsage %
1 4194369.0205078 12.8 47.119140625 0.0
2 4194369.1308594 12.8 47.201171875 12.8
3 4194369.1035156 12.8 47.173828125 12.5
4 4194369.1318359 12.8 47.2021484375 12.4
5 4194369.1318359 12.8 47.2021484375 12.4
6 4194369.1318359 12.8 47.2021484375 12.4
7 4194369.1318359 12.8 47.2021484375 12.4
8 4194369.1318359 12.8 47.2021484375 12.4
9 4194369.1318359 12.8 47.2021484375 12.4
10 4194369.1318359 12.8 47.2021484375 12.4
collectgarbage()? Can you repeat the tests usingcollectgarbage("count")to get the memory usage instead of executingtop?