I have a framework composed of different tools written in python in a multi-user environment.
The first time I log to the system and start one command it takes 6 seconds just to show a few line of help. If I immediately issue the same command again it takes 0.1s. After a couple of minutes it gets back to 6s. (proof of short-lived cache)
The system sits on a GPFS so disk throughput should be ok, though access might be low because of the amount of files in the system.
strace -e open python tool | wc -l
shows 2154 files being accessed when starting the tool.
strace -e open python tool | grep ENOENT | wc -l
shows 1945 missing files being looked for. (A very bad hit/miss ratio is you ask me :-)
I have a hunch that the excessive time involved in loading the tool is consumed by querying the GPFS about all those files, and these are cached for the next call (at either system or GPFS level), though I don't know how to test/prove it. I have no root access to the system and I can only write to GPFS and /tmp.
Is it possible to improve this python quest for missing files
?
Any idea on how to test this in a simple way? (Reinstalling everything on /tmp is not simple, as there are many packages involved, virtualenv will not help either (I think), since it's just linking the files on the gpfs system).
An option would be of course to have a daemon that forks, but that's far from "simple" and would be a last resort solution.
Thanks for reading.