118

What tools or techniques can I use to remove cached file contents to prevent my performance results from being skewed? I believe I need to either completely clear, or selectively remove cached information about file and directory contents.

The application that I'm developing is a specialised compression utility, and is expected to do a lot of work reading and writing files that the operating system hasn't touched recently, and whose disk blocks are unlikely to be cached.

I wish to remove the variability I see in IO time when I repeat the task of profiling different strategies for doing the file processing work.

I'm primarily interested in solutions for Windows XP, as that is my main development machine, but I can also test using linux, and so am interested in answers for that environment too.

I tried SysInternals CacheSet, but clicking "Clear" doesn't result in a measurable increase (restoration to timing after a cold-boot) in the time to re-read files I've just read a few times.

2

9 Answers 9

95

Use SysInternal's RAMMap app.

rammap empty standby

The Empty / Empty Standby List menu option will clear the Windows file cache.

7
  • 2
    RAMMap doesn't run on Windows XP. Do those menu options do something different from the "Clear" button in SysInternals Cacheset application? Mar 12, 2012 at 21:32
  • @stephen this technique worked perfectly for me under Windows 8! Windows XP is 12 years old now, any reason you can't try this on Vista, Windows 7, or Windows 8? Nov 13, 2012 at 5:06
  • @Jeff I no longer work with those limitations. So "Empty Standby List" results in the next read from a recently read file requiring disk IO? Nov 13, 2012 at 22:43
  • 2
    Press F5 to refresh the file list after clicking empty standby list
    – JoshG
    Aug 20, 2013 at 0:31
  • 3
    For a programmatic approach, see this SO post: stackoverflow.com/a/23085045/430360
    – snemarch
    Nov 11, 2015 at 8:35
17
+50

For Windows XP, you should be able to clear the cache for a specific file by opening the file using CreateFile with the FILE_FLAG_NO_BUFFERING options and then closing the handle. This isn't documented, and I don't know if it works on later versions of Windows, but I used this long ago when writing test code to compare file compression libraries. I don't recall if read or write access affected this trick.

6
  • Works perfectly for me under Windows 7 SP1, x64. Great tip!
    – cxxl
    Oct 22, 2012 at 16:26
  • 2
    So that must be repeated for every file? So if for example, you copy a directory containing 100MB spread across 30 files in 10 subdirectories, you must open each and every one separately to make sure you read the actual disk instead of the cache?
    – Synetech
    Dec 1, 2013 at 0:51
  • Other answers on the site affirm this works in Win7 and 8 as well. I presume it works on vista. And yes, you have to run it on each file, but it doesn't take all that long. All you need to do is open and close each file, and when you close, Windows clears the cache. After you've done this, THEN you run your performance test. Jan 4, 2014 at 0:36
  • 1
    This was a perfect answer. Simple to code, effective, and should work on most windows OSes (including XP, I've only tested on Win7x64). I opened the files with read privileges and no sharing. Not sure what combo matters.
    – Rosco
    Aug 6, 2014 at 17:06
  • Similar functionality on Linux (2.4.10 onward) with the O_DIRECT flag in the open(2) syscall. Well, more precisely, from what I understand, O_DIRECT does not clear the cache for that file, it tries it's best to bypass it.
    – kaiwan
    Jul 15, 2015 at 12:06
15

A command line utility can be found here

from source:

EmptyStandbyList.exe is a command line tool for Windows (Vista and above) that can empty:

  • process working sets,
  • the modified page list,
  • the standby lists (priorities 0 to 7), or
  • the priority 0 standby list only.

Usage:

EmptyStandbyList.exe workingsets|modifiedpagelist|standbylist|priority0standbylist
4
  • 4
    Highly underrated answer amidst all the noise, simple utility that does one job and does it well. You should probably include more details in the answer to get the upvotes you deserve. Sep 30, 2018 at 22:50
  • 1
    @HashimAziz, I concur! It is worth a short explanation of what those four things are that it can empty. I for example just want to get rid of the disk read cache. I'm testing file load times and get huge performance increases if I recently loaded that same file which we have concluded is probably because of a disk read cache in Windows (and how I land here). This utility looks grand but now I have to go research what these 4 stanby lists are and which one I want to clear. Aug 2, 2021 at 1:22
  • 1
    I'm using this to great effect in my tests now. Thanks. It's not well documented. I infer empirically that without arguments it just empties all the four mentioned things or at least the 8 standby lists. Either way, I run it with not arguments between test runs and I have seen a cycle of slowload, fastload, fastload change to slowload, slowload, slowload. Much as I was expecting and am happy. I was witnessing enormous file load speed ups on subsequent iterations of this test without explanation, well until we found this was the likely cause, and tested this utility and can confirm it works! Aug 3, 2021 at 3:18
  • 1
    Further I run it form Python test scripts. Which throws up a console while it runs. Would love to find a native Python solution, but haven't found any bindings yet. Aug 3, 2021 at 3:19
14

A quick googling gives these options for Linux

  1. Unmount and mount the partition holding the files
  2. sync && echo 1 > /proc/sys/vm/drop_caches
3
  • 2
    Thanks, that looks very useful, though I'd possibly want to echo 3 instead of 1. I'm primarily interested in Windows XP, which is why I hadn't found that in my googling. Jan 25, 2009 at 22:03
  • Unfortunately the linux environment I could make use of this in, has kernel version 2.6.9. drop_caches was added in kernel 2.6.16 Jan 25, 2009 at 22:45
  • Lots of other posts I have read suggest that you can unmount and mount the filesystem to remove all the chached items for it, I would assume that was the case before 2.6.16 as well as in more recent kernels.
    – TafT
    Aug 19, 2011 at 14:59
4
 #include <fcntl.h>

int posix_fadvise(int fd, off_t offset, off_t len, int advice);

with advice option POSIX_FADV_DONTNEED:
The specified data will not be accessed in the near future.

1
3

I've found one technique (other than rebooting) that seems to work:

  1. Run a few copies of MemAlloc
  2. With each one, allocate large chunks of memory a few times
  3. Use Process Explorer to observe the System Cache size reducing to very low levels
  4. Quit the MemAlloc programs

It isn't selective though. Ideally I'd like to be able to clear the specific portions of memory being used for caching the disk blocks of files that I want to no longer be cached.

1
  • That has the side-effect of pushing stuff into the pagefile which will kill performance for quite a while afterwards. If you are going to resort to a kludge, then you may as well just read some other large file; at least that only clears the disk cache and nothing else.
    – Synetech
    Dec 1, 2013 at 0:52
3

For a much better view of the Windows XP Filesystem Cache - try ATM by Tim Murgent - it allows you to see both the filesystem cache Working Set size and Standby List size in a more detailed and accurate view. For Windows XP - you need the old version 1 of ATM which is available for download here since V2 and V3 require Server 2003,Vista, or higher.

You will observe that although Sysinternals Cacheset will reduce the "Cache WS Min" - the actual data still continues to exist in the form of Standby lists from where it can be used until it has been replaced with something else. To then replace it with something else use a tool such as MemAlloc or flushmem by Chad Austin or Consume.exe from the Windows Server 2003 Resource Kit Tools.

0
0

As the question also asked for Linux, there is a related answer here.

The command line tool vmtouch allows for adding and removing files and directories from the system file cache, amongst other things.

0

There's a windows API call https://learn.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-setsystemfilecachesize that can be used to flush the file system cache. It can also be used to limit the cache size to a very small value. Looks perfect for these kinds of tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.