33

What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?

0

6 Answers 6

31

You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [240 to 240 + 1024] with two separate views.)

In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.

Sign up to request clarification or add additional context in comments.

3 Comments

Yes, I should have said "view". I need to have access to the whole file at one time, without re-mmapp()ing.
MapViewOfFile takes a 64-bit length on a 64-bit machine
an anyway one can pass ZERO for a size which means "as large as file is"
9

This has been my experience when using memory-mapped files under Win32:

If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.

If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.

I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.

3 Comments

On 64-bit the limit is the available address space, the same as on 32-bit. You can map a 1 GB view on a 32-bit machine which only has 32MB physical memory. The same way you can map a 10TB view on a 32-bit machine which only has 0.5GB of memory. Only the 4kb pages you really access are mapped into real memory.
@springy76: you cannot map a 10TB view on a 32-bit machine. Only on 64-bit.
@SergeyK. you are damn right, thats also what I wanted to write -- but my fingers typed something else. It should have read "The same way you can map a 10TB view on a 64-bit machine which only has 0.5GB of memory"
2

Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "

From MDSN.

I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.

2 Comments

just to clarify, that's the size of a file view, not the size of the file itself.
"2 GB minus the virtual memory already reserved by the process" is on 32 bit machine? Is it different on a 64 bit box?
2

Yes, there are limits to memory-mapped files. Most shockingly is:

Memory-mapped files cannot be larger than 2GB on 32-bit systems.

When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified. On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.

Even on my 64-bit, 32GB RAM system, I get the following error if I try to read in one big numpy memory-mapped file instead of taking portions of it using byte-offsets:

Overflow Error: memory mapped size must be positive

Big datasets are really a pain to work with.

1 Comment

Are you using a 32-bit executable?
1

The limit of virtual address space is >16 Terabyte on 64Bit Windows systems. The issue discussed here is most probably related to mixing DWORD with SIZE_T.

7 Comments

Hello Mark, the information and discussion here is confusing publishing wrong information. It is missguiding readers to assume there is a limitation to 4 gbyte. The maximum limit on windows system for mmaped files is 16 Terabyte.
Hello Mark, thank you for confirmation. Please let me add, that the given number is a theoretical limit only. In general the really available virtual address space is limited by the disk i/o speed versus the hard page fault frequency. This factor defines the maximum usable pagefile size.
On 32-bit Windows the limit is 16TB, and on 64-bit Windows the limit is 256TB. Virtual memory is also physically limited by the available disc space. This is taken from stackoverflow.com/questions/6608820/…
For allocating and managing huge amounts of memory in Windows Systems see also codeproject.com/Articles/5267559/…
For more information on Windows virtual address space please be advised to techcommunity.microsoft.com/t5/windows-blog-archive/…
|
-6

There should be no other limits. Aren't those enough? ;-)

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.