13

Mapping an input file into memory and then directly parsing data from the mapped memory pages can be a convenient and efficient way to read data from files.

However, this practice also seems fundamentally unsafe unless you can ensure that no other process writes to a mapped file, because even the data in private read-only mappings may change if the underlying file is written to by another process. (POSIX e.g. doesn't specify "whether modifications to the underlying object done after the MAP_PRIVATE mapping is established are visible through the MAP_PRIVATE mapping".)

If you wanted to make your code safe in the presence of external changes to the mapped file, you'd have to access the mapped memory only through volatile pointers and then be extremely careful about how you read and validate the input, which seems impractical for many use cases.

Is this analysis correct? The documentation for memory mapping APIs generally mentions this issue only in passing, if at all, so I wonder whether I'm missing something.

6
  • It's possible for another process to modify a file while you are using it. Memory mapping is one of a great many ways to use a file - it's no more unsafe than any of the others.
    – Casey
    Jan 22, 2014 at 16:16
  • 1
    Copying file content with the normal read-file APIs into a memory buffer is always safe, even if the file is concurrently modified. In contrast, if you access a memory-mapped file through normal pointers and an external process modifies it concurrently, you get undefined behaviour, because this is outside the constraints of the C(++) memory model. Generally, if memory that you or your compiler expect to be constant changes under your feet, bad things will happen. Input that you validate in one moment can become invalid in the next moment. Jan 22, 2014 at 16:58
  • You're misreading the purpose of MAP_PRIVATE. It doesn't mean give me a private copy it means modifications made by me are private to me. It has all the same concurrency issues as any other method for accessing a file Jan 22, 2014 at 17:16
  • @Petesh Where did I write or imply that MAP_PRIVATE means "give me a private copy"? I actually quoted the part of the spec stating the opposite. It would be nice though if there was an option that actually did ensure that mapped pages aren't changed by other processes once they have been accessed. Jan 22, 2014 at 18:17
  • @StephanTolksdorf it seemed to be the essence of your question. And the 'option to ensure that mapped pages aren't changed by other processes once they've been acccessed' is accomplished by writing to the page. mmap is nothing other than a convenience over read/write Jan 22, 2014 at 18:54

2 Answers 2

3

It is not really a problem.

Yes, another process may modify the file while you have it mapped, and yes, it is possible that you will see the modifications. It is even likely, since almost all operating systems have unified virtual memory systems, so unless one requests unbuffered writes, there's no way of writing without going through the buffer cache, and no way without someone holding a mapping seeing the change.
That isn't even a bad thing. Actually, it would be more disturbing if you couldn't see the changes. Since the file quasi becomes part of your address space when you map it, it makes perfect sense that you see changes to the file.

If you use conventional I/O (such as read), someone can still modify the file while you are reading it. Worded differently, copying file content to a memory buffer is not always safe in presence of modifications. It is "safe" insofar as read will not crash, but it does not guarantee that your data is consistent.
Unless you use readv, you have no guarantees about atomicity whatsoever (and even with readv you have no guarantee that what you have in memory is consistent with what is on disk or that it doesn't change between two calls to readv). Someone might modify the file between two read operations, or even while you are in the middle of it.
This isn't just something that isn't formally guaranteed but "probably still works" -- on the contrary, e.g. under Linux writes are demonstrably not atomic. Not even by accident.

The good news:
Usually, processes don't just open an arbitrary random file and start writing to it. When such a thing happens, it is usually either a well-known file that belongs to the process (e.g. log file), or a file that you explicitly told the process to write to (e.g. saving in a text editor), or the process creates a new file (e.g. compiler creating an object file), or the process merely appends to an existing file (e.g. db journals, and of course, log files). Or, a process might atomically replace a file with another one (or unlink it).

In every case, the whole scary problem boils down to "no issue" because either you are well aware of what will happen (so it's your responsibility), or it works seamlessly without interfering.

If you really don't like the possibility that another process could possibly write to your file while you have it mapped, you can simply omit FILE_SHARE_WRITE under Windows when you create the file handle. POSIX makes it somewhat more complicated since you need to fcntl the descriptor for a mandatory lock, which isn't necessary supported or 100% reliable on every system (for example, under Linux).

5
  • I'm not concerned about the atomicity of reads or writes. When you copy file contents into a memory buffer using read, you can validate the read input afterwards and then be sure that the content in the buffer stays valid. When an input file is mapped into memory, it is typically treated as a constant buffer afterwards and accessed and parsed directly through normal pointers. If this presumably constant memory is changed by an external process, it breaks compiler and developer expectations, which can lead to undefined behaviour. Jan 22, 2014 at 17:33
  • That's right, if this is really a problem, you can still do a memcpy to an anonymous mapping (or to a heap-allocated block). Of course then you lose one of the main advantages of memory mapping: Data is no longer available "magically" and "for free". I wouldn't bother with copying, though. Thing is, you normally know when you modify a file (or a program on your behalf), this does not just happen randomly on random files.
    – Damon
    Jan 22, 2014 at 17:44
  • @Damon This is a source file. People do open them at unexpected moments, and people do write them back when you're least expecting it. Jan 22, 2014 at 18:39
  • @JamesKanze: How do you know the question is about a source file? Did I miss something?
    – Damon
    Jan 22, 2014 at 18:57
  • @Damon He talked about parsing. So it's a source for his program. (But it's true that one parses many things... I've parsed log files before.) Jan 22, 2014 at 19:01
1

In theory, you're probably in real trouble if someone does modify the file while you're reading it. In practice: you're reading characters, and nothing else: no pointers, or anything which could get you into trouble. In practice... formally, I think it's still undefined behavior, but it's one which I don't think you have to worry about. Unless the modifications are very minor, you'll get a lot of compiler errors, but that's about the end of it.

The one case which might cause problems is if the file was shortened. I'm not sure what happens then, when you're reading beyond the end.

And finally: the system isn't arbitrarily going to open and modify the file. It's a source file; it will be some idiot programmer who does it, and he deserves what he gets. In no case will your undefined behavior corrupt the system or other peoples files.

Note too that most editors work on a private copy; when the write back, they do so by renaming the original, and creating a new file. Under Unix, once you've opened the file to mmap it, all that counts is the inode number. And when the editor renames or deletes the file, you still keep your copy. The modified file will get a new inode. The only thing you have to worry about is if someone opens the file for update, and then goes around modifying it. Not many programs do this on text files, except for appending additional data to the end.

So while formally, there's some risk, I don't think you have to worry about it. (If you're really paranoid, you could turn off write authorisation while you're mmaped. And if there's really an enemy agent out to get your, he can turn it right back on.)

2
  • I don't think the OP is talking about reading source files per se, just any file the program may read at run-time. As such, your point about compiler errors seem out of place. Jan 22, 2014 at 18:57
  • @MarcClaesen He mentioned parsing. Compiler errors may be a bit strong, but anytime you're parsing text, you need to be able to handle errors in the input in some way. Jan 22, 2014 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.