Questions tagged [mmap]
mmap is a POSIX-compliant Unix system call that maps files or devices into memory.
1,914
questions
-1
votes
0
answers
9
views
How to implement ”privatechat“ in the host version Linux chat room (using message queue),How to implement private chat function in Linux chat room
I have initially realized the function of group chat through shared memory, daemon process, message queue, etc., but there are always errors in private chat. The detailed code is as follows
/* "...
0
votes
1
answer
58
views
How to fix 'No such file or directory' error when using mmap shared memory in C? [closed]
I am trying to solve this problem in C with shared memory (using mmap) for my operating systems class. The purpose of this code is to use anonymous pipe(P0 to P1), named pipe(P1 to P2) and shared ...
0
votes
1
answer
38
views
Question about process race conditions on system calls
When you use mmap with MAP_PRIVATE on a file:
It is unspecified whether changes made to the file
after the mmap() call are visible in the mapped region
Does this mean that if you are mid-read, ...
0
votes
1
answer
44
views
What happens if you write to `mmap`ed sparse file when disk is full?
Many systems have mmap or equivalent functionality, where the contents of a file can be mapped to a memory address range.
Also many systems have sparse files, i.e. files with ranges of non-allocated ...
1
vote
0
answers
62
views
Mmap : failing with MAP_SYNC
[linux kernel 6.1.27 on Debian testing, up to date. Clang 16 or gcc 12.2]
May someone tell we what's wrong and why mmap returns MAP_FAILED:
int fd = ::open( file_name.data(), O_RDWR );
auto *ptr = ...
0
votes
1
answer
27
views
mmap failed on libcamera returned frame data
I'm working with camera on a Raspberry Pi 3B. The frame data returned by libcamera is given by tuples of raw fd, size and offset. And according to the official document I have to mmap them to access ...
0
votes
1
answer
20
views
Manually swap memory to files?
On platforms which does not have enough memory and which does not actively swap memory to files, mmap can be used to manually get "more memory than physically available".
My question is, if ...
0
votes
0
answers
25
views
file backed mmap with huge page?
-> Linux - Debian - Ext4 - Kernel 6.1
Other discussions
other relevant questions:
this try
This answer
This other question
Debian's
My context
I currently use a mmap file ("/user_db/db....
0
votes
1
answer
65
views
How can the parent process perceive the modification of the shared memory
I want to write log files using mmap in multi-process server, the processes can write to the initial log file, but if one process modify the shared address to a new file, other processes can not ...
1
vote
0
answers
26
views
boost mapped_file and page fault
I have done the following test using boost::iostreams::mapped_file and encountered some page fault (detected using getrusage). The page faults in step 4 are something I could not understand.
the file ...
-1
votes
1
answer
54
views
Why mmap fails to allocate executable page on macOS?
I am trying to run the code example in this answer https://stackoverflow.com/a/38974980 on macOS 13, and the program gives segmentation fault.
In fact, let's simplify the program linked above to
#...
1
vote
1
answer
31
views
How to query how many processes has a page mapped?
I have a pointer returned by mmap with MAP_SHARED. Is there a way to query the kernel for how many processes are currently mapping the pointed page?
Preferably in a portable POSIX way, but also ...
0
votes
0
answers
19
views
mmap memory kernel space to user space - too slow, any improvements?
In kernel driver, using remap_pfn_range to remap a physical memory block
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
vma->vm_flags |= VMEM_FLAGS | VM_PFNMAP;
vma->vm_pgoff = ...
0
votes
1
answer
39
views
How can I move ld.so higher in virtual memory?
My tool needs to recreate a portion of the memory layout of another processes. To do this reliably, I created a stub/wrapper that turns off ASLR, links only against libdl and libc to minimize the ...
0
votes
0
answers
46
views
Using multiple threads to read and write the same file without mmap?
Related: Multiple threads reading from the same file
mmap can be used to read and write files, but it does not support large files, for example, on a 32-bit system.
The stdio FILE* in C, or linux file ...
2
votes
2
answers
39
views
Transparently replace file mapping with anonymous
I am doing a checkpoint-and restore using CRIU; in turn after restore, my application wakes with some threads that have their stack mmaped into files on disk (CRIU doesn't do it by default, this is a ...
1
vote
0
answers
16
views
How to detect if `struct page` is backed by a file on a filesystem in Linux?
I am writing a Linux device module that need to occasionally pin userspace memory pages. Some of these pages may be backed by a file placed on a filesystem. That happens if a userspace program which ...
0
votes
0
answers
19
views
Are physical address of huge pages returned by mmap are divided by the huge page size?
It seems to me that OS won't allocate huge pages which are not align to the huge page size (not only the 4KB system page size), but I am not sure about it.
The documentation of mmap ensures that the ...
2
votes
1
answer
80
views
Why pages reclaims still present after using `munmap()`?
for a study project I have to code a reimplementation of malloc() and free() using mmap() and munmap().
I'm running on the last Ubuntu. For my tests I use the command time -v (from /usr/bin/time) ...
0
votes
0
answers
19
views
Does reading the void* from mmap as another type cause UB with regards to strict aliasing
When reading from mmap that another process wrote to, technically the object doesn't exist in the reading process.
So for example (writing process):
void* mmap_memory = mmap(....);
memcpy(mmap_memory, ...
0
votes
0
answers
23
views
Host slows down with heavy use of mmap?
I have some single threaded java nodes that which process some batch data. These nodes make heavy use of mmap specifically they write to an mmap file that is around 500gb in size which another process ...
-2
votes
1
answer
66
views
Mmap throws an error: mmap object has no attribute 'split'
I am writing a code that should search faster in a large file using mmap.
import re
import mmap
with open('words.txt', 'r', encoding="utf8") as f1, open('source.txt', 'r', encoding="...
0
votes
0
answers
80
views
Linux/C/mmap: Delay involved with IPC mechanism using Shared Memory Object in "dev/shm"
I have two processes(let's say process1 and process2) which are independent processes running on different cores. Also, process1 has higher time frequency than process2.
I'm trying to send one piece ...
0
votes
0
answers
95
views
Linux/C/mmap(): Avoiding the flushing of memory mapped region created using mmap to the underlying file
I am using mmap() to map device file "mtdblock0" into virtual address space of one process1. Some data whose value changes continuously for some time are stored in memory mapped region as ...
0
votes
0
answers
18
views
Android Java Studio - mmap failure (volley http posts)
Thanks for reading.
I am getting a weird mmap failure after a while. While running, the following function is being repeated 4 times a second:
public void sendLocationData(int id, double latitude, ...
0
votes
0
answers
20
views
Looking for a map structure, shareable between processes and language
I'm looking for an in-memory binary format for a map structure (key-value store) with quick access time.
My goal is to:
Store this binary format as a file.
Be able to retrieve it and with zero-copy, ...
0
votes
1
answer
49
views
Best way to load a mmap dictionary in Python without deserializing
Context:
I've got Python processes running on the same container and I want to be able to share a read-only key-value object between them.
I'm aware I could use something like Redis to share that info,...
4
votes
1
answer
150
views
Reasoning behind high latency when using python ctypes during process interrupts
While investigating a critical path in our python codebase, we found out that the behaviour of ctypes in terms of latencies is quite unpredictable.
A bit more background of our application. We have ...
0
votes
0
answers
55
views
If I private-`mmap` a file and read it, then another process writes to the same file, will another read at the same location return the same value?
(Context: I'm trying to establish which sequences of mmap operations are safe from the "memory safety" point of view, i.e. what assumptions I can make about mmaped memory without risking ...
3
votes
0
answers
269
views
Perf collection on kubernetes pods
I am trying to find performance bottlenecks by using the perf tool on a kubernetes pod. I have already set the following on the instance hosting the pod:
"kernel.kptr_restrict" = "0&...
1
vote
1
answer
160
views
mmap and instruction / data cache coherency: Why can we copy and run shared libraries?
On ARM after writing instructions to memory a memory barrier is needed before executing the instructions. Specifically clean the data cache, invalidate the instruction cache, then execute an ...
0
votes
0
answers
102
views
Too many brk() noticed in strace
I have a c++ service, where I am trying to debug the cause of high service startup time. From strace logs I notice a lot of brk() (which contributes to about 300ms, which is very high for our SLA). On ...
3
votes
2
answers
107
views
How do I open a mmap of size of a TB
I need to open a huge memory map. The file is one terabyte.
I however am getting an errno: ENOMEM 12 Cannot allocate memory. I don't get what is holding me up. Requesting the RLIMIT_AS results in the ...
1
vote
1
answer
175
views
x86_64 assembly mmap syscall [duplicate]
I try to make very simple memory allocator, but it gives me error on output, additionally the print_sys_error doesn't work properly. It works if I call it by this code:
_start:
sub $16, %rsp # ...
0
votes
0
answers
59
views
userspace-kernel shared memory with mmap works strange on multi-thread
I'm currently making shared memory between userspace to kernel.
The following code is simple version of my mmap method.
for (j = 0; j < 3; j++){
RxFrame[j] = kmalloc(4096*6, GFP_KERNEL);
...
2
votes
1
answer
93
views
How does GLIBC decide segment for malloc
I look at some Linux Glibc(2.25) system and see that when the code use malloc .
sometimes the buffer has been allocated at heap segment and sometimes in anonymous segment, It's not relate for size, I ...
0
votes
1
answer
45
views
Is there a way to wrap an numpy `ndarray` interface around an existing binary file?
I have a binary network capture (.pcapng) file that contains video data. I am parsing the .pcapng with scapy and I can extract the data, but the video sequences I am working with are very large and ...
0
votes
1
answer
36
views
msync guarantees on linux
Is msync guaranteed to physically write dates into the hard-drive? Can I be 100% sure that if the OS crashes immediately after msync returns, that I won't lose data?
By 100% sure I mean that the data ...
0
votes
2
answers
59
views
Musl mmap ENOMEM
I compiled some cross compilers against musl(x86_64, i686, arm). I need to compile code, that allocate like 2048 Mb +- 200Mb. However I noticed some errors with i686 musl compiler:
#include <stdio....
-2
votes
1
answer
205
views
What's the best practice for swap apache arrow data between different processes? [closed]
I have a data api which could get stream data use rust as an independent service process, and plan to write several client process to read the data, because the client process have some function based ...
4
votes
0
answers
53
views
munmap of an unlinked file - is the data flushed?
I am interested in using an mmap'ed file as a private swap space in case the machine doesn't have enough RAM for my huge tables. When I'm done with this processing, I no longer need these tables and ...
2
votes
1
answer
142
views
Azerothcore error azerothcore-wotlk-ac-worldserver-1 | MMAP:loadMap: 5303231.mmtile was built with generator v15, expected v16
I recently updated my Azerothcore docker version, but I am encountering an error: "azerothcore-wotlk-ac-worldserver-1 | MMAP:loadMap: 5303231.mmtile was built with generator v15, expected v16&...
2
votes
2
answers
128
views
Failed to read pci csr via mmap using uint64_t pointer [closed]
I'm trying to reading PCI CSR (Configuration Space Register) on my system via open,mmap /dev/mem.
I met some problems when using 8 byte length reading
Here is the minimal working example of my code
#...
0
votes
0
answers
58
views
I am trying to dynamically allocate memory for a matrix in assembly using mmap but it does not work
I am having trouble with the syscall mmap to dynamically allocate memory for a matrix,%eax should return at the end of the call the memory address but it returns a big negative value and the program ...
0
votes
0
answers
61
views
How could I do to mmap a large memory area with Anonymous Mappings but no-zero-filled to be done?
As *The Linux Programming Interface * said Anonymous Mappings: With both the MAP_ANONYMOUS and the /dev/zero techniques, the bytes of the resulting mapping are initialized to 0.
I want to mmap a ...
1
vote
1
answer
64
views
Use mmap for random read from file but the read throuput is not as expected
I write k-v data into file with value size of 100KB and key size of 10Bytes. Only value data is written into file and indices of keys are stored in ram. I write 10M keys and I get a 1TB value file. I ...
0
votes
2
answers
108
views
How to pass void pointer by reference
I want to have a void* as a function parameter and then inside a function modify that pointer (change it to NULL).
Code below doesn't change memory1 to null after function call. How can I change that.
...
1
vote
1
answer
65
views
mallinfo doesn't show mmap allocation's information
In mallinfo structure there are two fields hblks and hblkhd. The man documentation says that they are responsible for the number of blocks allocated by mmap and the total number of bytes. But when I ...
3
votes
0
answers
90
views
Valgrind massif maximum mmap size
When I use mmaping for a large file(80-150 GB) valgrind failed with the error (mmap): Invalid argument, intuitively, I realized that the problem was probably with the file size, I tried to reduce the ...
4
votes
1
answer
90
views
Window control for mmapped large file(linux, mmap)
How can we control the window in RSS when mapping a large file? Now let me explain what i mean.
For example, we have a large file that exceeds RAM by several times, we do shared memory mmaping for ...