I've noticed a really weird behavior when I was playing with libc's system() function on x86-64 linux, sometimes the call to system()
fails with a segmentation fault, here's what I got after debugging it with gdb
.
I've noticed that the segmentation fault is cased in this line:
=> 0x7ffff7a332f6 <do_system+1094>: movaps XMMWORD PTR [rsp+0x40],xmm0
According to the manual, this is the cause of the SIGSEGV:
When the source or destination operand is a memory operand, the operand must be aligned on a 16-byte boundary or a general-protection exception (#GP) is generated.
Looking deeper down, I've noticed that indeed my rsp
value was not 16 byte padded (that is, its hex representation didn't end with 0
). Manually modifying the rsp
right before the call to system
actually makes everything work.
So I've written the following program:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
register long long int sp asm ("rsp");
printf("%llx\n", sp);
if (sp & 0x8) /* == 0x8*/
{
printf("running system...\n");
system("touch hi");
}
return 0;
}
Compiled with gcc 7.3.0 And sure enough, when observing the output:
sha@sha-desktop:~/Desktop/tda$ ltrace -f ./o_sample2
[pid 26770] printf("%llx\n", 0x7ffe3eabe6c87ffe3eabe6c8
) = 13
[pid 26770] puts("running system..."running system...
) = 18
[pid 26770] system("touch hi" <no return ...>
[pid 26771] --- SIGSEGV (Segmentation fault) ---
[pid 26771] +++ killed by SIGSEGV +++
[pid 26770] --- SIGCHLD (Child exited) ---
[pid 26770] <... system resumed> ) = 139
[pid 26770] +++ exited (status 0) +++
So with this program, I cannot execute system()
what so ever.
Small thing also, and I cannot tell if its relevant to the problem, almost all of my runs end up with a bad rsp
value and a child that is killed by SEGSEGV.
This makes me wonder a few things:
- Why does
system
mess around with thexmm
s registers? - Is it a normal behavior? or maybe I'm missing something elementary in regards to how to use the
system()
function properly?
Thanks in advance
register long long int sp asm ("rsp");
and accessing it as a variable without extended inline assembly is behavior that isn't defined. It is only lucky it works. I'd like to see the original code that apparently fails. Can you show us the original code where you call system and it fails? This smells like an XY problemsystem
'ssystem
per the x86-64 ABI. As you suggested rounding RSP down to the nearest 16-byte boundary works. I assume you do it with something likeand rsp, -16
? The ABI states that at the point of a function call the stack pointer needs to be 16 byte (some cases 32-byte) aligned. And there is nothing wrong with functions likesystem
using aligned vector instruction to improve performance (that is normal).rsp
isn't aligned is because of theasm
? I thought it's a known directive togcc
and besides the fact that it compiles my program, shouldn't it also not "mess up" myrsp
?system
If whatever you do in the ROB code misaligns the stack, it is your job to align it.