I have a scenario where I need to call a function from LLDB (but it could be any C++ API) on Linux x64, from an app written in a different language. For that reason, I need to properly understand the calling-convention and how the arguments are passed.
I am trying to call SBDebugger::GetCommandInterpreter
, defined as:
lldb::SBCommandInterpreter GetCommandInterpreter();
The full header file can be found here: https://github.com/llvm-mirror/lldb/blob/master/include/lldb/API/SBDebugger.h
My assumption was that the method would expect a pointer to SBDebugger
as the this
argument, in the RDI register. However, when calling the function that way, I get a segmentation fault.
If I look at the disassembly of the function, I see:
push %r13
push %r12
mov %rsi,%r12
push %rbp
push %rbx
mov %rdi,%rbx
The function is reading from both RDI and RSI, despite only expecting the this
argument. The only explanation I see is that the function is expecting this
as a value rather than a reference. SBDebugger
has a size of 16 bytes (one shared_ptr
), and the calling convention states that a single 16 bytes parameter would be split into the RDI and RSI registers, which matches what I'm seeing.
However, that doesn't make sense to me for multiple reasons:
- If
this
is passed by value, how would it work if the method has any side-effect? The changes wouldn't be reflected on the caller - The System V ABI states that:
If a C++ object has either a non-trivial copy constructor or a non-trivial destructor 11, it is passed by invisible reference
. TheSBDebugger
does have a custom destructor and copy-constructor:
SBDebugger();
SBDebugger(const lldb::SBDebugger &rhs);
SBDebugger(const lldb::DebuggerSP &debugger_sp);
~SBDebugger();
Despite this, I tried calling the method by passing SBDebugger
by value and it seems to work, but I get a segmentation fault later when I try to use the returned SBCommandInterpreter
, so it's possible the method is only returning garbage.
There is something I don't understand about this method call, but I haven't been able to figure out what. What value should I pass in what registers, and why?
extern "C" lldb::SBCommandInterpreter SBDebugger_GetCommandInterpreter(SBDebugger *self) { return self->GetCommandInterpreter(); }
.