1

I have a scenario where I need to call a function from LLDB (but it could be any C++ API) on Linux x64, from an app written in a different language. For that reason, I need to properly understand the calling-convention and how the arguments are passed.

I am trying to call SBDebugger::GetCommandInterpreter, defined as:

  lldb::SBCommandInterpreter GetCommandInterpreter();

The full header file can be found here: https://github.com/llvm-mirror/lldb/blob/master/include/lldb/API/SBDebugger.h

My assumption was that the method would expect a pointer to SBDebugger as the this argument, in the RDI register. However, when calling the function that way, I get a segmentation fault.

If I look at the disassembly of the function, I see:

push   %r13 
push   %r12 
mov    %rsi,%r12 
push   %rbp 
push   %rbx 
mov    %rdi,%rbx 

The function is reading from both RDI and RSI, despite only expecting the this argument. The only explanation I see is that the function is expecting this as a value rather than a reference. SBDebugger has a size of 16 bytes (one shared_ptr), and the calling convention states that a single 16 bytes parameter would be split into the RDI and RSI registers, which matches what I'm seeing.

However, that doesn't make sense to me for multiple reasons:

  • If this is passed by value, how would it work if the method has any side-effect? The changes wouldn't be reflected on the caller
  • The System V ABI states that: If a C++ object has either a non-trivial copy constructor or a non-trivial destructor 11, it is passed by invisible reference. The SBDebugger does have a custom destructor and copy-constructor:
  SBDebugger();

  SBDebugger(const lldb::SBDebugger &rhs);

  SBDebugger(const lldb::DebuggerSP &debugger_sp);

  ~SBDebugger();

Despite this, I tried calling the method by passing SBDebugger by value and it seems to work, but I get a segmentation fault later when I try to use the returned SBCommandInterpreter, so it's possible the method is only returning garbage.

There is something I don't understand about this method call, but I haven't been able to figure out what. What value should I pass in what registers, and why?

2
  • How do you even get the name of the function to call? Dealing with the name mangling of C++ is ugly. I found it's often simpler to create a wrapper extern "C" lldb::SBCommandInterpreter SBDebugger_GetCommandInterpreter(SBDebugger *self) { return self->GetCommandInterpreter(); }. Commented Jul 15, 2022 at 22:09
  • @GoswinvonBrederlow Yeah my first approach was to use a C++ wrapper to expose the LLDB objects as virtual classes, so they can be easily consumed. Now I just wanted to try removing the C++ wrapper, purely for the technical challenge. Right now I hardcoded the mangled name of the functions but I'll replace it with an actual mangler (assuming I don't switch to something else half-way :p ) Commented Jul 16, 2022 at 15:27

1 Answer 1

2

I figured it out. I was focusing on the arguments but the trick was the return value. Quoting the System V calling convention:

A struct, class or union object can be returned from a function in registers only if it is sufficiently small and not too complex. If the object is too complex or doesn't fit into the appropriate registers then the caller must supply storage space for the object and pass a pointer to this space as a parameter to the function.

Basically, because SBCommandInterpreter is considered as a complex object, the GetCommandInterpreter expects an additional hidden parameter, which is the address to which the return value will be written. So the "real" method signature is:

 lldb::SBCommandInterpreter* GetCommandInterpreter(SBCommandInterpreter& returnValue, SBDebugger* this);

So the this argument is passed by reference as expected, I was just missing an additional hidden argument.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.