47

Recent versions of GCC and Clang feature Undefined Behavior Sanitizer (UBSan) which is a compile flag (-fsanitize=undefined) that adds runtime instrumentation code. On errors, a warning such as this one is shown:

packet-ber.c:1917:23: runtime error: left shift of 54645397829836991 by 8 places cannot be represented in type 'long int'

Now I would like to debug this and get a debug break on said line. For Address Sanitizer (ASAN) there is ASAN_OPTIONS=abort_on_error=1 which results in a fatal error that is catchable. The only UBSan option that seems usable is UBSAN_OPTIONS=print_stacktrace=1 which results in a call trace dump for reports. This however does not allow me to inspect the local variables and then continue the program. Use of -fsanitize-undefined-trap-on-error therefore not possible.

How should I break in gdb on UBSan reports? While break __sanitizer::SharedPrintfCode seems to work, the name looks quite internal.

5
  • 4
    I think that until an API is implemented and documented, a good way to catch a call to the UBSan runtime library with the intent of continuing your program is to do rbreak ^__ubsan_handle_, which will stop execution before the library ventures into C++ territory wherein it allocates instances of the Diag class. Poke around all you want, then type return to continue your program. Commented Jun 15, 2015 at 19:03
  • 11
    For future reference, abort_on_error seems unimplemented for UBSAN. Use this instead: UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1
    – Lekensteyn
    Commented Jul 7, 2015 at 17:08
  • 1
    Which GCC/Clang version are you using, precisely? Commented Jul 16, 2015 at 21:01
  • 2
    @IwillnotexistIdonotexist Very recent versions on Arch Linux, GCC 5.1.0, Clang 3.6.1.
    – Lekensteyn
    Commented Jul 16, 2015 at 23:51
  • @MarkPlotnick I don't understand the comment. What is the exact GDB breakpoint command required to achieve the OP's question? Commented Jan 22, 2023 at 0:48

3 Answers 3

31

While breaking on the detection functions (as described by @Mark Plotnick and @Iwillnotexist Idonotexist) is one option, a better approach is breaking on the functions that report these issues after detection. This approach is also used for ASAN where one would break on __asan_report_error.

Summary: You can stop on an ubsan report via a breakpoint on __ubsan::ScopedReport::~ScopedReport or __ubsan::Diag::~Diag. These are private implementation details which might change in the future though. Tested with GCC 4.9, 5.1.0, 5.2.0 and Clang 3.3, 3.4, 3.6.2.

For GCC 4.9.2 from ppa:ubuntu-toolchain-r/test, you need libubsan0-dbg to make the above breakpoints available. Ubuntu 14.04 with Clang 3.3 and 3.4 do not support the __ubsan::ScopedReport::~ScopedReport breakpoints, so you can only break before printing the message using __ubsan::Diag::~Diag.

Example buggy source code and a gdb session:

$ cat undef.c
int main(void) { return 1 << 1000; }
$ clang --version
clang version 3.6.2 (tags/RELEASE_362/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
$ clang -w -fsanitize=undefined undef.c -g
$ gdb -q -ex break\ __ubsan::ScopedReport::~ScopedReport -ex r ./a.out 
Reading symbols from ./a.out...done.
Breakpoint 1 at 0x428fb0
Starting program: ./a.out 
undef.c:1:27: runtime error: shift exponent 1000 is too large for 32-bit type 'int'

Breakpoint 1, 0x0000000000428fb0 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428fb0 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042affb in handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042a952 in __ubsan_handle_shift_out_of_bounds ()
#3  0x000000000042d057 in main () at undef.c:1

Detailled analysis follows. Note that both ASAN and ubsan both originate from a LLVM project, compiler-rt. This is used by Clang and ends up in GCC as well. Links in the following sections point to the compiler-rt project code, release 3.6.

ASAN has made its internal __asan_report_error part of the documented public interface. This function gets called whenever a violation is detected, its flow continues in lib/asan/asan_report.c:938:

void __asan_report_error(uptr pc, uptr bp, uptr sp, uptr addr, int is_write,
                         uptr access_size) {
  // Determine the error type.
  const char *bug_descr = "unknown-crash";
  ...

  ReportData report = { pc, sp, bp, addr, (bool)is_write, access_size,
                        bug_descr };
  ScopedInErrorReport in_report(&report);

  Decorator d;
  Printf("%s", d.Warning());
  Report("ERROR: AddressSanitizer: %s on address "
             "%p at pc %p bp %p sp %p\n",
             bug_descr, (void*)addr, pc, bp, sp);
  Printf("%s", d.EndWarning());

  u32 curr_tid = GetCurrentTidOrInvalid();
  char tname[128];
  Printf("%s%s of size %zu at %p thread T%d%s%s\n",
         d.Access(),
         access_size ? (is_write ? "WRITE" : "READ") : "ACCESS",
         access_size, (void*)addr, curr_tid,
         ThreadNameWithParenthesis(curr_tid, tname, sizeof(tname)),
         d.EndAccess());

  GET_STACK_TRACE_FATAL(pc, bp);
  stack.Print();

  DescribeAddress(addr, access_size);
  ReportErrorSummary(bug_descr, &stack);
  PrintShadowMemoryForAddress(addr);
}

ubsan on the other hand has no public interface, but its current implementation is also much simpler and limited (less options). On errors, a stacktrace can be printed when the UBSAN_OPTIONS=print_stacktrace=1 environment variable is set. Thus, by searching the source code for print_stacktrace, one finds function MaybePrintStackTrace which is called though the ScopedReport destructor:

ScopedReport::~ScopedReport() {
  MaybePrintStackTrace(Opts.pc, Opts.bp);
  MaybeReportErrorSummary(SummaryLoc);
  CommonSanitizerReportMutex.Unlock();
  if (Opts.DieAfterReport || flags()->halt_on_error)
    Die();
}

As you can see, there is a method to kill the program on errors, but unfortunately there is no builtin mechanism to trigger a debugger trap. Let's find a suitable breakpoint then.

The GDB command info functions <function name> made it possible to identify MaybePrintStackTrace as function on which a breakpoint can be set. Execution of info functions ScopedReport::~ScopedReport gave another function: __ubsan::ScopedReport::~ScopedReport. If none of these functions seem available (even with debugging symbols installed), you can try info functions ubsan or info functions sanitizer to get all (UndefinedBehavior)Sanitizer-related functions.

7
  • +1. It springs to my mind that your question exposes a need, and this need would be met if the sanitizers had a force-no-inline, extern-linkage, void-returning empty internal function that could be breakpointed by a debugger, and that this function be called with arguments useful to the debugger when reporting insanity. Something similar to the JIT registration interface. Commented Jul 23, 2015 at 19:03
  • 1
    This answer would be more useful if the answer to the actual question were made more prominent.
    – xaxxon
    Commented Mar 9, 2018 at 6:08
  • @xaxxon I am open for suggestions, could you clarify what part needs improvement?
    – Lekensteyn
    Commented Mar 9, 2018 at 12:55
  • clear specific steps for what do do not interspersed with a ton of unrelated stuff about asan.
    – xaxxon
    Commented Mar 9, 2018 at 18:26
  • @xaxxon The first part contains all information you need to set break on UBSAN reports. The second part is informational only, you can ignore it if you do not need to understand how things work. Should the internal implementation change in the future, then the second part should help the reader to figure out how to adapt. ASAN and UBSan are closely related, so I thought it would make sense to compare both.
    – Lekensteyn
    Commented Mar 11, 2018 at 15:41
19
+25

As @Mark Plotnick points out, the way to do so is to breakpoint at UBSan's handlers.

UBSan has a number of handlers, or magic function entry points, that are called for undefined behaviour. The compiler instruments code by injecting checks as appropriate; If the check code detects UB, it calls these handlers. They all start with __ubsan_handle_ and are defined in libsanitizer/ubsan/ubsan_handlers.h. Here's a link to GCC's copy of ubsan_handlers.h.

Here's the relevant bits of the UBSan header (breakpoint on any of these):

#define UNRECOVERABLE(checkname, ...) \
  extern "C" SANITIZER_INTERFACE_ATTRIBUTE NORETURN \
    void __ubsan_handle_ ## checkname( __VA_ARGS__ );

#define RECOVERABLE(checkname, ...) \
  extern "C" SANITIZER_INTERFACE_ATTRIBUTE \
    void __ubsan_handle_ ## checkname( __VA_ARGS__ ); \
  extern "C" SANITIZER_INTERFACE_ATTRIBUTE NORETURN \
    void __ubsan_handle_ ## checkname ## _abort( __VA_ARGS__ );

/// \brief Handle a runtime type check failure, caused by either a misaligned
/// pointer, a null pointer, or a pointer to insufficient storage for the
/// type.
RECOVERABLE(type_mismatch, TypeMismatchData *Data, ValueHandle Pointer)

/// \brief Handle an integer addition overflow.
RECOVERABLE(add_overflow, OverflowData *Data, ValueHandle LHS, ValueHandle RHS)

/// \brief Handle an integer subtraction overflow.
RECOVERABLE(sub_overflow, OverflowData *Data, ValueHandle LHS, ValueHandle RHS)

/// \brief Handle an integer multiplication overflow.
RECOVERABLE(mul_overflow, OverflowData *Data, ValueHandle LHS, ValueHandle RHS)

/// \brief Handle a signed integer overflow for a unary negate operator.
RECOVERABLE(negate_overflow, OverflowData *Data, ValueHandle OldVal)

/// \brief Handle an INT_MIN/-1 overflow or division by zero.
RECOVERABLE(divrem_overflow, OverflowData *Data,
            ValueHandle LHS, ValueHandle RHS)

/// \brief Handle a shift where the RHS is out of bounds or a left shift where
/// the LHS is negative or overflows.
RECOVERABLE(shift_out_of_bounds, ShiftOutOfBoundsData *Data,
            ValueHandle LHS, ValueHandle RHS)

/// \brief Handle an array index out of bounds error.
RECOVERABLE(out_of_bounds, OutOfBoundsData *Data, ValueHandle Index)

/// \brief Handle a __builtin_unreachable which is reached.
UNRECOVERABLE(builtin_unreachable, UnreachableData *Data)
/// \brief Handle reaching the end of a value-returning function.
UNRECOVERABLE(missing_return, UnreachableData *Data)

/// \brief Handle a VLA with a non-positive bound.
RECOVERABLE(vla_bound_not_positive, VLABoundData *Data, ValueHandle Bound)

/// \brief Handle overflow in a conversion to or from a floating-point type.
RECOVERABLE(float_cast_overflow, FloatCastOverflowData *Data, ValueHandle From)

/// \brief Handle a load of an invalid value for the type.
RECOVERABLE(load_invalid_value, InvalidValueData *Data, ValueHandle Val)

RECOVERABLE(function_type_mismatch,
            FunctionTypeMismatchData *Data,
            ValueHandle Val)

/// \brief Handle returning null from function with returns_nonnull attribute.
RECOVERABLE(nonnull_return, NonNullReturnData *Data)

/// \brief Handle passing null pointer to function with nonnull attribute.
RECOVERABLE(nonnull_arg, NonNullArgData *Data)

ASan is even easier. If you look in libsanitizer/include/sanitizer/asan_interface.h, which you should browse here, you can read a dead giveaway of a comment:

  // This is an internal function that is called to report an error.
  // However it is still a part of the interface because users may want to
  // set a breakpoint on this function in a debugger.
  void __asan_report_error(void *pc, void *bp, void *sp,
                           void *addr, int is_write, size_t access_size);

Numerous other functions in this header are explicitly commented as having been made public so as to be callable from a debugger.

I definitely advise you to explore other headers of libsanitizer/include/sanitizer here. There are numerous goodies to be had there.


Breakpoints for UBSan and ASan can be added as follows:

(gdb) rbreak ^__ubsan_handle_ __asan_report_error
(gdb) commands
(gdb) finish
(gdb) end

This will breakpoint on the handlers, and finish immediately afterwards. This allows the report to be printed, but the debugger gets control right after it gets printed.

12
  • 3
    I was hoping that a single breakpoint or environment variable existed for this. Anyway, can you show an example on using this feature with gdb? It would be great to have an approach (or simple gdbinit macro) that acts like ASAN. That is, display the message and break.
    – Lekensteyn
    Commented Jul 16, 2015 at 19:12
  • @Lekensteyn Environment variables can't inject breakpoints; and for a single breakpointable function to exist, UBSan would have to use a single, multiplexing, varargs function (which is not necessarily good design, and loses you the ability to efficiently breakpoint only on some UB types). With respect to gdb/gdbinit, sure, I'll add that in a couple hours. Commented Jul 16, 2015 at 20:44
  • @Lekensteyn Seems like (gdb) rbreak __asan_report_error and (gdb) rbreak ^__ubsan bluntly breakpoint all functions matching those regexes. There is no function called after __asan_report_error returns, so it would seem to me that you'd somehow have to program gdb to immediately (gdb) finish after hitting one of these breakpoints. Commented Jul 16, 2015 at 21:57
  • There are env vars such as UBSAN_OPTIONS that are supported by libsanitizer (not gdb or such), those vars I am referring to by "environment variables". As for the breakpoint, I tried rbreak ^__ubsan_handle_ commands finish end, but somehow this behaves strange with a very simple test program (function is still stuck in the __ubsan_handle_... frame) while a following info breakpoints somehow triggers the print and changes the frame. (GDB 7.9.1)
    – Lekensteyn
    Commented Jul 16, 2015 at 23:55
  • UBSan has an unsigned-integer-overflow check. Is there a way to say that a given expression with unsigned-integer-overflow is ok? (So that it is ignored from the report?).
    – gnzlbg
    Commented Oct 26, 2015 at 17:23
7

A breakpoint set at __asan_report_error is not hit for me and the program simply exits after printing the diagnostics without the debugger triggering. __asan::ReportGenericError before printing the diagnostics and __sanitizer::Die after printing the diagnostics do get hit as described in the asan wiki.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.