UBSan: signed integer overflow in load_sixel() at src/loader.c

[MMu-Aqua]

Hi team, thanks for your work on libsixel.

I found a UBSan-reported undefined-behavior issue when converting a crafted input with img2sixel.

Summary

A crafted input triggers a reproducible signed integer overflow in load_sixel() at src/loader.c:703.

Impact

A malformed input can trigger signed integer overflow while loading image data. Since signed integer overflow is undefined behavior in C, this may result in an incorrect buffer-size calculation or cause sanitizer-enabled builds to abort.

Environment

Target: img2sixel
Build: AddressSanitizer + UndefinedBehaviorSanitizer
Crash type: UndefinedBehaviorSanitizer: signed integer overflow
Component: image loader
Function: load_sixel()
Location: src/loader.c:703

Root Cause

The root cause appears to be missing integer overflow checks in load_sixel() when handling untrusted image data.

At the crashing site, a signed int multiplication overflows:
loader.c:703:91: runtime error: signed integer overflow: 774779005 * 3 cannot be represented in type int

This suggests that the loader performs arithmetic on input-derived dimensions without validating their range before multiplication.

A fix likely requires validating the decoded dimensions and using checked size_t arithmetic before calculating the allocation size.

Reproduction

Set sanitizer options:

export ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:detect_leaks=0:allocator_may_return_null=1:symbolize=1
export UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_stacktrace=1
export ASAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer

Command:

./img2sixel -b gray8 poc_loader_703_input -o /tmp/out.six

PoC:

poc_loader_703_input.zip

Actual result:

loader.c:703:91: runtime error: signed integer overflow: 774779005 * 3 cannot be represented in type 'int'
    #0 0x560e467d7f1e in load_sixel /home/shangzhi/mmu/libsixel-afl-asan/src/loader.c:703:91
    #1 0x560e467d7f1e in load_with_builtin /home/shangzhi/mmu/libsixel-afl-asan/src/loader.c:868:18
    #2 0x560e467d7f1e in sixel_helper_load_image_file /home/shangzhi/mmu/libsixel-afl-asan/src/loader.c:1464:18
    #3 0x560e46792171 in sixel_encoder_encode /home/shangzhi/mmu/libsixel-afl-asan/src/encoder.c:1952:14
    #4 0x560e4678571b in main /home/shangzhi/mmu/libsixel-afl-asan/converters/img2sixel.c:455:22
    #5 0x7f577a429d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #6 0x7f577a429e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #7 0x560e466af6e4 in _start (/home/shangzhi/mmu/libsixel-afl-asan/converters/img2sixel+0x7d6e4) (BuildId: 0dec88ac02ad4286cee0287c89866762870b3fe8)

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior loader.c:703:91 in
Aborted