Repository navigation
Description
Origin: the sanitizer CI job added in #3350 (part of #3338). Both of its runs on that PR failed with the same reports: run 44383473209 (commit 7fb12e984) and run 44405408309 (commit 5d039250b). Requested by @asclearuc as the next step of #3338.
Description
The job Sanitizers (ASan + UBSan) builds the engine, its C++ test programs and the C++ nodes with --cmake-config=Sanitize (ASan and UBSan in one build), then runs server:test and nodes:test. Every test passes. The step Fail on sanitizer reports then fails: both runs wrote 492 reports in 119 report files, which scripts/ci/sanitizer-report.mjs sorts into 5 groups.
| # | Reports | Sanitizer | Kind | Where it surfaced | Programs that reported it |
|---|---|---|---|---|---|
| 1 | 414 | UBSan | function-type-mismatch |
packages/server/engine-core/apLib/factory/Factory.hpp:84:16 |
every program: 114 engine processes, aptest, engtest, nodetest |
| 2 | 71 | ASan | alloc-dealloc-mismatch (operator new vs free) |
free called from __cxa_end_catch in libc++abi.so.1, reached from pybind11::cpp_function::dispatcher; the summary table names packages/server/engine-lib/engLib/store/python/python-endpoint.source.cpp:110:9, the first frame in our code |
71 engine processes during nodes:test |
| 3 | 4 | ASan | heap-use-after-free, READ of size 4 |
strlen in Catch2 argument parsing, called from packages/server/engine-lib/test/testMain.ipp:230:36 |
start of engtest and nodetest |
| 4 | 2 | ASan | alloc-dealloc-mismatch (operator new [] vs operator delete) |
packages/server/engine-core/apLib/memory/pmr.hpp:173:13, when the indexer closes its word DB |
nodetest |
| 5 | 1 | ASan | alloc-dealloc-mismatch (operator new vs free) |
the same frames as group 2 (free from __cxa_end_catch in libc++abi.so.1), reached from the test at packages/server/engine-core/test/async/Callback.cpp:125; the summary table names packages/server/engine-core/apLib/error/call.hpp:75:13 |
aptest |
In groups 2 and 5 the memory was allocated by operator new called from std::runtime_error::runtime_error(char const*) in libc++.so.1, and freed by free called from __cxa_end_catch in libc++abi.so.1. The frame between them is named std::range_error::~range_error(), also for the std::runtime_error thrown in Callback.cpp:125. In both groups the frames from free up to __cxa_end_catch are all in the sanitizer runtime and libc++abi.so.1; the first frame outside them is pybind11 (group 2) or our code (group 5).
Steps to Reproduce
In CI: run the CI workflow manually (Actions → CI → Run workflow) on develop, or open a pull request that changes C++ code and has the ci:sanitize label. Open the job Sanitizers (ASan + UBSan): the step Fail on sanitizer reports fails, the job summary shows the table above, and the sanitizer-reports artifact holds the report files.
Locally, on Linux with clang 18 (the commands from Sanitizer build):
mkdir -p "$PWD/sanitizer-reports"
export ASAN_OPTIONS="halt_on_error=0:detect_leaks=0:log_path=$PWD/sanitizer-reports/report"
export UBSAN_OPTIONS="halt_on_error=0:print_stacktrace=1:report_error_type=1:log_path=$PWD/sanitizer-reports/report"
export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-18/bin/llvm-symbolizer
export RR_SUBPROCESS_ENV="ASAN_* UBSAN_*"
./builder server:build --cmake-config=Sanitize --nodownload
./builder server:test --cmake-config=Sanitize --nodownload
./builder nodes:test --cmake-config=Sanitize --nodownload
node scripts/ci/sanitizer-report.mjs "$PWD/sanitizer-reports"Expected vs Actual Behavior
Expected: no ASan or UBSan reports, so the job passes.
Actual: 492 reports in 5 groups in each run, and the job fails. With the default ASan options (halt_on_error=1), engtest and nodetest stop with exit code 1 at group 3, before their first test.
Severity
P3 - Low (minor inconvenience)
Platform
- Windows
- macOS
- Linux
The Sanitize build exists on Linux only; Windows and macOS were not checked.
Version / Environment
develop with #3350 (f49ce3c4e); CI runner ubuntu-22.04, clang 18 and llvm-18 from apt, build type Sanitize.
Screenshots / Logs
Excerpts from the report files of run 44383473209. Paths are shortened to the repo root, BuildIds are removed, and long frames are cut with ....
Group 1: UBSan function-type-mismatch (report.11464, engine start)
packages/server/engine-core/apLib/../apLib/factory/Factory.hpp:84:16: runtime error: call to function ap::ErrorOr<std::__1::unique_ptr<engine::monitor::Monitor, void (*)(engine::monitor::Monitor*)>> ap::Factory::objectConstructor<engine::monitor::Console, engine::monitor::Monitor, engine::monitor::Monitor::FactoryArgs>(ap::Location, engine::monitor::Monitor::FactoryArgs&&) through pointer to incorrect function type 'ap::ErrorOr<std::unique_ptr<engine::monitor::Monitor, void (*)(engine::monitor::Monitor *)>> (*)(ap::Location, engine::monitor::Monitor::FactoryArgs) noexcept'
packages/server/engine-core/apLib/../apLib/factory/Factory.hpp:114: note: ap::ErrorOr<std::__1::unique_ptr<engine::monitor::Monitor, void (*)(engine::monitor::Monitor*)>> ap::Factory::objectConstructor<engine::monitor::Console, engine::monitor::Monitor, engine::monitor::Monitor::FactoryArgs>(ap::Location, engine::monitor::Monitor::FactoryArgs&&) defined here
#0 0x7f076aa8dc62 in ap::ErrorOr<std::__1::unique_ptr<engine::monitor::Monitor, void (*)(engine::monitor::Monitor*)>> ap::Factory::find<engine::monitor::Monitor, engine::monitor::Monitor::FactoryArgs&>(ap::Location, unsigned int, ap::string::StrView<char, std::__1::char_traits<char>>, engine::monitor::Monitor::FactoryArgs&) packages/server/engine-core/apLib/../apLib/factory/Factory.hpp:84:16
#1 0x7f076958806c in engine::monitor::Monitor::__factory(ap::Location, unsigned int, engine::monitor::Monitor::FactoryArgs) packages/server/engine-lib/engLib/../engLib/monitor/Monitor.hpp:74:16
#2 0x7f076958806c in ap::ErrorOr<std::__1::unique_ptr<engine::monitor::Monitor, void (*)(engine::monitor::Monitor*)>> ap::Factory::make<engine::monitor::Monitor, ap::string::Str<char, std::__1:...
#3 0x7f076958806c in engine::monitor::init() packages/server/engine-lib/engLib/monitor/init.cpp:77:20
#4 0x7f0769584edf in engine::init() packages/server/engine-lib/engLib/core/init.cpp:69:22
#5 0x7f07692ff9d9 in (anonymous namespace)::engineMain() packages/server/engine-mod/src/engine.cpp:29:13
#6 0x7f076b38f1aa in ap::application::bootstrap(int, char const**, std::__1::error_code (*)())::$_0::operator()() const packages/server/engine-core/apLib/application/api.cpp:141:52
...
SUMMARY: UndefinedBehaviorSanitizer: function-type-mismatch packages/server/engine-core/apLib/../apLib/factory/Factory.hpp:84:16
The same report comes from the other factory lookups too; every report file in both runs has at least one.
Group 2: ASan alloc-dealloc-mismatch, operator new vs free (report.18255, engine during nodes:test)
==18255==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new vs free) on 0x5080058674a0
#0 0x563bc010a2f6 in free (dist/server/engine+0xc62f6)
#1 0x7fdb16e44bdb in std::range_error::~range_error() (dist/server/libc++abi.so.1+0x27bdb)
#2 0x7fdb16e46f63 in __cxa_end_catch (dist/server/libc++abi.so.1+0x29f63)
#3 0x7fdb1241f9cc in pybind11::cpp_function::dispatcher(_object*, _object* const*, unsigned long, _object*) build/vcpkg_installed/x64-linux-clang-rocketride/include/pybind11/pybind11.h
#4 0x7fdb139bb3a0 in cfunction_vectorcall_FASTCALL_KEYWORDS build/vcpkg/buildtrees/python3/x64-linux-clang-rocketride-rel/./../src/v3.12.13-6aa6a2d879.clean/Objects/methodobject.c:438:24
#5 0x7fdb139798bf in _PyObject_VectorcallTstate build/vcpkg/buildtrees/python3/x64-linux-clang-rocketride-rel/./../src/v3.12.13-6aa6a2d879.clean/Include/internal/pycore_call.h:92:11
...
0x5080058674a0 is located 0 bytes inside of 90-byte region [0x5080058674a0,0x5080058674fa)
allocated by thread T143 here:
#0 0x563bc01478ed in operator new(unsigned long) (dist/server/engine+0x1038ed)
#1 0x7fdb1115859f in std::runtime_error::runtime_error(char const*) (dist/server/libc++.so.1+0x5259f)
#2 0x7fdb120e0637 in void pybind11::cpp_function::initialize<engine::store::pythonBase::pybind11_init_engLib(pybind11::module_&)::$_9, void, pybind11::args, pybind11::name, pybind11::scope, pyb...
#3 0x7fdb120e0637 in void pybind11::cpp_function::initialize<engine::store::pythonBase::pybind11_init_engLib(pybind11::module_&)::$_9, void, pybind11::args, pybind11::name, pybind11::scope, pyb...
#4 0x7fdb1241d32d in pybind11::cpp_function::dispatcher(_object*, _object* const*, unsigned long, _object*) build/vcpkg_installed/x64-linux-clang-rocketride/include/pybind11/pybind11.h:1209:30
#5 0x7fdb139bb3a0 in cfunction_vectorcall_FASTCALL_KEYWORDS build/vcpkg/buildtrees/python3/x64-linux-clang-rocketride-rel/./../src/v3.12.13-6aa6a2d879.clean/Objects/methodobject.c:438:24
SUMMARY: AddressSanitizer: alloc-dealloc-mismatch (dist/server/engine+0xc62f6) in free
Group 3: ASan heap-use-after-free (report.30952, engtest start)
==30952==ERROR: AddressSanitizer: heap-use-after-free on address 0x5070001403b0 at pc 0x556409d0545b bp 0x7ffeb69cea80 sp 0x7ffeb69ce248
READ of size 4 at 0x5070001403b0 thread T0
#0 0x556409d0545a in strlen (dist/server/engtest+0xa0045a)
#1 0x55640ba304a1 in Catch::StringRef::StringRef(char const*) (dist/server/engtest+0x272b4a1)
#2 0x55640ba1363a in Catch::Clara::Args::Args(int, char const* const*) (dist/server/engtest+0x270e63a)
#3 0x55640ba07192 in Catch::Session::applyCommandLine(int, char const* const*) (dist/server/engtest+0x2702192)
#4 0x55640a39d7b2 in int Catch::Session::run<char>(int, char const* const*) build/vcpkg_installed/x64-linux-clang-rocketride/include/catch2/catch_session.hpp:47:30
#5 0x55640a39d7b2 in ap::application::TestMain(std::__1::vector<char const*, std::__1::allocator<char const*>>) packages/server/engine-lib/test/./testMain.ipp:230:36
...
0x5070001403b0 is located 0 bytes inside of 80-byte region [0x5070001403b0,0x507000140400)
freed by thread T0 here:
#0 0x556409dc5dbd in operator delete(void*) (dist/server/engtest+0xac0dbd)
#1 0x55640a39d680 in void std::__1::__libcpp_operator_delete[abi:ne180100]<void*>(void*) /usr/lib/llvm-18/bin/../include/c++/v1/new:280:3
#2 0x55640a39d680 in void std::__1::__do_deallocate_handle_size[abi:ne180100]<>(void*, unsigned long) /usr/lib/llvm-18/bin/../include/c++/v1/new:302:10
#3 0x55640a39d680 in std::__1::__libcpp_deallocate[abi:ne180100](void*, unsigned long, unsigned long) /usr/lib/llvm-18/bin/../include/c++/v1/new:317:12
#4 0x55640a39d680 in std::__1::allocator<char>::deallocate[abi:ne180100](char*, unsigned long) /usr/lib/llvm-18/bin/../include/c++/v1/__memory/allocator.h:139:7
#5 0x55640a39d680 in std::__1::allocator_traits<std::__1::allocator<char>>::deallocate[abi:ne180100](std::__1::allocator<char>&, char*, unsigned long) /usr/lib/llvm-18/bin/../include/c++/v1/__m...
...
previously allocated by thread T0 here:
#0 0x556409dc555d in operator new(unsigned long) (dist/server/engtest+0xac055d)
#1 0x7f2b4d67192c in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__grow_by_and_replace(unsigned long, unsigned long, unsigned long, unsigned long, unsi...
SUMMARY: AddressSanitizer: heap-use-after-free (dist/server/engtest+0xa0045a) in strlen
nodetest shows the same report at its start.
Group 4: ASan alloc-dealloc-mismatch, operator new [] vs operator delete (report.30953, nodetest)
==30953==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new [] vs operator delete) on 0x7f09d1aee800
#0 0x558dace4f64d in operator delete(void*) (dist/server/nodetest+0x23264d)
#1 0x7f0a3c5d455e in ap::memory::constant_mbr::_M_release_buffers() packages/server/engine-core/apLib/memory/pmr.hpp:173:13
#2 0x7f0a3c5d455e in ap::memory::constant_mbr::release() packages/server/engine-core/apLib/memory/pmr.hpp:96:22
#3 0x7f0a3c5edabd in engine::index::db::WordDb<(ap::file::Mode)3>::close(bool) packages/server/engine-lib/engLib/index/db/Write.hpp:790:19
#4 0x7f0a3c269a4f in engine::store::filter::indexer::IFilterGlobal::closeWordDb()::$_0::operator()() const nodes/src/nodes/indexer/src/indexer.write.cpp:213:38
#5 0x7f0a3c269a4f in std::__1::enable_if<0 == std::tuple_size<std::__1::tuple<>>::value, void>::type ap::Callback::Invoker_CopiedArgs<engine::store::filter::indexer::IFilterGlobal::closeWordDb(...
...
0x7f09d1aee800 is located 0 bytes inside of 536870920-byte region [0x7f09d1aee800,0x7f09f1aee808)
allocated by thread T0 here:
#0 0x558dace4eefd in operator new[](unsigned long) (dist/server/nodetest+0x231efd)
#1 0x7f0a3c5d42b2 in ap::memory::constant_mbr::_M_new_buffer(unsigned long) packages/server/engine-core/apLib/memory/pmr.hpp:151:28
#2 0x7f0a3c5d42b2 in ap::memory::constant_mbr::do_allocate(unsigned long, unsigned long) packages/server/engine-core/apLib/memory/pmr.hpp:115:13
#3 0x7f0a3c50612c in boost::container::pmr::memory_resource::allocate(unsigned long, unsigned long) build/vcpkg_installed/x64-linux-clang-rocketride/include/boost/container/pmr/memory_resource....
...
SUMMARY: AddressSanitizer: alloc-dealloc-mismatch (dist/server/nodetest+0x23264d) in operator delete(void*)
Group 5: ASan alloc-dealloc-mismatch, operator new vs free (report.30951, aptest)
==30951==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new vs free) on 0x503000057a00
#0 0x55bd9daf8696 in free (dist/server/aptest+0x606696)
#1 0x7f854780bbdb in std::range_error::~range_error() (dist/server/lib/libc++abi.so.1+0x27bdb)
#2 0x7f854780df63 in __cxa_end_catch (dist/server/lib/libc++abi.so.1+0x29f63)
#3 0x55bd9de5f426 in auto ap::error::call<ap::Callback::Invoker_CopiedArgs<CATCH2_INTERNAL_TEST_49()::$_8, void, ap::string::Str<char, std::__1::char_traits<char>, std::__1::allocator<char>>>::...
#4 0x55bd9de5f426 in ap::Error ap::error::callCheck<ap::Callback::Invoker_CopiedArgs<CATCH2_INTERNAL_TEST_49()::$_8, void, ap::string::Str<char, std::__1::char_traits<char>, std::__1::allocator...
#5 0x55bd9de5f426 in ap::Callback::Invoker_CopiedArgs<CATCH2_INTERNAL_TEST_49()::$_8, void, ap::string::Str<char, std::__1::char_traits<char>, std::__1::allocator<char>>>::invoke(ap::Location) ...
...
0x503000057a00 is located 0 bytes inside of 29-byte region [0x503000057a00,0x503000057a1d)
allocated by thread T0 here:
#0 0x55bd9db35c8d in operator new(unsigned long) (dist/server/aptest+0x643c8d)
#1 0x7f854787159f in std::runtime_error::runtime_error(char const*) (dist/server/lib/libc++.so.1+0x5259f)
#2 0x55bd9db4eefc in CATCH2_INTERNAL_TEST_49() packages/server/engine-core/test/async/Callback.cpp:125:27
#3 0x55bd9ee3f3ed in Catch::RunContext::runCurrentTest() (dist/server/aptest+0x194d3ed)
SUMMARY: AddressSanitizer: alloc-dealloc-mismatch (dist/server/aptest+0x606696) in free
Activity
github-actions commentedon Nov 19, 2026
🤖 Internal: Discord sync marker
Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.
fix(server): call factory methods at the root interface's type
fix(server): keep the test program path alive while Catch2 runs
fix(server): free word DB pool chunks with delete[]
fix(server): fix the three engine errors the sanitizer job found (#3373)