Repository navigation
fix(server): claim the GNU C++ runtime at engine startup - #2720
Conversation
A task running parse beside a node that imports torch died at start with SIGSEGV at 0x20 in libstdc++: Tika's prebuilt lib7-Zip-JBinding.so carries a statically linked GCC libstdc++ whose GNU-unique locale facet ids bind process-wide to the first definer, so loaded into a libc++ engine it renumbered the system runtime's facet table and the next std::ostream to format a number reached a facet that was not there. plat::lin::init() now claims libstdc++.so.6 with RTLD_GLOBAL, beside the Crashpad registration and for the same reason. One runtime owns the ids and a bundled copy arriving later is inert. Fails open where the library is absent, since a host without it cannot load GCC-built code either. Linux only. Costs 0.86 MB resident, shared and file-backed, and 0.42 ms once per process. Fixes #2676 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
…binding #2720 closes the GNU-unique collision in the engine, so the probes target that: libstdc++.so.6 is mapped and in the global scope before any Python runs, and the jar's prebuilt 7-Zip-JBinding then a GCC-built std::ostream user survives in the engine. A precondition test keeps the jar's binding honest: once upstream stops exporting a bundled libstdc++, it says the probes can go. No torch and no compiler needed. Refs #2676
A task running parse beside a node that imports torch died at start with SIGSEGV at 0x20 in libstdc++: Tika's prebuilt lib7-Zip-JBinding.so carries a statically linked GCC libstdc++ whose GNU-unique locale facet ids bind process-wide to the first definer, so loaded into a libc++ engine it renumbered the system runtime's facet table and the next std::ostream to format a number reached a facet that was not there. plat::lin::init() now claims libstdc++.so.6 with RTLD_GLOBAL, beside the Crashpad registration and for the same reason. One runtime owns the ids and a bundled copy arriving later is inert. Fails open where the library is absent, since a host without it cannot load GCC-built code either. Linux only. Costs 0.86 MB resident, shared and file-backed, and 0.42 ms once per process. Fixes #2676 (cherry picked from commit 074c316) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…binding #2720 closes the GNU-unique collision in the engine, so the probes target that: libstdc++.so.6 is mapped and in the global scope before any Python runs, and the jar's prebuilt 7-Zip-JBinding then a GCC-built std::ostream user survives in the engine. A precondition test keeps the jar's binding honest: once upstream stops exporting a bundled libstdc++, it says the probes can go. No torch and no compiler needed. Refs #2676
…libstdc++ claim (#2677) * fix(parse): bring RarSevenZipParser along from packages/tika; RAR extracted nothing since #2440 #2440 moved the Tika code from packages/tika into the parse node's Java library but left parsers/rar/RarSevenZipParser.java behind, in a module nothing builds any more. ConfigBuilder still removed Tika's junrar RarParser and added that class by name, so with the binding loaded RAR had no parser at all: Tika mapped application/x-rar-compressed to nothing and every RAR4/RAR5 archive came out empty, silently, on every platform. The class moves to the node library; the Java-side probe now maps RAR to RarSevenZipParser and a RAR5 archive yields its member text. Refs #2676 * test(parse): RAR5 archive end to end through 7-Zip-JBinding webhook -> parse -> response_text with a one-file RAR5 archive built in the test (stored, no compression, standard library only); the member's text must be in the response. Exercises SevenZip.initSevenZipFromPlatformJAR() and RarSevenZipParser, which the ctypes probes and a text/plain pipeline cannot: TikaApi.init() swallows a load failure and parse carries on with RAR5 off. Objects need objinfo["size"]: parse skips a size-0 object as empty, so the pipeline test sets it too. CodeRabbit on #2677. Refs #2678 * fix(parse): swap in RarSevenZipParser only when the class resolves ConfigBuilder trusted the class name: when #2440 left RarSevenZipParser out of the jar, the swap removed Tika's RarParser and installed nothing, and RAR extracted nothing without a word. Check Class.forName as well as isSevenZipReady(), log which one is missing, and keep junrar otherwise. JUnit pins that the class is in the jar. Refs #2678 * test(parse): probe the engine's libstdc++ claim instead of a rebuilt binding #2720 closes the GNU-unique collision in the engine, so the probes target that: libstdc++.so.6 is mapped and in the global scope before any Python runs, and the jar's prebuilt 7-Zip-JBinding then a GCC-built std::ostream user survives in the engine. A precondition test keeps the jar's binding honest: once upstream stops exporting a bundled libstdc++, it says the probes can go. No torch and no compiler needed. Refs #2676 * chore: changelog and NOTICE for the RAR fix and the runtime claim; drop the packages/tika labeler entry The directory is empty since RarSevenZipParser moved; the labeler glob pointed at nothing. Refs #2678, #2676, #2720 * test(parse): skip the RAR5 end-to-end test on Apple Silicon sevenzipjbinding-all-platforms 16.02-2.01 ships a Mac-x86_64 binding only; on an arm64 JVM it does not load, TikaApi.init() turns RAR5 off and ConfigBuilder keeps junrar, so the test cannot pass there until the bump to 23.01-2.2 (#2734), whose Mac dylib is universal. Refs #2678, #2734 * build(docker): run the image's ldd check over the Linux natives inside the jars too The JVM loads them in a task, and a jar bump can bring a build that needs a library the image lacks. Extract the x86-64 glibc .so entries of java/lib/*.jar with the engine's Python, run the same ldd sweep, drop them. ldd proves that dependencies resolve, not that symbols do not collide (#2676 had clean dependencies); that stays a separate check. Refs #2677
…org#2720) A task running parse beside a node that imports torch died at start with SIGSEGV at 0x20 in libstdc++: Tika's prebuilt lib7-Zip-JBinding.so carries a statically linked GCC libstdc++ whose GNU-unique locale facet ids bind process-wide to the first definer, so loaded into a libc++ engine it renumbered the system runtime's facet table and the next std::ostream to format a number reached a facet that was not there. plat::lin::init() now claims libstdc++.so.6 with RTLD_GLOBAL, beside the Crashpad registration and for the same reason. One runtime owns the ids and a bundled copy arriving later is inert. Fails open where the library is absent, since a host without it cannot load GCC-built code either. Linux only. Costs 0.86 MB resident, shared and file-backed, and 0.42 ms once per process. Fixes rocketride-org#2676 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… the engine's libstdc++ claim (rocketride-org#2677) * fix(parse): bring RarSevenZipParser along from packages/tika; RAR extracted nothing since rocketride-org#2440 rocketride-org#2440 moved the Tika code from packages/tika into the parse node's Java library but left parsers/rar/RarSevenZipParser.java behind, in a module nothing builds any more. ConfigBuilder still removed Tika's junrar RarParser and added that class by name, so with the binding loaded RAR had no parser at all: Tika mapped application/x-rar-compressed to nothing and every RAR4/RAR5 archive came out empty, silently, on every platform. The class moves to the node library; the Java-side probe now maps RAR to RarSevenZipParser and a RAR5 archive yields its member text. Refs rocketride-org#2676 * test(parse): RAR5 archive end to end through 7-Zip-JBinding webhook -> parse -> response_text with a one-file RAR5 archive built in the test (stored, no compression, standard library only); the member's text must be in the response. Exercises SevenZip.initSevenZipFromPlatformJAR() and RarSevenZipParser, which the ctypes probes and a text/plain pipeline cannot: TikaApi.init() swallows a load failure and parse carries on with RAR5 off. Objects need objinfo["size"]: parse skips a size-0 object as empty, so the pipeline test sets it too. CodeRabbit on rocketride-org#2677. Refs rocketride-org#2678 * fix(parse): swap in RarSevenZipParser only when the class resolves ConfigBuilder trusted the class name: when rocketride-org#2440 left RarSevenZipParser out of the jar, the swap removed Tika's RarParser and installed nothing, and RAR extracted nothing without a word. Check Class.forName as well as isSevenZipReady(), log which one is missing, and keep junrar otherwise. JUnit pins that the class is in the jar. Refs rocketride-org#2678 * test(parse): probe the engine's libstdc++ claim instead of a rebuilt binding rocketride-org#2720 closes the GNU-unique collision in the engine, so the probes target that: libstdc++.so.6 is mapped and in the global scope before any Python runs, and the jar's prebuilt 7-Zip-JBinding then a GCC-built std::ostream user survives in the engine. A precondition test keeps the jar's binding honest: once upstream stops exporting a bundled libstdc++, it says the probes can go. No torch and no compiler needed. Refs rocketride-org#2676 * chore: changelog and NOTICE for the RAR fix and the runtime claim; drop the packages/tika labeler entry The directory is empty since RarSevenZipParser moved; the labeler glob pointed at nothing. Refs rocketride-org#2678, rocketride-org#2676, rocketride-org#2720 * test(parse): skip the RAR5 end-to-end test on Apple Silicon sevenzipjbinding-all-platforms 16.02-2.01 ships a Mac-x86_64 binding only; on an arm64 JVM it does not load, TikaApi.init() turns RAR5 off and ConfigBuilder keeps junrar, so the test cannot pass there until the bump to 23.01-2.2 (rocketride-org#2734), whose Mac dylib is universal. Refs rocketride-org#2678, rocketride-org#2734 * build(docker): run the image's ldd check over the Linux natives inside the jars too The JVM loads them in a task, and a jar bump can bring a build that needs a library the image lacks. Extract the x86-64 glibc .so entries of java/lib/*.jar with the engine's Python, run the same ldd sweep, drop them. ldd proves that dependencies resolve, not that symbols do not collide (rocketride-org#2676 had clean dependencies); that stays a separate check. Refs rocketride-org#2677
Summary
A task that runs
parseand then any node importing torch dies at start withSIGSEGV at
0x20in libstdc++ — mechanism, dumps and repro in #2676.The engine now claims the system libstdc++ while the process is still coming up, in
plat::lin::init()beside the Crashpad registration, so the runtime that owns thelocale facet ids is the real one. A JNI library carrying its own statically linked
copy — Tika's prebuilt
lib7-Zip-JBinding.sois the one we hit — then loads withoutconsequence instead of renumbering the facet table under the system runtime's feet.
Every executable picks this up through
ap::init().Linux only; no GNU-unique binding exists on macOS or Windows.
The cost: 0.86 MB resident (file-backed, shared), 0.42 ms at startup, once per
process. Nothing is interposed and no call path changes.
Re-implements the runtime half of #2677. That PR fixes the same crash by building
the binding from source so no second libstdc++ exists in the process, plus a
Python-side preload in
ai/node.py. This does it once, in the engine, and leaves thebinding alone. #2677 keeps everything else it carries — see the comment there.
Type
fix
Testing
WSL Ubuntu 22.04,
server:build-core. A/B on one dist withwebhook -> parse -> general_text -> embedding_transformerand one document, thisfile the only difference between runs:
exit=0RuntimeError: Received terminal system signalexit=0, no fatality/proc/<task pid>/mapsin the surviving runs carries all of it at once:libjvm.so,lib7-Zip-JBinding.so,libc++.so.1,libstdc++.so.6,libtorch.so,libtorch_cpu.so,libtorch_cuda.so,libtorch_python.so.Isolated in a subprocess, the jar's binding followed by a
longthroughstd::ostreamis SIGSEGV on both the 16.02-2.01 and 23.01-2.2 prebuilts, and cleanwith libstdc++ claimed first.
aptest99 cases / 40,742,613 assertions andengtest31 cases pass on the rebuiltengine. No regression test added yet.
Linked Issue
Fixes #2676 — Linux task dies on start (SIGSEGV at 0x20 in libstdc++) when a pipeline
runs
parsefollowed by General Text or an embedding node.Not fixed here, and still owned by #2677:
RarSevenZipParserisabsent from the built parse jar while
ConfigBuilderinstalls it by name.library on macOS and Windows.
🤖 Generated with Claude Code
Summary by CodeRabbit