Skip to content

Latest commit

 

History

History
300 lines (237 loc) · 13.3 KB

reference.md

File metadata and controls

300 lines (237 loc) · 13.3 KB

Build System Reference

How to run builds: commands, modules, output layout, CLI flags, and the C++ compiler toolchain. ./builder is a declarative, modular build system for the RocketRide monorepo.

To write build tasks — a package's scripts/tasks.js, control-flow helpers, deduplication, state, patterns — see Build System Authoring.

Primary Command: ./builder build

The recommended way to build the project is:

./builder build

This configures the environment, resolves dependencies, and builds all modules. Use ./builder build --sequential if parallel builds cause resource issues.


User Reference: Commands, Modules, and Output

Per-module builds

# Windows
.\builder <module>:<command>

# macOS/Linux
./builder <module>:<command>

Build commands

Command Description
<module>:build Full build with all dependencies
<module>:compile Quick compile (skip setup if already done)
<module>:clean Remove build artifacts
<module>:test Run tests

Not all modules support all commands. Run ./builder --help for the full list.

Modules reference

Module Description Commands
ai AI/ML modules build, clean, test
aparavi-ui Aparavi AQL chat application build, clean, dev
builder Build system maintenance inject, update
chat-ui Chat web interface build, clean, dev
check-externals 3rd-party interface contract test framework run, test
client-mcp MCP Protocol client build, clean, test
client-python Python SDK build, clean, test
client-typescript TypeScript/JavaScript SDK build, check, clean, freeze, regen, test
docs Documentation site build, check, clean, dev, export, serve, test
dropper-ui File drop web interface build, clean, dev
events-ui Event monitor application build, clean, dev
explorer-ui File explorer application build, clean
hash C++ filter fingerprinting document content, for deduplication build
hello-ui RocketRide Hello — OSS landing application build, clean, dev
indexer C++ filter building the inverted word index for full-text search build
java JDK, JRE, and Maven (auto-installed for Tika) setup-jdk, setup-jre, setup-maven
mcp-widgets MCP Apps widgets (embedded UI served by the ai MCP module) build, clean, test
models LLM model sync stamp-locals, update
monitor-ui Server monitor web interface build, clean
nodes Pipeline nodes build, clean, test, test-contracts, test-full
parse C++ and Java filter extracting content from documents (Tika) build
profiler-ui cProfile process profiler build, clean
rocket-ui Data toolchain interface application build, clean, dev
server C++ engine (downloads pre-built first, or compile from source) build, compile, clean, test, build-all, clean-all, configure-cmake, dev, package, run, setup-test-deps
shared RocketRide shared source library check-gallery-tokens, gen-gallery-tokens, test
shell Shell platform (host + frozen API surface) build, check, clean, dev, freeze, regen, regen-derived, test
sql-ui SQL Explorer application build, clean, dev
test-ui Test UI application build, clean, dev
ui All UI applications build, clean, register
vcpkg C++ package manager (auto-installed for server build) bootstrap, clone
vscode VSCode extension build, compile, clean
world-ui Hello World demo application build, clean

Examples

./builder build
./builder server:build
./builder client-typescript:build client-python:build client-mcp:build
./builder chat-ui:build dropper-ui:build
./builder vscode:build
./builder clean
./builder server:clean nodes:clean
./builder --help

Building a C++ node

Each C++ node (hash, indexer, parse) is its own CMake project with its own build action, for a change-build-test cycle on one node:

./builder hash:build

It takes the engine the same way server:build does — a prebuilt download when the local packages/server sources match a release, a source compile otherwise — and links the node against the engine in dist/server, never a build tree.

A node also links the native packages the engine was built with. It has no vcpkg manifest of its own, so it installs from the engine's (packages/server/vcpkg.json) into the same build/vcpkg_installed the engine uses, and vcpkg does that while the node configures. Nothing has to be installed by hand — but on a tree whose engine was downloaded, the first node build pays for the packages, which is long when the binary cache has nothing to offer.

Changing anything under packages/server gives the engine a new build hash, and since a node's library records the engine it was built against, every node rebuilds with it. The reverse is not true: changing one node rebuilds that node only.

Build output layout

Directory Contents
build/ Temporary build artifacts
dist/ Final distributable outputs
dist/server/ Engine executable and runtime
dist/server/test-logs/ Output of the test servers, one <suite>-server.log per suite, written with --verbose and always in CI, traced at debugOut unless --trace is given. Each run replaces the file of the previous one. When tests fail, CI prints these files into the job log (step "Show test server logs"). Not packaged. Keep --trace=debugProtocol out of CI: it logs raw messages, auth keys included
dist/clients/ Client library packages
dist/vscode/ VSCode extension (.vsix)
dist/examples/ Example applications

CLI Usage

# Run a single action
./builder my-package:build

# Run multiple actions
./builder server:build nodes:build ai:build

# Run all builds (global command)
./builder build

# Run with options
./builder my-package:test --force           # Force rebuild (ignore cache/state)
./builder my-package:test --verbose         # Detailed output
./builder my-package:test --pytest="-s -v"  # Pass pytest args
./builder build --sequential                # Run modules sequentially
./builder build --autoinstall               # Install missing tools automatically
./builder build --arch=arm                  # Target architecture (macOS cross-compile)
./builder build --suffix=cpp-nodes          # Take the server from the server-v<version>-cpp-nodes release

# Show help
./builder --help

# List all actions (including internal)
./builder --list-actions

# Show dependency diagram for an action
./builder my-package:test --list-deps

C++ build type: --cmake-config

--cmake-config=<type> sets the CMake build type (CMAKE_BUILD_TYPE) of the engine, its test programs (aptest, engtest, nodetest) and the C++ nodes.

Value What it builds
Release Default. Optimized, with symbols
RelWithDebInfo For C++ debugging on Windows: unoptimized, full PDBs
Sanitize Linux only. ASan and UBSan in one build; builder rejects it on other platforms
  • builder does not remember the type between calls. Every call that builds or tests C++ code repeats it — the build call and the test call. A call without it asks for Release: it stops at the engine check below, or rebuilds parts of the tree as Release.
  • The engine's "already built" check compares the sources and the type. When dist/server holds an engine built from the same sources as another type, builder stops and names both ways out: pass that type with --cmake-config to use the engine, or rerun with --force to rebuild it as the requested type. After a source change the build call rebuilds the engine as the requested type by itself.
  • Only a Release build looks up and downloads the published engines: they are all Release, so any other type compiles. CI passes --nodownload as well, so the job makes no network lookup even without this rule.
  • server:package packs only a Release engine and stops on any other type, so a Sanitize engine is never packaged.

Sanitizer build (Sanitize)

A Sanitize engine is for tests only: never deploy it, and never run untrusted pipelines on it.

A Sanitize build needs these variables in every call, the build call too (the build already runs the new engine, for example to set up pip). The report directory must exist and its path must be absolute:

mkdir -p "$PWD/sanitizer-reports"
export ASAN_OPTIONS="halt_on_error=0:detect_leaks=0:log_path=$PWD/sanitizer-reports/report"
export UBSAN_OPTIONS="halt_on_error=0:print_stacktrace=1:report_error_type=1:log_path=$PWD/sanitizer-reports/report"
export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-18/bin/llvm-symbolizer
export RR_SUBPROCESS_ENV="ASAN_* UBSAN_*"

./builder server:build --cmake-config=Sanitize --nodownload
./builder server:test --cmake-config=Sanitize --nodownload
./builder nodes:test --cmake-config=Sanitize --nodownload

node scripts/ci/sanitizer-report.mjs "$PWD/sanitizer-reports"
  • halt_on_error=0: a program keeps running after a report, so one run shows every error.
  • log_path: ASan and UBSan share one runtime, so both options name the same prefix, and each process that reports writes all its reports to <log_path>.<pid>.
  • ASAN_SYMBOLIZER_PATH: the path to llvm-symbolizer, which adds file and line to the reports. The apt llvm-18 package puts no unversioned llvm-symbolizer on PATH.
  • RR_SUBPROCESS_ENV: the engine runs each task in its own engine process and passes it only an allowlist of variables. A trailing * passes every variable with that prefix, so these processes keep the sanitizer options.
  • The exit code stays 0 after a report under halt_on_error=0. node scripts/ci/sanitizer-report.mjs <dir> counts and groups the reports in the directory and exits 1 when there is any.

CI runs this build and these tests in a separate job, on pull requests that change C++ files and carry the ci:sanitize label (add the label, then push), and on manual runs. The job is not part of CI OK yet.


Compiler toolchain (C++ engine)

The engine builds with clang 16–18 + libc++ (it doesn't compile with clang ≥ 19). server:setup-tools (run by scripts/compiler-unix.sh) provisions it on Fedora, Ubuntu, and macOS:

  • macOS — uses the system Apple Clang (Xcode Command Line Tools) as-is.
  • Linux:
    1. if bare clang is 16–18 with a working libc++ and ld.lld (Crashpad links with -fuse-ld=lld) → used as-is;
    2. otherwise, by default, a self-contained LLVM 18 toolchain (latest 18.x clang+llvm release, bundles its own libc++) is unpacked into ~/toolchains/llvm-18 — user-local, no root, system compiler untouched. The builder points the build at it via PATH/CC/CXX/LD_LIBRARY_PATH.
  • dump_syms (crash-symbol generator) is fetched into ~/toolchains/bin (root-free) and added to the build PATH.

--autoinstall vs --autoinstall --system-compiler

Both install the non-compiler build dependencies via apt/dnf (needs root). They differ only in where the C++ compiler goes:

  • --autoinstall (default) — keeps clang local: uses the system clang if it's already 16–18, else the ~/toolchains/llvm-18 toolchain. Never touches the system compiler.
  • --autoinstall --system-compiler — installs a compatible clang system-wide via the package manager and repoints the default clang++:
    • apt — the distro's default clang if it's 16–18 (e.g. Ubuntu 24.04), else clang-18 from the distro archive or apt.llvm.org (e.g. Ubuntu 22.04) + update-alternatives.
    • dnf — only if the default clang is 16–18; Fedora's clang-22 has no matching libc++, so it falls back to the ~/toolchains toolchain.
    • Needs root. If root isn't available (or sudo can't authenticate non-interactively) it errors and asks you to run with sudo or drop --system-compiler — it does not silently fall back to the tarball.

Install policy & ownership

Root-free ~/toolchains downloads (the LLVM toolchain, dump_syms) install automatically, with or without --autoinstall. Distro packages install only under --autoinstall. Anything placed in a user home (~/toolchains) is installed as the invoking user — never as root — even when the script runs under sudo.

Compatibility (glibc) — why CI builds on the oldest LTS

A binary's glibc floor is set by the build host, not the compiler. Building on Ubuntu 22.04 (glibc 2.35) with either the tarball or apt.llvm.org clang-18 produces a binary that runs on 22.04 and newer; building on 24.04 (glibc 2.39) would not run on 22.04. So release/CI builds run on the oldest supported LTS (currently ubuntu-22.04), and --system-compiler there installs clang-18 via apt.llvm.org (faster than the tarball, same glibc floor). The clang version never changes the floor.

Building manually with clang

To compile outside the builder (e.g. a raw cmake/ninja invocation), source the env helper to point CC/CXX/PATH/LD_LIBRARY_PATH at the same toolchain:

. scripts/setenvs.sh

It uses ~/toolchains/llvm-18 if present, otherwise the system clang, and always adds ~/toolchains/bin (for dump_syms) to PATH.