Openresearchtools-Engine

Early-stage repository: this repo is still evolving quickly. The goal is to deliver a single, embeddable, local AI runtime that covers the common building blocks you usually end up wiring together from multiple projects.

Openresearchtools-Engine is a local AI runtime primarily based on Llama.CPP, that you can embed directly into an application.

Available now to try!!

Experimental cluster controller for orchestrating single-node and multi-node runtimes across multiple graphics cards on the same and/or different machines.

Experimental development release. Expect instability and bugs, especially on clusters larger than 2 nodes.

A few examples using openresearchtools-engine:

Live and file based transcription and diarization: Transcribe Offline

PDF to Markdown VLM based Converter and Editor: PDF Markdown Studio

Implementation goals

It aims to unify chat, vision, embeddings, reranking, audio transcription/diarization, and PDF-to-Markdown in one native stack (Rust + C++), so you don't have to glue together separate runtimes for each task.

Collapse multi-service pipelines into one embeddable engine with a consistent API surface.
Keep deployment and runtime paths lightweight (avoid heavy Python-first stacks in the inference/runtime layer).
Support true in-process integration, not just process-spawn or HTTP-only approaches.
Make inference behavior controllable in production with explicit GPU/CPU selection, offload controls, and multi-GPU knobs.
Handle both "easy" and "hard" PDFs by supporting a fast digital-PDF path and a VLM-based conversion path for difficult layouts.

What's implemented so far

In-process llama-server-bridge (no HTTP requirement for app embedding).
Chat, VLM, embeddings, reranking.
Audio session runtime with Whisper transcription, native Sortformer diarization, native Voxtral realtime transcription, and Rust-side speaker/text assembly.
PDF -> Markdown:
- fast native digital PDF path (pdf.dll)
- VLM document-conversion path (pdfvlm.dll)
GPU/CPU controls (single GPU, multi-GPU split, offload knobs).

Project status and affiliation

Some components are experimental, especially the diarization path.
This project is an independent engineering effort and is not affiliated with, sponsored by, or endorsed by the upstream projects it builds on.
References to third-party project names are for compatibility and attribution only.
All third-party names and marks (including llama.cpp, whisper.cpp, NVIDIA NeMo, Docling, PDFium, FFmpeg, and Qwen/Qwen3-VL) remain the property of their respective owners.

License and notices

Openresearchtools-engine source code is licensed under the MIT License; third-party dependencies and bundled components remain licensed under their respective original licenses.

For full notices, license types, and source provenance:

Packaged app bundles ship these as root files:

LICENSES.md
Third-Party-Notices.md

How to embed it

example-cli.exe is an example wrapper showing how to call functions.
example-cli is not the embedding boundary; treat it as a reference CLI only.
The controller app ships as Engine.exe on Windows and as Engine.dmg containing Engine.app on macOS.
Production embedding target is native binaries:
- llama-server-bridge.dll
- pdf.dll
- pdfvlm.dll

Engine app

Engine.exe / Engine.app is the desktop controller for ENGINE’s local and multi-node runtime stack.

It lets you discover and pair machines, see which devices and models are available on each node, create named runtimes on any connected system, and keep one stable control point even when the actual runtime is running somewhere else in the cluster.

Engine is based on the llama.cpp runtime family together with ENGINE’s own native bridge and clustering layers. That includes remote runtime ownership, cross-node workers, model transfer between nodes, named-instance HTTP routing, native audio and transcription paths, and mixed-device orchestration across CUDA, Vulkan, and Metal systems.

Cluster behavior:

Any connected node can act as the initiator, the runtime owner, or a worker.
A runtime can be started from one machine while the primary GPU and actual owner live on another machine.
Cluster instance calls still resolve by instance owner, so the local API/bridge entrypoint can forward to the real runtime host.
The cluster is backend-agnostic across nodes: CUDA, Vulkan, and Metal nodes can participate in the same cluster, with each machine using its native backend locally.

Transport guidance:

Direct Thunderbolt or USB4 link-local connections are preferred where available because they provide the best practical speeds for model transfer and cross-node traffic.
Normal private LAN still works, but large model copies and RPC-heavy workloads benefit materially from the higher direct-link bandwidth.

CPU guidance:

CPU execution is available by enabling CPU devices in Settings.
For clustered tensor-model workloads it is generally not recommended, is often untested, and is dramatically slower than GPU-backed execution.

If you want to report a bug or request a feature, use GitHub issues in this repository:

https://github.com/openresearchtools/engine/issues

Detailed DLL docs

For function-by-function embedding docs (minimal calls, GPU selection, and full parameter coverage), see:

docs/README.md

Project layout

bridge/ native in-process bridge for llama runtime APIs.
engine/ Rust CLI wrapper crate (builds example-cli).
clusterui/ controller app crate (builds Engine).
pdf/ fast PDF extraction module.
pdfvlm/ PDF->image->VLM->Markdown module.
diarize/ patch assets for integrated audio stack.
build/ fetch/build scripts.
third_party/ runtime sources/binaries/licenses.

CLI examples

Device enumeration

Start here to confirm what the runtime can see (CPU/GPU devices) before you tune offload or multi-GPU splits.

# Device enumeration
example-cli.exe list-devices
# Equivalent bridge form:
example-cli.exe bridge list-devices

Common runtime flags (all bridge commands)

These runtime controls are available across chat, vlm, audio, embed, rerank, and pdfvlm:

--gpu <int> single-device shortcut (device index from list-devices)
--devices <csv> (for example 0, 1, 0,1, or none for CPU-only)
--main-gpu <int> index inside selected devices (default -1, auto in single-device mode)
--n-gpu-layers <int> (-1 = full offload where supported)
--mmproj-use-gpu <-1|0|1> (-1 auto, 1 GPU, 0 CPU)
--split-mode <none|layer|row> and --tensor-split <csv> (multi-GPU split)
--threads <int> and --threads-batch <int> (CPU compute thread controls)

Reasoning controls for chat, vlm, and pdfvlm:

If --reasoning is omitted, no reasoning flags are sent and the model/runtime decides what to do.
--reasoning off automatically sends reasoning_budget 0.
--reasoning on and --reasoning auto automatically send reasoning_budget -1 unless you override it.
If --reasoning is set and --reasoning-format is omitted, the runtime sends deepseek.
--reasoning-format none keeps visible thinking in the main output.
--reasoning-format deepseek asks the server/runtime to parse reasoning separately where supported.

Default behavior:

Windows/Linux: if no GPU is specified (--gpu/--devices omitted), runtime is CPU-only
macOS (Apple Silicon / Metal builds): if no GPU is specified, runtime selects the first GPU
if --gpu N is specified, defaults become single-device on that GPU with full offload
by default there is no tensor split; split only happens if --split-mode / --tensor-split is passed
--devices none forces CPU-only on all platforms
explicit flags always override defaults

Minimal runtime contract (DLL/embedded callers):

pass only required task/model inputs plus optional device selector (--gpu N or --devices ...) when you want simple routing
if a device selector is passed, model + KV cache + mmproj (auto mode) run on that selected device by default
audio follows the same selected runtime device unless explicitly overridden with --whisper-gpu-device / --diarization-device
all advanced flags are still supported; passing them overrides these defaults

Chat

This runs a direct prompt-based chat request with full GPU offload on a single GPU. It's a good baseline test for "does the model run and is the GPU config correct?"

# Chat with prompt (single GPU, full offload)
example-cli.exe chat `
  --model ".\models\model.gguf" `
  --prompt "Summarize key findings in 5 bullets." `
  --n-gpu-layers -1 `
  --main-gpu 0 `
  --n-ctx 50000 `
  --n-batch 1024 `
  --n-ubatch 1024 `
  --n-parallel 1 `
  --n-predict 10000

# Chat with reasoning disabled
example-cli.exe chat `
  --model ".\models\model.gguf" `
  --gpu 0 `
  --reasoning off `
  --prompt "What is 17 multiplied by 19? Reply with only the final answer." `
  --out ".\answer_reasoning_off.txt"

# Chat with visible reasoning output
example-cli.exe chat `
  --model ".\models\model.gguf" `
  --gpu 0 `
  --reasoning on `
  --reasoning-format none `
  --prompt "What is 17 multiplied by 19? Reply with only the final answer." `
  --out ".\answer_reasoning_on.txt"

If you already have a Markdown file you want summarized, you can pass it directly. When you omit --prompt, the CLI uses a default summary prompt.

# Chat from markdown only (uses default summary prompt)
example-cli.exe chat `
  --model ".\models\model.gguf" `
  --markdown ".\input.md" `
  --n-gpu-layers -1 `
  --main-gpu 0

To do targeted extraction or structured analysis, combine a prompt and a Markdown context file.

# Chat with both prompt and markdown context
example-cli.exe chat `
  --model ".\models\model.gguf" `
  --prompt "Extract all statistical tests and p-values." `
  --markdown ".\input.md" `
  --n-gpu-layers -1 `
  --main-gpu 0

Vision and VLM

Use vlm when you want to run a vision-language model over an image (including page renders) and produce Markdown or a prompt-driven description.

This example runs "image -> Markdown" conversion with the default extraction prompt.

# VLM markdown conversion (default prompt = markdown extraction)
example-cli.exe vlm `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --image ".\page.png" `
  --out ".\page.md" `
  --mmproj-use-gpu 1 `
  --n-gpu-layers -1 `
  --main-gpu 0

# VLM with reasoning disabled
example-cli.exe vlm `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --image ".\image.png" `
  --prompt "Describe the image briefly, then identify the main object shown." `
  --gpu 0 `
  --reasoning off `
  --out ".\image_reasoning_off.txt"

# VLM with visible reasoning output
example-cli.exe vlm `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --image ".\image.png" `
  --prompt "Describe the image briefly, then identify the main object shown." `
  --gpu 0 `
  --reasoning on `
  --reasoning-format none `
  --out ".\image_reasoning_on.txt"

If you want the model to answer a specific question about an image, provide your own prompt.

# VLM image chat (set your own prompt)
example-cli.exe vlm `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --image ".\image.png" `
  --prompt "Describe this image and summarize key elements." `
  --mmproj-use-gpu 0 `
  --n-gpu-layers -1 `
  --main-gpu 0

--mmproj-use-gpu controls where the vision projector runs:

-1 (default): auto-follow selected runtime device
1: run mmproj on GPU
0: run mmproj on CPU

Multi-GPU split

If a model is too large for one GPU, you can split across multiple devices. This example shows a layer split with an explicit tensor split ratio.

# Multi-GPU split
example-cli.exe chat `
  --model ".\models\model.gguf" `
  --markdown ".\input.md" `
  --devices 0,1 `
  --split-mode layer `
  --tensor-split 0.6,0.4 `
  --n-gpu-layers -1 `
  --main-gpu 0

Audio: transcription with and without diarization

example-cli bridge audio-session is the preferred audio entrypoint for current work.

Use audio-session when you want:

file transcription only
file diarization only
file transcription + diarization
live/session-style PCM ingress
native realtime Voxtral transcription

example-cli audio remains available as the compatibility CLI route, but the current end-to-end session work is centered on audio-session.

Recommended `audio-session` commands

Whisper transcription only from file:

example-cli.exe bridge audio-session `
  --audio-file ".\sample.wav" `
  --whisper-hf-repo ggerganov/whisper.cpp `
  --whisper-hf-file ggml-large-v3-turbo.bin `
  --gpu 0 `
  --out ".\sample_transcript.txt"

Whisper + native Sortformer diarization from file:

example-cli.exe bridge audio-session `
  --audio-file ".\meeting.wav" `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-device Vulkan0 `
  --whisper-hf-repo ggerganov/whisper.cpp `
  --whisper-hf-file ggml-large-v3-turbo.bin `
  --gpu 0 `
  --out ".\meeting_diarized.md" `
  --timeline-json-out ".\meeting_timeline.json"

Diarization only from file:

example-cli.exe bridge audio-session `
  --audio-file ".\meeting.wav" `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-device Vulkan0 `
  --out ".\meeting_spans.md"

Native Voxtral transcription on the same continuous session path:

example-cli.exe bridge audio-session `
  --audio-file ".\meeting.wav" `
  --transcription-realtime-model ".\models\voxtral-mini-4b-realtime.gguf" `
  --transcription-device Vulkan0 `
  --out ".\meeting_voxtral.txt"

Native Voxtral + native Sortformer on the same continuous session path:

example-cli.exe bridge audio-session `
  --audio-file ".\meeting.wav" `
  --transcription-realtime-model ".\models\voxtral-mini-4b-realtime.gguf" `
  --transcription-device Vulkan0 `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-device Vulkan0 `
  --out ".\meeting_voxtral_diarized.md" `
  --timeline-json-out ".\meeting_voxtral_timeline.json"

Live PCM/stdin -> native Voxtral transcription:

Get-Content ".\meeting_mono16k_s16le.raw" -Encoding Byte | `
example-cli.exe bridge audio-session `
  --stdin-pcm-s16le `
  --stdin-chunk-frames 7680 `
  --session-sample-rate 16000 `
  --session-channels 1 `
  --transcription-realtime-model ".\models\voxtral-mini-4b-realtime.gguf" `
  --transcription-device Vulkan0 `
  --out ".\live_voxtral.txt"

Live PCM/stdin -> native Voxtral + native Sortformer:

Get-Content ".\meeting_mono16k_s16le.raw" -Encoding Byte | `
example-cli.exe bridge audio-session `
  --stdin-pcm-s16le `
  --stdin-chunk-frames 7680 `
  --session-sample-rate 16000 `
  --session-channels 1 `
  --transcription-realtime-model ".\models\voxtral-mini-4b-realtime.gguf" `
  --transcription-device Vulkan0 `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-device Vulkan0 `
  --out ".\live_voxtral_diarized.md"

Live PCM/stdin -> Whisper + native Sortformer (staged):

Get-Content ".\meeting_mono16k_s16le.raw" -Encoding Byte | `
example-cli.exe bridge audio-session `
  --stdin-pcm-s16le `
  --stdin-chunk-frames 7680 `
  --session-sample-rate 16000 `
  --session-channels 1 `
  --staged `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-device Vulkan0 `
  --whisper-hf-repo ggerganov/whisper.cpp `
  --whisper-hf-file ggml-large-v3-turbo.bin `
  --gpu 0 `
  --out ".\live_whisper_diarized.md"

For the full session ABI surface, rolling-write behavior, and more live PCM/stdin examples, see:

docs/bridge-audio-dll.md

Compatibility route: `example-cli audio`

example-cli audio remains available for compatibility and one-shot file-oriented runs, but new work should prefer example-cli bridge audio-session.

Command shape

You can invoke the audio pipeline in either of these forms:

example-cli audio ... or example-cli bridge audio ...
Audio modes are always executed in audio-only runtime mode. A text --model GGUF is not required.
If --model is provided on example-cli audio, it is ignored for compatibility.
If --gpu N is set and no explicit --whisper-gpu-device / --diarization-device is provided, both follow the selected runtime device.
If no device is set, audio follows the same defaults as the rest of the runtime (Windows/Linux CPU-only, macOS first GPU).

Output behavior:

--output-dir <dir> writes the final file (.srt for subtitle, .md for speech/transcript) into that directory.
If --output-dir is omitted, output defaults to the same directory as --audio-file.
The pipeline keeps only the final output artifact.
Audio input is normalized through FFmpeg conversion in RAM; supported input formats depend on the FFmpeg build you ship.

`--mode` (main mode)

Current options are:

subtitle
speech
transcript

`--custom` (mode-specific behavior)

--custom is interpreted differently depending on the chosen mode:

subtitle: default / auto or a positive number of seconds (float/int) to control windowing
speech: default / auto or a positive number of seconds (float/int) to control windowing
transcript: default / auto or a positive integer for a fixed speaker count

If --custom is omitted, the CLI uses default.

Whisper model source (required)

Whisper is required for audio processing. Provide exactly one of:

Local: --whisper-model (or --whisper-model-path)
Hugging Face: --whisper-hf-repo + --whisper-hf-file

Diarization sources and device (for `--mode transcript`)

When you want speaker-aware transcripts, provide diarization models in one of these ways:

Local GGUF file: --diarization-model-path <file.gguf>
Local directory: --diarization-models-dir <dir>

You can also optionally set:

--diarization-device <value> (defaults to auto)
--diarization-feed-ms <number> (external audio ingress cadence in milliseconds, for example 100, 470, 1000)

Advanced audio knobs (CLI flags or `--body-json`)

Advanced audio controls are available directly as example-cli.exe audio flags (including raw-bytes --audio-file runs), and can also be passed in request JSON via --body-json.

Whisper controls:

--whisper-threads, --whisper-processors, --whisper-max-len, --whisper-audio-ctx
--whisper-best-of, --whisper-beam-size, --whisper-temperature
--whisper-language, --whisper-prompt, --whisper-translate
--whisper-no-fallback, --whisper-suppress-nst
--whisper-no-gpu, --whisper-gpu-device, --whisper-flash-attn, --whisper-no-flash-attn
--whisper-offline
--whisper-word-time-offset-sec

Diarization controls:

--diarization-model-path, --diarization-models-dir
--diarization-backend (auto, native_cpp, realtime, sortformer)
--diarization-device
--diarization-feed-ms
--speaker-seg-max-gap-sec
--speaker-seg-max-words
--speaker-seg-max-duration-sec
--speaker-seg-split-on-hard-break, --speaker-seg-no-split-on-hard-break

Pipeline/runtime controls:

--audio-only (legacy compatibility flag; optional no-op)
--ffmpeg-convert, --no-ffmpeg-convert
--transcription-backend
--seconds-per-timeline-token, --source-audio-seconds

Custom Whisper model: timestamp alignment tuning (diarization)

If you use a custom Whisper model and diarized transcript speaker turns look shifted relative to words/subtitles, tune:

--whisper-word-time-offset-sec (primary alignment control)
--source-audio-seconds (optional timeline clamp)
--seconds-per-timeline-token (fallback timing when word timestamps are sparse)

Recommended tuning flow:

Start with --whisper-word-time-offset-sec 0.73 (default behavior).
Run a known audio sample in --mode transcript and inspect where speaker boundaries drift.
Increase offset if words appear too early; decrease offset if words appear too late.
Adjust in small steps (for example 0.05-0.15) until speaker turns and transcript timing match.
If needed, set --source-audio-seconds to the known audio duration to prevent end-of-file overshoot.

Example with a custom local Whisper model:

example-cli.exe audio `
  --audio-file ".\meeting.mp3" `
  --output-dir ".\outputs" `
  --mode transcript `
  --custom auto `
  --whisper-model ".\models\my-custom-whisper.bin" `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-feed-ms 470 `
  --whisper-word-time-offset-sec 0.85 `
  --source-audio-seconds 1032.4

Audio examples

This is a straightforward "speech mode" transcription run (no diarization). Use this when you just want clean text output and don't need speaker separation.

# speech mode, default custom
example-cli.exe audio `
  --audio-file ".\sample.mp3" `
  --output-dir ".\outputs" `
  --audio-format mp3 `
  --mode speech `
  --custom default `
  --whisper-model ".\models\whisper.bin"

This produces subtitle-style output, where you can control the window size via --custom (here, 4.5 seconds). It's useful when you want timestamps/segments rather than one continuous paragraph.

# subtitle mode, 4.5-second windowing via custom
example-cli.exe audio `
  --audio-file ".\sample.wav" `
  --output-dir ".\outputs" `
  --mode subtitle `
  --custom 4.5 `
  --whisper-model ".\models\whisper.bin"

This generates a speaker-aware transcript by enabling diarization. With --custom auto, the system estimates speaker count, and --diarization-device lets you choose where diarization runs (for example, CUDA, Vulkan, or auto).

# transcript mode, auto speaker count, local diarization models
example-cli.exe audio `
  --audio-file ".\meeting.mp3" `
  --output-dir ".\outputs" `
  --mode transcript `
  --custom auto `
  --whisper-model ".\models\whisper.bin" `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-feed-ms 470 `
  --diarization-device auto

Offline note: if you want to run diarization fully offline, point to a local GGUF file with --diarization-model-path, or place one inside --diarization-models-dir. The runtime does not fetch diarization models from Hugging Face anymore.

This example also runs a diarized transcript, but changes the ingress cadence for lower-latency updates while keeping the same native realtime backend.

# transcript mode, lower-latency feed cadence
example-cli.exe audio `
  --audio-file ".\meeting.mp3" `
  --output-dir ".\outputs" `
  --mode transcript `
  --custom auto `
  --whisper-hf-repo ggerganov/whisper.cpp `
  --whisper-hf-file ggml-tiny.en.bin `
  --diarization-model-path ".\models\sortformer.gguf" `
  --diarization-feed-ms 100 `
  --diarization-device cuda

If you prefer a JSON request payload, you can still pass the same advanced controls through --body-json.

# advanced audio knobs via body JSON
example-cli.exe audio --body-json ".\audio_request.json"

PDF to Markdown

For PDFs, you generally have two paths:

Use the fast digital extractor when the PDF has good text structure. (Important note, tables, formulas and any special layouts will be rendered in line, not great for complex tables, but extremely fast).
Use the VLM conversion when the PDF is scanned, layout-heavy, or loses structure in digital extraction.

Fast digital PDF conversion:

# Fast digital PDF conversion
example-cli.exe pdf extract --input ".\paper.pdf" --output ".\paper_fast.md" --overwrite

VLM PDF conversion (PDF -> render -> VLM -> Markdown). Choose the option that matches how you ship the PDFium runtime library:

# PDF VLM conversion
# Option A: pass library path each call
example-cli.exe pdfvlm `
  --pdf ".\paper.pdf" `
  --pdfium-lib ".\vendor\pdfium\pdfium.dll" `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --out ".\paper_vlm.md" `
  --threads 32 `
  --threads-batch 32 `
  --mmproj-use-gpu 1 `
  --n-gpu-layers -1 `
  --main-gpu 0

# Option B: bundled app - if PDFium is under vendor/pdfium next to engine(.exe), omit --pdfium-lib
example-cli.exe pdfvlm --pdf ".\paper.pdf" --model ".\models\vision.gguf" --mmproj ".\models\mmproj.gguf" --out ".\paper_vlm.md"

# Option C: set env var once (PDFIUM_DLL is still accepted for compatibility)
$env:PDFIUM_LIB=".\vendor\pdfium\pdfium.dll"
example-cli.exe pdfvlm --pdf ".\paper.pdf" --model ".\models\vision.gguf" --mmproj ".\models\mmproj.gguf" --out ".\paper_vlm.md"

# PDF VLM with reasoning disabled
example-cli.exe pdfvlm `
  --pdf ".\paper.pdf" `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --gpu 0 `
  --reasoning off `
  --out ".\paper_vlm_reasoning_off.md"

# PDF VLM with visible reasoning output
example-cli.exe pdfvlm `
  --pdf ".\paper.pdf" `
  --model ".\models\vision.gguf" `
  --mmproj ".\models\mmproj.gguf" `
  --gpu 0 `
  --reasoning on `
  --reasoning-format none `
  --out ".\paper_vlm_reasoning_on.md"

VLM model note (tested configuration)

For scientific PDF -> Markdown conversion, we tested the Qwen3-VL GGUF release: https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF/tree/main

In our testing, we got reasonably high quality results with:

Qwen3VL-8B-Instruct-Q8_0.gguf (fits a 16GB VRAM GPU)
mmproj-Qwen3VL-8B-Instruct-F16.gguf

One important caveat: on large, complex tables, the model can occasionally make structural mistakes (for example, attributing a number to the wrong row or the wrong column). If you plan to extract data from tables, it's strongly recommended to inspect the original PDF and the tables themselves before trusting downstream derived values.

We have no affiliation with the Qwen team. This note reflects internal evaluation of models that fit within a 16GB VRAM GPU.

Embeddings

embed is JSON-in/JSON-out.

--body-json accepts either:
- a JSON string payload, or
- a path to a JSON file
If --out is not set, response JSON is printed to console.
If --out is set, response JSON is written to that file.

Batching (important):

API-style batching: put multiple items in "input": [...] inside one JSON request.
Markdown batching mode: use --markdown with:
- --batch-size <N>
- --chunk-words <N>
- --batches <N>
--batch-size/--chunk-words/--batches apply to --markdown mode (not --body-json mode).

API-style batched request (single call, many input rows):

$payload = '{"input":["row 1 text","row 2 text","row 3 text","row 4 text"],"encoding_format":"float"}'

example-cli.exe embed `
  --model ".\models\embedding.gguf" `
  --body-json $payload `
  --out ".\embed_response.json"

Markdown batching example (multiple embedding calls generated by CLI):

example-cli.exe embed `
  --model ".\models\embedding.gguf" `
  --markdown ".\corpus.md" `
  --batch-size 64 `
  --chunk-words 300 `
  --batches 10 `
  --out ".\embed_batched_response.json"

Minimal inline payload (no temp file):

$payload = '{"input":["row one text","row two text","row three text"],"encoding_format":"float"}'

example-cli.exe embed `
  --model ".\models\embedding.gguf" `
  --body-json $payload

Save response to file:

$payload = '{"input":["row one text","row two text","row three text"],"encoding_format":"float"}'

example-cli.exe embed `
  --model ".\models\embedding.gguf" `
  --body-json $payload `
  --out ".\embed_response.json"

File-based payload (if you prefer files):

@'
{
  "input": ["a","b","c"],
  "encoding_format": "float"
}
'@ | Set-Content .\embed_request.json

example-cli.exe embed `
  --model ".\models\embedding.gguf" `
  --body-json ".\embed_request.json" `
  --out ".\embed_response.json"

Reranking

rerank is also JSON-in/JSON-out.

Request shape:

{
  "query": "text to match against",
  "documents": ["candidate 1", "candidate 2", "candidate 3"],
  "top_n": 2
}

Notes:

query is required.
documents is required (rerank needs candidates to score/sort).
top_n is optional.
--body-json accepts inline JSON string or a file path.
Without --out, response JSON prints to console.

What documents means:

documents is an array of plain text strings (sentences/paragraphs/chunks).
They are candidate texts that will be ranked by relevance to query.
documents is not a file-format field and not a file path list.
If your source is a file (PDF/MD/TXT), extract/split text first, then pass those text chunks in documents.

Example payload with real candidate texts:

{
  "query": "find statements about adverse effects",
  "documents": [
    "Mild headache was reported in 3% of patients.",
    "The model uses grouped-query attention and rotary embeddings.",
    "Nausea and dizziness were the most common side effects."
  ],
  "top_n": 2
}

Minimal inline payload (no temp file):

$payload = '{"query":"table extraction quality","documents":["doc1","doc2","doc3"],"top_n":2}'

example-cli.exe rerank `
  --model ".\models\reranker.gguf" `
  --body-json $payload

Save response to file:

$payload = '{"query":"find adverse effects","documents":["row A text","row B text","row C text"],"top_n":3}'

example-cli.exe rerank `
  --model ".\models\reranker.gguf" `
  --body-json $payload `
  --out ".\rerank_response.json"

File-based payload (if you prefer files):

@'
{
  "query": "find rows about adverse effects",
  "documents": [
    "document row A",
    "document row B",
    "document row C"
  ],
  "top_n": 3
}
'@ | Set-Content .\rerank_request.json

example-cli.exe rerank `
  --model ".\models\reranker.gguf" `
  --body-json ".\rerank_request.json" `
  --out ".\rerank_response.json"

With multi-GPU split:

$payload = '{"query":"find adverse effects","documents":["row A text","row B text","row C text"],"top_n":3}'

example-cli.exe rerank `
  --model ".\models\reranker.gguf" `
  --body-json $payload `
  --devices 0,1 `
  --split-mode layer `
  --tensor-split 0.6,0.4 `
  --n-gpu-layers -1

Build docs

Build/fetch instructions are in:

build/README.md

Acknowledgments

Openresearchtools-Engine is possible because of the open work done by these projects. We are genuinely grateful to their maintainers and contributors. Without them, this project would not exist.

llama.cpp: core model runtime, GPU offload controls, KV-cache behavior, multi-GPU split controls, and server-side inference lifecycle patterns used by the bridge and engine orchestration.
whisper.cpp: transcription pipeline foundations, including audio-to-token flow, timestamp-oriented decoding behavior, and integration patterns for speech tasks.
NVIDIA NeMo / Sortformer: model/archive semantics and parity baselines used as the source basis for the repo-written native Sortformer conversion and validation flow.
parakeet.cpp: additional external C++ Sortformer reference studied during native Sortformer bring-up and converter/runtime cross-checking.
voxtral-cpp: primary native ggml implementation base for the current Voxtral realtime runtime, and the source origin for the repo-kept Voxtral GGUF conversion tool.
vLLM, voxtral.c, voxtral-mini-realtime-rs, and mlx-audio: external reference implementations and throughput/behavior baselines used during Voxtral bring-up and validation.
Docling: practical references for VLM document-conversion behavior, including page rendering/scaling heuristics and Markdown-oriented extraction expectations for PDF-to-Markdown workflows.
PDFium and pdfium-render: PDF rasterization and page access primitives used for native page rendering/extraction in the PDF modules.
FFmpeg (LGPL shared builds): audio normalization and format conversion path used when input media needs conversion to inference-friendly audio format.
Rust ecosystem crates in example-cli, pdf, and pdfvlm: CLI plumbing, parsing, and runtime glue that make the native components usable as a cohesive application layer.

For full notices, license types, and source provenance:

third_party/LICENSES.md
third_party/README.md

How to cite

Suggested citation:

Rutkauskas, L. (2026). Openresearchtools-Engine (Version 1.0.0) [Computer software]. OpenResearchTools. https://github.com/openresearchtools/engine.

BibTeX:

@software{Rutkauskas_Openresearchtools-Engine_2026,
  author    = {Rutkauskas, L.},
  title     = {Openresearchtools-Engine},
  version   = {1.0.0},
  date      = {2026-03-04},
  url       = {https://github.com/openresearchtools/engine},
  publisher = {OpenResearchTools},
  license   = {MIT}
}

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
bridge		bridge
build		build
clusterui		clusterui
diarize		diarize
docs		docs
engine		engine
extras		extras
multi_node_server/native		multi_node_server/native
pdf		pdf
pdfvlm		pdfvlm
runtime-manifests		runtime-manifests
third_party		third_party
.dockerignore		.dockerignore
.gitignore		.gitignore
AGENTS.md		AGENTS.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation