Skip to content

feat(capi): ABI v5 - surface <EOU> vs <EOB> distinction across the C boundary #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mudler merged 1 commit into from
Jun 12, 2026
Merged

feat(capi): ABI v5 - surface <EOU> vs <EOB> distinction across the C boundary #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ parakeet_capi_free_string
parakeet_capi_last_error
# streaming (cache-aware EOU model parakeet_realtime_eou_120m-v1):
parakeet_capi_stream_begin
parakeet_capi_stream_feed # 16k mono f32 PCM -> newly-finalized text; *eou_out=1 on <EOU>/<EOB>
parakeet_capi_stream_feed # 16k mono f32 PCM -> newly-finalized text; *eou_out = event bitmask (ABI v5)
parakeet_capi_stream_finalize # flush the end-of-stream tail
parakeet_capi_stream_free
```
Expand All @@ -274,6 +274,16 @@ semantics. Additive changes (new functions) are fine without bumping.
Streaming semantics: `parakeet_capi_stream_feed` buffers PCM, decodes encoder
chunks as audio arrives (carried encoder/decoder caches), and returns the
newly-finalized text (`<EOU>`/`<EOB>` STRIPPED, surfaced via `*eou_out`).
Since ABI v5 `*eou_out` is a bitmask — `PARAKEET_EVENT_EOU` (end of utterance:
respond) | `PARAKEET_EVENT_EOB` (backchannel: do not treat as a turn) — and the
streaming JSON documents carry separate `"eou"`/`"eob"` 0/1 flags (in v4 a
single conflated any-event flag). Per-event timestamps come from
`parakeet_capi_stream_drain_events`
(`parakeet_stream_event{token,is_eob,encoder_frame,time_sec}`, free with
`parakeet_capi_free_events`) or the `"events"` array
(`{"type":"eou"|"eob","frame","t"}`) in the `stream_feed_json` /
`stream_finalize_json` documents. The event queue is shared between the typed
drain and the JSON entry points — use one style per stream.
`parakeet_capi_stream_finalize` flushes the streaming tail and does NOT
fabricate an `<EOU>` NeMo's cache-aware streaming would not emit (for a final
chunk whose right context is incomplete, the trailing `<EOU>` is dropped exactly
Expand Down
11 changes: 10 additions & 1 deletion docs/parity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -570,8 +570,17 @@ from `libparakeet.so` (verified via `nm -D`).
- `tests/test_capi_stream.cpp` (`test_capi_stream`) — feeds `speech.wav` PCM in
chunks through the streaming C-API; the concatenated text + `finalize` equals
`baseline.stream_text` from `/tmp/baseline_eou_stream.gguf` (NeMo streaming).
Skips (exit 77) unless `PARAKEET_TEST_GGUF_EOU` +
A second phase streams a two-utterance clip (speech + 0.6 s silence + speech,
the `gen_stream_reset_baseline.py` construction) and asserts
`parakeet_capi_stream_drain_events` (ABI v5) surfaces the mid-stream `<EOU>`
as a typed record agreeing with the per-feed `eou_out` event mask
(PARAKEET_EVENT_EOU | PARAKEET_EVENT_EOB), with sane monotone timestamps. Skips (exit 77) unless `PARAKEET_TEST_GGUF_EOU` +
`PARAKEET_TEST_BASELINE_EOU_STREAM` are set.
- `tests/test_capi_stream_json.cpp` (`test_capi_stream_json`) — drives the
streaming JSON entry points on `speech.wav` and asserts the documents carry
`frame_sec`, per-word timestamps, and the `"events"` array; a second phase
streams the two-utterance clip and asserts a typed `{"type":"eou",...}` event
appears. Skips (exit 77) unless `PARAKEET_TEST_GGUF_EOU` is set.

Reproduce:

Expand Down
69 changes: 61 additions & 8 deletions include/parakeet_capi.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,16 @@ typedef struct parakeet_ctx parakeet_ctx;
// (start/end/conf) plus frame_sec alongside the newly-finalized text, and
// added "frame_sec" to the transcribe_*_json documents. The original entry
// points are unchanged.
//
// v5: the <EOU> (end of utterance) vs <EOB> (end of backchannel) distinction is
// now visible across the C boundary. BREAKING semantics on the streaming
// surface: parakeet_capi_stream_feed's `*eou_out` is now a bitmask
// (PARAKEET_EVENT_EOU | PARAKEET_EVENT_EOB) instead of an any-event 0/1,
// and the JSON "eou" field now means "an <EOU> fired" only, with a new
// "eob" field beside it (in v4 both meant "an <EOU> OR <EOB> fired").
// Added parakeet_capi_stream_drain_events (typed per-event records with
// is_eob + timestamps, freed with parakeet_capi_free_events) and an
// "events" array in the stream_feed_json / stream_finalize_json documents.
int parakeet_capi_abi_version(void);

// Load a GGUF model. Returns an owning context, or NULL on failure.
Expand Down Expand Up @@ -174,13 +184,22 @@ parakeet_stream* parakeet_capi_stream_begin(parakeet_ctx* ctx);
parakeet_stream* parakeet_capi_stream_begin_lang(parakeet_ctx* ctx,
const char* target_lang);

// Bits for parakeet_capi_stream_feed's *eou_out mask. <EOU> = the user
// finished a complete utterance (a voice agent responds); <EOB> = the user
// finished a backchannel, a short acknowledgment like "uh-huh" while the other
// party speaks (a voice agent must NOT treat it as the user taking the turn).
#define PARAKEET_EVENT_EOU 1
#define PARAKEET_EVENT_EOB 2

// Feed a block of 16 kHz MONO float PCM (`pcm`, length `n_samples`). The session
// buffers the audio and decodes as full encoder chunks become available.
// Returns the newly-finalized text since the last call as a malloc'd UTF-8
// string (free with parakeet_capi_free_string) — "" (empty, non-NULL) if no new
// text was finalized this call, NULL only on error. <EOU>/<EOB> are stripped
// from the text and surfaced as events: if `eou_out` is non-NULL it is set to 1
// when an <EOU>/<EOB> event fired during this feed, else 0.
// from the text and surfaced as events: if `eou_out` is non-NULL it is set to
// the bitwise OR of PARAKEET_EVENT_EOU / PARAKEET_EVENT_EOB for the event types
// that fired during this feed (0 if none). Per-event timestamps are available
// via parakeet_capi_stream_drain_events.
char* parakeet_capi_stream_feed(parakeet_stream* s, const float* pcm,
int n_samples, int* eou_out);

Expand All @@ -190,16 +209,50 @@ char* parakeet_capi_stream_feed(parakeet_stream* s, const float* pcm,
// complete. Does NOT fabricate an <EOU> NeMo's streaming would not emit.
char* parakeet_capi_stream_finalize(parakeet_stream* s);

// One <EOU>/<EOB> event emitted by the streaming decoder. <EOU> marks the end
// of a complete utterance (the user yielded the turn); <EOB> marks the end of a
// backchannel (a short acknowledgment like "uh-huh" while the other party
// speaks — a voice agent typically responds on <EOU> but must NOT treat <EOB>
// as the user taking the turn). time_sec is the absolute (stream-relative)
// emission time: encoder_frame * frame_sec.
typedef struct parakeet_stream_event {
int token; // raw vocab id of the special token
int is_eob; // 0 = <EOU> (end of utterance), 1 = <EOB> (backchannel)
int encoder_frame; // absolute encoder-output frame index of the emission
float time_sec; // encoder_frame * frame_sec, seconds from stream start
} parakeet_stream_event;

// Drain the <EOU>/<EOB> events accumulated since the last drain. On success
// returns the event count (>= 0) and, when the count is nonzero, sets
// `*out_events` to a malloc'd array of that many records (release with
// parakeet_capi_free_events); `*out_events` is NULL when the count is 0.
// Returns -1 on error (NULL stream/out pointer) with `*out_events` NULL.
// The queue is shared with the JSON entry points: stream_feed_json /
// stream_finalize_json also drain it (into their "events" array), so use one
// style or the other per stream.
int parakeet_capi_stream_drain_events(parakeet_stream* s,
parakeet_stream_event** out_events);

// Free an event array previously returned by parakeet_capi_stream_drain_events.
// Safe on NULL.
void parakeet_capi_free_events(parakeet_stream_event* events);

// Like parakeet_capi_stream_feed but returns a malloc'd UTF-8 JSON document
// instead of bare text:
// {"text":"...","eou":0,"frame_sec":0.080000,
// {"text":"...","eou":0,"eob":0,"frame_sec":0.080000,
// "events":[{"type":"eou","frame":31,"t":2.480}, ...],
// "words":[{"w":"...","start":0.480,"end":0.640,"conf":0.9100}, ...]}
// "text" is the newly-finalized text since the last call ("" if none); "eou" is
// 1 iff an <EOU>/<EOB> fired during this feed; "frame_sec" is the encoder frame
// stride in seconds; "words" are the words finalized this call with absolute
// (stream-relative) start/end seconds and 'min'-aggregate confidence (the same
// drain as the offline pk::group_words). Returns NULL only on error (see
// parakeet_capi_last_error). Free with parakeet_capi_free_string.
// 1 iff an <EOU> fired during this feed and "eob" 1 iff an <EOB> fired (see
// parakeet_stream_event for the semantics — they are distinct turn-taking
// signals, not conflated); "frame_sec" is the encoder frame stride in seconds;
// "events" are the <EOU>/<EOB> events drained this call, each with "type"
// ("eou" = end of utterance, "eob" = backchannel), the absolute encoder frame
// and the emission time in seconds (frame * frame_sec); "words" are the words
// finalized this call with absolute (stream-relative) start/end seconds and
// 'min'-aggregate confidence (the same drain as the offline pk::group_words).
// Returns NULL only on error (see parakeet_capi_last_error). Free with
// parakeet_capi_free_string.
char* parakeet_capi_stream_feed_json(parakeet_stream* s, const float* pcm,
int n_samples);

Expand Down
119 changes: 96 additions & 23 deletions src/parakeet_capi.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,14 @@
// surface per-word timestamps (start/end/conf) plus frame_sec alongside the
// newly-finalized text + eou flag, and a "frame_sec" field added to the
// transcribe_*_json documents. Original entry points unchanged.
#define PARAKEET_CAPI_ABI_VERSION 4
// v5: <EOU> vs <EOB> distinction across the C boundary. BREAKING semantics:
// stream_feed's *eou_out is now a bitmask (PARAKEET_EVENT_EOU |
// PARAKEET_EVENT_EOB) instead of an any-event 0/1, and the JSON "eou"
// field now means "an <EOU> fired" only, with a new "eob" field beside it.
// Added stream_drain_events / free_events (typed per-event records) and
// the "events" array in the stream_feed_json / stream_finalize_json
// documents.
#define PARAKEET_CAPI_ABI_VERSION 5

// The opaque context: a loaded model plus a buffer for the last error message.
struct parakeet_ctx {
Expand Down Expand Up @@ -137,6 +144,13 @@ void append_json_string(std::string& out, const std::string& s) {
out += '"';
}

// Append an int to `out` as a bare JSON number.
void append_json_int(std::string& out, int v) {
char buf[16];
std::snprintf(buf, sizeof(buf), "%d", v);
out += buf;
}

// Append a float to `out` formatted with `fmt` (e.g. "%.3f"). NaN/Inf are
// emitted as 0 (JSON has no NaN/Inf literal); confidences/times are finite here.
void append_json_float(std::string& out, const char* fmt, float v) {
Expand Down Expand Up @@ -178,10 +192,8 @@ std::string transcription_to_json(const pk::Transcription& tr, float frame_sec)
out += "],\"tokens\":[";
for (size_t i = 0; i < tr.tokens.size(); ++i) {
if (i) out += ',';
char idbuf[16];
std::snprintf(idbuf, sizeof(idbuf), "%d", tr.tokens[i].id);
out += "{\"id\":";
out += idbuf;
append_json_int(out, tr.tokens[i].id);
out += ",\"t\":";
append_json_float(out, "%.3f", (float)tr.tokens[i].frame * frame_sec);
out += ",\"conf\":";
Expand Down Expand Up @@ -428,11 +440,16 @@ namespace {
// new PCM is produced frame-local by StreamingMel in the feed/finalize entry
// points, NOT recomputed over the whole buffer here. `flush` marks the final
// partial chunk is_last (keep_all_outputs), draining the remaining frames. Sets
// *eou to 1 if an <EOU>/<EOB> event fired in this pass. Returns the newly-
// eou_flag / eob_flag to 1 if an <EOU> / <EOB> (respectively) fired in this
// pass — attributed by watermarking the session's un-drained event queue, so
// the queue itself is left intact for the caller to drain. Returns the newly-
// finalized text.
std::string feed_available(parakeet_stream* s, bool flush, int& eou_flag) {
std::string feed_available(parakeet_stream* s, bool flush, int& eou_flag,
int& eob_flag) {
eou_flag = 0;
eob_flag = 0;
pk::StreamingSession& sess = *s->sess;
const size_t ev0 = sess.events().size();

const int n_mels = s->n_mels;
const int T = s->mel_T;
Expand Down Expand Up @@ -474,12 +491,13 @@ std::string feed_available(parakeet_stream* s, bool flush, int& eou_flag) {

sess.feed_mel_chunk(win, win_frames, is_last);
new_text += sess.take_new_text();
if (sess.last_chunk_had_eou()) eou_flag = 1;

s->mel_buffer_idx += chunk_size; // shift_size == chunk_size here
s->first_chunk = false;
if (is_last) break; // flushed the end-of-stream tail
}
for (size_t i = ev0; i < sess.events().size(); ++i)
(sess.events()[i].is_eob ? eob_flag : eou_flag) = 1;
return new_text;
}

Expand Down Expand Up @@ -536,9 +554,10 @@ extern "C" char* parakeet_capi_stream_feed(parakeet_stream* s, const float* pcm,
std::vector<float> frames = s->mel->feed(pcm, n_samples, n_new);
append_mel_frames(s, frames, n_new);
}
int eou = 0;
std::string delta = feed_available(s, /*flush=*/false, eou);
if (eou_out) *eou_out = eou;
int eou = 0, eob = 0;
std::string delta = feed_available(s, /*flush=*/false, eou, eob);
if (eou_out) *eou_out = (eou ? PARAKEET_EVENT_EOU : 0) |
(eob ? PARAKEET_EVENT_EOB : 0);
s->ctx->last_error.clear();
char* out = dup_to_c(delta);
if (!out) { s->ctx->last_error = "out of memory"; return nullptr; }
Expand All @@ -563,8 +582,8 @@ extern "C" char* parakeet_capi_stream_finalize(parakeet_stream* s) {
std::vector<float> tail = s->mel->finalize(n_tail);
append_mel_frames(s, tail, n_tail);
}
int eou = 0;
std::string delta = feed_available(s, /*flush=*/true, eou);
int eou = 0, eob = 0;
std::string delta = feed_available(s, /*flush=*/true, eou, eob);
// After the flush the session's finalize() is a no-op text-wise (no extra
// audio) but documents the end-of-stream tail semantics.
delta += s->sess->finalize();
Expand All @@ -582,22 +601,74 @@ extern "C" char* parakeet_capi_stream_finalize(parakeet_stream* s) {
}
}

extern "C" int parakeet_capi_stream_drain_events(parakeet_stream* s,
parakeet_stream_event** out_events) {
if (out_events) *out_events = nullptr;
if (!s || !out_events) return -1;
if (!s->ctx || !s->ctx->model || !s->sess) return -1;
try {
std::vector<pk::EouEvent> evs = s->sess->drain_events();
s->ctx->last_error.clear();
if (evs.empty()) return 0;
auto* arr = static_cast<parakeet_stream_event*>(
std::malloc(evs.size() * sizeof(parakeet_stream_event)));
if (!arr) { s->ctx->last_error = "out of memory"; return -1; }
for (size_t i = 0; i < evs.size(); ++i) {
arr[i].token = (int)evs[i].token;
arr[i].is_eob = evs[i].is_eob ? 1 : 0;
arr[i].encoder_frame = evs[i].encoder_frame;
arr[i].time_sec = (float)evs[i].time_sec;
}
*out_events = arr;
return (int)evs.size();
} catch (const std::exception& e) {
s->ctx->last_error = e.what();
return -1;
} catch (...) {
s->ctx->last_error = "unknown error";
return -1;
}
}

extern "C" void parakeet_capi_free_events(parakeet_stream_event* events) {
std::free(events);
}

namespace {

// Serialize a streaming feed/finalize result to JSON: the newly-finalized text,
// the eou flag, frame_sec, and the words drained this call (absolute seconds).
// Shape matches the header doc on parakeet_capi_stream_feed_json.
std::string stream_json(const std::string& text, int eou, float frame_sec,
// the per-type eou/eob flags, frame_sec, the <EOU>/<EOB> events drained this
// call, and the words drained this call (absolute seconds). Shape matches the
// header doc on parakeet_capi_stream_feed_json. "eou" means an <EOU> fired and
// "eob" an <EOB> — they are NOT conflated (a voice agent responds on eou and
// must not treat eob as the user taking the turn); "events" carries the
// per-event timestamps.
std::string stream_json(const std::string& text, int eou, int eob,
float frame_sec,
const std::vector<pk::EouEvent>& events,
const std::vector<pk::Word>& words) {
std::string out;
out.reserve(80 + words.size() * 48);
out.reserve(80 + events.size() * 36 + words.size() * 48);
out += "{\"text\":";
append_json_string(out, text);
out += ",\"eou\":";
out += (eou ? "1" : "0");
out += ",\"eob\":";
out += (eob ? "1" : "0");
out += ",\"frame_sec\":";
append_json_float(out, "%.6f", frame_sec);
out += ",\"words\":[";
out += ",\"events\":[";
for (size_t i = 0; i < events.size(); ++i) {
if (i) out += ',';
out += "{\"type\":";
out += events[i].is_eob ? "\"eob\"" : "\"eou\"";
out += ",\"frame\":";
append_json_int(out, events[i].encoder_frame);
out += ",\"t\":";
append_json_float(out, "%.3f", (float)events[i].time_sec);
out += '}';
}
out += "],\"words\":[";
for (size_t i = 0; i < words.size(); ++i) {
if (i) out += ',';
out += "{\"w\":";
Expand Down Expand Up @@ -636,10 +707,11 @@ extern "C" char* parakeet_capi_stream_feed_json(parakeet_stream* s,
std::vector<float> frames = s->mel->feed(pcm, n_samples, n_new);
append_mel_frames(s, frames, n_new);
}
int eou = 0;
std::string delta = feed_available(s, /*flush=*/false, eou);
int eou = 0, eob = 0;
std::string delta = feed_available(s, /*flush=*/false, eou, eob);
std::vector<pk::EouEvent> events = s->sess->drain_events();
std::vector<pk::Word> words = s->sess->drain_words();
std::string json = stream_json(delta, eou, stream_frame_sec(s), words);
std::string json = stream_json(delta, eou, eob, stream_frame_sec(s), events, words);
s->ctx->last_error.clear();
char* out = dup_to_c(json);
if (!out) { s->ctx->last_error = "out of memory"; return nullptr; }
Expand All @@ -662,11 +734,12 @@ extern "C" char* parakeet_capi_stream_finalize_json(parakeet_stream* s) {
std::vector<float> tail = s->mel->finalize(n_tail);
append_mel_frames(s, tail, n_tail);
}
int eou = 0;
std::string delta = feed_available(s, /*flush=*/true, eou);
int eou = 0, eob = 0;
std::string delta = feed_available(s, /*flush=*/true, eou, eob);
delta += s->sess->finalize();
std::vector<pk::EouEvent> events = s->sess->drain_events();
std::vector<pk::Word> words = s->sess->drain_words();
std::string json = stream_json(delta, eou, stream_frame_sec(s), words);
std::string json = stream_json(delta, eou, eob, stream_frame_sec(s), events, words);
s->finalized = true;
s->ctx->last_error.clear();
char* out = dup_to_c(json);
Expand Down
5 changes: 5 additions & 0 deletions src/streaming.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,11 @@ class StreamingSession {
// Move out all EOU/EOB events collected so far (drains the queue).
std::vector<EouEvent> drain_events();

// Peek at the not-yet-drained EOU/EOB events without consuming them (the
// C-API uses the size as a watermark to attribute events to one feed pass
// while leaving the queue intact for a later drain).
const std::vector<EouEvent>& events() const { return events_; }

// Move out the WORDS finalized since the previous drain_words() call, with
// per-word start/end (seconds) and 'min'-aggregate confidence (matching the
// offline pk::group_words / NeMo timestamps=True convention). A word is
Expand Down
Loading