Proposal: add live tail by nico34638 · Pull Request #6899 · quickwit-oss/quickwit

nico34638 · 2026-10-12T10:25:11Z

Description

When something breaks, you want to see logs as they arrive, like tail -f. In Quickwit, a log is only searchable once its split is published, which can take from 30s to a few minutes. That's too slow to follow a deploy or debug a live issue.

This PR adds GET /api/v1/{index_id}/tail?query=.... It streams the matching documents as Server-Sent Events as soon as an ingester has persisted them, before they are indexed and committed.

flowchart LR
    subgraph ingester["Each ingester"]
        P[persist] -->|"valid docs, after the WAL write<br/>(only while someone tails)"| HUB[LiveTailHub<br/>broadcast per index]
        HUB --> S[LiveTailSession<br/>filter + rate limit]
    end

    subgraph rest["Any REST node"]
        C[Coordinator]
    end

    S -->|gRPC stream| C
    C -->|SSE| CL[Client]

The documents are taken as early as possible: in the ingester, right after they're written to the WAL. When someone is tailing an index, the ingester pushes each batch into an in-memory channel. Nothing is copied, and when nobody is tailing, nothing happens at all. Since a WAL position is only written once, there's no replay to worry about.

Each tail session reads that channel, and every 250ms indexes the new docs into a small in-RAM Tantivy index to run the query. So the query behaves exactly like a search. Matches are rate limited and streamed back over gRPC.

The node that receives the request asks chitchat which ingesters have open shards for the index, opens a stream to each, and forwards everything to the client as SSE. The list is refreshed every 5s.

Scope of this first version:

Only the ingest V2 path is covered: ingest API, Elasticsearch bulk and OTLP. Sources read by the indexing pipelines (Kafka, Kinesis, Pulsar, SQS, ...) are not tailed.
No VRL: the ingester persists documents before the transform, so an index whose _ingest-source has a transform is rejected with a 400 rather than returning results that don't match what gets indexed.

Limits are in the node config and can be overridden:

live_tail:
  max_sessions: 4            # QW_LIVE_TAIL_MAX_SESSIONS
  max_docs_per_sec: 10000    # QW_LIVE_TAIL_MAX_DOCS_PER_SEC

chatgpt-codex-connector · 2026-10-12T10:25:23Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T14:32:00.967993Z	`9f59c89`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

nico34638 · 2026-10-12T10:33:53Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91c8f0ec7a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

nico34638 · 2026-10-26T20:21:49Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7e75ee904

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-10-26T20:30:59Z

+                let live_tail_doc_mapper_opt = self
+                    .live_tail_hub
+                    .is_active(subrequest.index_uid())
+                    .then(|| doc_mapper.clone());


Check tail subscribers at WAL publication time

When a persist call reaches this check before a tail subscribes but finishes document validation and the WAL append after open_live_tail has returned, this cached None prevents the later publish even though the document was persisted after the stream opened. This race occurs on busy existing shards independently of the documented new-shard gossip delay and silently loses an in-flight batch, so retain the cheap batch/mapper clone and decide whether to publish after the append instead of freezing subscriber activity before the intervening awaits.

Useful? React with 👍 / 👎.

Nice catch, that was a real race. We now decide whether to publish after the WAL write, not before validation. Fixed

chatgpt-codex-connector · 2026-10-26T20:30:59Z

+use quickwit_common::metrics::exponential_buckets;
+use quickwit_metrics::{LazyCounter, LazyHistogram, lazy_counter, lazy_histogram};
+
+pub(crate) static LIVE_TAIL_DOCS_PUBLISHED: LazyCounter = lazy_counter!(


Regenerate the inventory for live-tail metrics

For CI runs touching this Rust file, the metrics-inventory workflow includes quickwit/**/*.rs and compares the generated inventory byte-for-byte (.github/workflows/metrics-inventory.yml:47-50,66-73). The new quickwit_live_tail_* metrics are absent from quickwit/quickwit-metrics-inventory/metrics.yaml, so that job will generate a diff and fail; run make metrics-inventory and commit the updated inventory.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-26T20:30:59Z

+        let event = match message_result?.message {
+            Some(live_tail_message::Message::Hits(hits)) => hits_event(index_id, hits)?,
+            Some(live_tail_message::Message::Dropped(dropped)) => LiveTailEvent::Dropped {
+                reason: dropped.reason(),


Reject unspecified live-tail drop reasons

When a malformed or newer ingester sends a LiveTailDropped with reason 0 or an unknown numeric value, Prost's reason() maps it to Unspecified, and this line exposes "unspecified" to SSE although the public contract permits only lagged and rate_limited. Treat this protocol state as an error so the forwarder reports and reconnects instead of silently changing the API shape.

AGENTS.md reference: AGENTS.md:L21-L22

Useful? React with 👍 / 👎.

it's handled the same way as empty messages, as an error that makes the stream reopen.

nico34638 · 2026-10-27T14:20:19Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f59c89d41

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-10-27T14:31:56Z

+use crate::metrics::LIVE_TAIL_DOCS_PUBLISHED;
+
+/// Number of batches buffered per index before slow subscribers start lagging.
+const DEFAULT_CHANNEL_CAPACITY: usize = 64;


Bound the hub buffer by bytes instead of batches

When a matching tail client stops reading from a busy index, its session eventually stops draining this receiver and the hub retains up to 64 whole DocBatchV2 values. With the default 10 MiB ingest request limit, that is roughly 640 MiB of raw document data for one index, and the default session limit permits the same accumulation across several indexes, so a slow client can exhaust an ingester's memory; bound this queue by bytes rather than persist-call count.

Useful? React with 👍 / 👎.

True, a client that stops reading could hold a lot of memory. There's now a 64 MiB cap per index; past that, batches are skipped and show up as lagged.

chatgpt-codex-connector · 2026-10-27T14:31:56Z

+        self.rate_limiter
+            .lock()
+            .expect("lock should not be poisoned")
+            .consume(num_allowed_docs);


Don't charge oversized documents against the rate budget

When an oversized matching document precedes normal matches and the requested budget is small, this consumes quota for the oversized document before the later size check drops it. For example, with max_docs_per_sec=1 and [oversized, small], the first document is reported too_large, the second is reported rate_limited, and no hit is sent even though zero documents used the send budget; skip or refund oversized matches and continue selecting sendable documents before consuming quota.

Useful? React with 👍 / 👎.

Good catch, oversized docs no longer count against the budget.

chatgpt-codex-connector · 2026-10-27T14:31:56Z

+    grpc_max_message_num_bytes
+        .saturating_sub(HITS_MESSAGE_OVERHEAD_NUM_BYTES)
+        .min(TARGET_HITS_PAYLOAD_NUM_BYTES)


Negotiate the coordinator's gRPC receive limit

When role-specific node configs differ and the REST/coordinator node has a smaller grpc.max_message_size than the ingester, this cap is computed only from the ingester's limit while live_tail_client configures Tonic decoding from the coordinator's limit. A default ingester can therefore send a near-1 MiB chunk that a coordinator configured at the valid 1 MB minimum rejects, causing repeated stream errors and missed documents; fresh evidence after the earlier size fix is that OpenLiveTailRequest still carries no receiver limit to negotiate.

Useful? React with 👍 / 👎.

the coordinator now sends its limit and the ingester uses the smaller one.

chatgpt-codex-connector · 2026-10-27T14:31:56Z

+            continue;
+        }
+        // Ingesters serialize valid JSON objects: this is a violated invariant, not a client error.
+        let doc = serde_json::from_slice(line).map_err(|error| {


Reject non-object hit documents

If a malformed or newer ingester places a valid JSON scalar or array in docs_ndjson, deserialization into JsonValue succeeds and the coordinator publishes it in docs, even though the protocol and local producer require JSON objects. Check that each value is an object and fail the stream so the protocol violation is reported and reconnected instead of silently changing the SSE shape.

AGENTS.md reference: AGENTS.md:L21-L22

Useful? React with 👍 / 👎.

Makes sense, it's now treated as a protocol error. fix

nico34638 requested a review from a team as a code owner October 12, 2026 10:25

nico34638 marked this pull request as draft October 12, 2026 10:26

chatgpt-codex-connector Bot reviewed Oct 12, 2026

View reviewed changes

dayaffe added area:core Server layer, CLI, config, actors, shared crates area:indexing Indexing and merge pipelines, compactor type:feature A new capability, such as an endpoint, query type, source, setting or command labels Oct 22, 2026

nicolas-fidel-wmx force-pushed the live-tail branch from 91c8f0e to d7e75ee Compare October 26, 2026 20:21

chatgpt-codex-connector Bot reviewed Oct 26, 2026

View reviewed changes

nicolas-fidel-wmx force-pushed the live-tail branch from 97e1499 to 9f59c89 Compare October 27, 2026 13:41

chatgpt-codex-connector Bot reviewed Oct 27, 2026

View reviewed changes

nico34638 added 5 commits October 29, 2026 07:39

quickwit: poc live taill

a6d64b2

quickwit-live-tail: make configurable

1da8bb6

quickwit-live-tail: move to ingester

5624b9c

quickwit-live-tail: address review comments

dbb5ba1

quickwit-live-tail: bound memory and negotiate message size

633c08b

nico34638 force-pushed the live-tail branch from 1692542 to 633c08b Compare October 29, 2026 06:39

quickwit-live-tail: reduce the memory used by filtering

b05e143

Conversation

nico34638 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

chatgpt-codex-connector Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

nico34638 commented Oct 12, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nico34638 commented Oct 26, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 commented Oct 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

nico34638 Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

nico34638 commented Oct 12, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 12, 2026 •

edited

Loading

nico34638 Oct 27, 2026 •

edited

Loading