Web GUI: show what a running Parallel Consumer instance is actually doing #215

The idea

A small web GUI, served by the running Parallel Consumer instance, showing what it is actually doing right now.

PC keeps a lot of state that decides its behaviour - per-partition offset encoding, shard queues, in-flight work, retry backlog, rebalance and drain phase - and today the only ways to see any of it are a Micrometer registry you have to wire up yourself, or DEBUG logs. That is a high bar for "why has this stopped?", which is the question people actually arrive with.

Seeding this with ideas rather than a spec. Nothing here is decided.

Why it is worth doing

• Most PC support questions are state questions. "Why is it not committing?", "why is this partition stuck?", "why is memory growing?" are all answerable from state PC already tracks. The information exists; it is just not reachable.
• Some behaviour cannot be fixed, only observed. The JStream result deque is unbounded by design (confluentinc#912: Memory Leak from JStreamVertxParallelStreamProcessor #122): a pull-based Stream has no backpressure against a push-based producer, so the buffer can grow without limit and no amount of internal cleverness changes that. For that class of thing, visibility is the mitigation - see the sibling metrics issue.
• Micrometer already exists but assumes an operator. There are ~25 metrics defined in PCMetricsDef, and a MeterRegistry you must supply. That is right for production, and wrong for a developer trying to understand a local run or a one-off incident.

Seed ideas: what it could show

Offset state per partition - the highest-seen, highest-sequential-succeeded, latest-committed and incomplete-offset set, side by side. This is PC's least intuitive internal model and the source of most confusion about "lost" or "replayed" records.

Offset-encoding health - payload ratio used, metadata space used, which codec was selected. The encoded offset map is bounded by the broker's metadata limit, and hitting that has real consequences most users never see coming.

Work in flight - shard count and depth, waiting records, in-flight records, and the load factor. Enough to tell "slow user function" from "not being fed".

Failure and retry - failed records, slow records, retry backlog, and the failure history PC already keeps per record.

Lifecycle and rebalance - current run state, drain phase, partition assignment epoch, recent rebalance activity. The #119-family silent stalls are precisely a lifecycle state that looks fine from outside.

Buffers that have no bound - the JStream result deque and anything else that grows when nobody drains it (see the metrics issue below).

Seed ideas: shape

• Read-only first. Control actions (pause, resume, force commit, close with drain / without) are an obvious later step, but they change the risk profile completely and should not gate the first version.
• Optional module, no new dependencies for library users. Nobody should inherit a web server by depending on parallel-consumer-core. A separate module they opt into.
• Localhost by default. No auth means no exposure: bind loopback unless explicitly configured otherwise, and be loud about it if not.
• JSON underneath the HTML. Whatever the GUI renders should be fetchable as JSON, so it is scriptable and testable without a browser. This is also the cheapest useful first slice on its own.
• No build step if it can be avoided. A single page served from the jar beats a toolchain that has to be maintained alongside the library.
• Complements Micrometer, does not replace it. Same underlying numbers; different audience and time horizon.

A plausible first slice

The JSON endpoint alone - one read-only HTTP endpoint exposing the metric set and per-partition offset state, no UI. It is independently useful (curl, scripts, tests), it forces the data model out into the open, and the HTML can be added on top later without redesigning anything.

Open questions

• Which HTTP server? The parallel-consumer-vertx module already brings one, but making the GUI depend on Vert.x drags a large dependency into an unrelated concern. JDK HttpServer is ugly and dependency-free.
• One instance or many? A single PC instance serving its own page is simple. A fleet view is far more useful operationally and much harder.
• Does this overlap enough with an existing Grafana dashboard that the effort is better spent shipping a dashboard definition instead? Worth answering before building anything.

Related

• confluentinc#912: Memory Leak from JStreamVertxParallelStreamProcessor #122 - JStream memory growth: unbounded by design, so observability is the mitigation rather than a fix
• confluentinc#859: Memory leak in PCMetrics class #120 - PCMetrics memory leak: the metrics layer has its own reliability history worth knowing before building on it
• confluentinc#27: Micrometer metrics #125 - the original Micrometer request, long since delivered; this is the layer above it