OtlpHttpFanOut::gc (crates/aisix-obs/src/otlp_http_sink.rs:263)
documents itself as "Called periodically by the server to GC pipelines for
deleted / disabled exporters", but git grep '\.gc(' returns no callers
anywhere in the tree. Nothing ever reaps them.
Consequence: when an operator deletes or disables an observability
exporter, the fan-out stops handing it events (the exporter is gone from
the snapshot), but its pipeline — task, client, queue — stays alive for the
life of the process. A deployment that churns exporters accumulates them.
Found while auditing #953, which tightened the adjacent
behaviour: the fan-out now re-reads the live snapshot rather than the
request's frozen one, so a deleted exporter stops receiving events
immediately. That closes the delivery side; this issue is the resource side,
and it predates that PR — gc has never been wired.
Worth deciding between two shapes rather than just adding a caller:
- call
gc from the existing snapshot-apply path, so reaping is driven by
the config change that caused it, or
- call it from a periodic task, which is what the doc comment assumes.
The first is more precise and has no idle cost; the second is more forgiving
if a pipeline can outlive its exporter for reasons other than deletion.
OtlpHttpFanOut::gc(crates/aisix-obs/src/otlp_http_sink.rs:263)documents itself as "Called periodically by the server to GC pipelines for
deleted / disabled exporters", but
git grep '\.gc('returns no callersanywhere in the tree. Nothing ever reaps them.
Consequence: when an operator deletes or disables an observability
exporter, the fan-out stops handing it events (the exporter is gone from
the snapshot), but its pipeline — task, client, queue — stays alive for the
life of the process. A deployment that churns exporters accumulates them.
Found while auditing #953, which tightened the adjacent
behaviour: the fan-out now re-reads the live snapshot rather than the
request's frozen one, so a deleted exporter stops receiving events
immediately. That closes the delivery side; this issue is the resource side,
and it predates that PR —
gchas never been wired.Worth deciding between two shapes rather than just adding a caller:
gcfrom the existing snapshot-apply path, so reaping is driven bythe config change that caused it, or
The first is more precise and has no idle cost; the second is more forgiving
if a pipeline can outlive its exporter for reasons other than deletion.