Repository navigation
parser: don't read a JSON line's key names as its level - #38
reckless-sherixx wants to merge 1 commit into
Conversation
A container log line has no level from its source, so the line collector
guesses one from the raw text. For a JSON line that has no level key, that
guess came from the key names: an access log's {"status":200,"error":""}
read as an error, because "error" is a key. JSON parsing kept the guess,
since it found no level key to override it with.
The collector now records that a level was guessed. When the line turns
out to be a JSON object without a level key, the parser guesses again from
the JSON message field alone ({"msg":"error: db down"} is still an error),
or leaves the level unknown. A level the source reports, such as a journald
priority, is kept. Both parseJson modes behave the same.
There was a problem hiding this comment.
Code Review
This pull request introduces a levelGuessed flag to track whether a log level was guessed from the content. In parser.go, this flag is used to refine the log level for JSON lines that lack an explicit level key, preventing false-positive level guesses from key names (such as "error"). A performance optimization was suggested to avoid the expensive JSON decoding in parser.go when msg.Level is already LevelUnknown.
| if msg.levelGuessed && structuredLevel == LevelUnknown && (jl == nil || jl.Level == LevelUnknown) { | ||
| if jl != nil { | ||
| msg.Level = GuessLevel(jl.Message) | ||
| } else if !p.parseJson { | ||
| if fields := decodeJsonObject(msg.Content); fields != nil { | ||
| msg.Level = GuessLevel(jsonLogFromFields(fields).Message) | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
When p.parseJson is false, the parser treats logs as plain text but attempts to decode JSON to correct potential false-positive level guesses (e.g., from an "error" key name). However, if msg.Level is already LevelUnknown, there is no guessed level to correct.
By adding msg.Level != LevelUnknown to the condition, we can completely avoid the expensive decodeJsonObject call (which uses json.NewDecoder and allocates maps) for all log lines that are already LevelUnknown. This is a significant performance optimization for high-throughput log streams when JSON parsing is disabled.
| if msg.levelGuessed && structuredLevel == LevelUnknown && (jl == nil || jl.Level == LevelUnknown) { | |
| if jl != nil { | |
| msg.Level = GuessLevel(jl.Message) | |
| } else if !p.parseJson { | |
| if fields := decodeJsonObject(msg.Content); fields != nil { | |
| msg.Level = GuessLevel(jsonLogFromFields(fields).Message) | |
| } | |
| } | |
| } | |
| if msg.levelGuessed && structuredLevel == LevelUnknown && (jl == nil || jl.Level == LevelUnknown) { | |
| if jl != nil { | |
| msg.Level = GuessLevel(jl.Message) | |
| } else if !p.parseJson && msg.Level != LevelUnknown { | |
| if fields := decodeJsonObject(msg.Content); fields != nil { | |
| msg.Level = GuessLevel(jsonLogFromFields(fields).Message) | |
| } | |
| } | |
| } |
Summary
A JSON log line with no level key but an
errorkey was emitted as an error. Go access logs (Echo, Gin) are the common case: every{"method":"GET","uri":"/healthz","status":200,"error":""}health check came out aserror.When the source reports no level, the multiline collector guesses one from the raw line, and
GuessLevelreads the key"error"as the word error. JSON parsing kept that guess because the line had no level key to override it.Messagenow records whether its level was guessed or reported by the source (unexportedlevelGuessed).{"msg":"error: db down"}is still an error), or left unknown when there is no message field.parseJsonmodes behave the same.Trade-off: a level-less line whose only error signal is a value, such as
{"status":502,"error":"upstream connect error"}with no message field, is nowunknownrather thanerror. A field's value is not a level either, and no other rule tells it apart from an empty error field.No API change.
Test plan
TestParserJsonWithoutLevelKey, 7 cases × bothparseJsonmodes, run throughNewParser(collector + parser). The cases are an emptyerrorkey, anerrvalue, a level in the message, a message without one, a level key still winning, a source level kept, and plain text still guessed. The twoerror-key cases fail onmain.go vet ./...,go build -mod=readonly ./...,go test -count=1 ./...pass. (-racenot run locally: no cgo on the dev box; CI runs it.)v0.0.0-20261023171829-38071d9d9983and this branch: the two health checks go fromerrortounknown;{"level":"error",...},{"msg":"error: ..."}andERROR ...stayerror.