Skip to content

Plain text ingestion WIP: read frames of input with trailing lines#18

Merged
nblumhardt merged 1 commit into
datalust:feat/plainfrom
nblumhardt:feat/plain
Feb 21, 2018
Merged

Plain text ingestion WIP: read frames of input with trailing lines#18
nblumhardt merged 1 commit into
datalust:feat/plainfrom
nblumhardt:feat/plain

Conversation

@nblumhardt

Copy link
Copy Markdown
Member

This PR establishes a way of reading logs in real-time from STDIN; the eventual usage will be:

./myapp | seqcli ingest --plain

The major requirement dealt with here is the need to ingest multi-line messages; we do this by:

  • Establishing what the start of a message ("frame") will look like, via a TextParser<TextSpan>
  • Reading from the start of a message until we see the start of the next message, or, until a timed deadline elapses

This ensures that output like:

Error: System.InvalidOperationException: failed!
  at a.cs
  at b.cs

can be read into a single frame, without waiting for another message to confirm that no further stack trace lines are coming.

Not the most efficient implementation, but does successfully avoid synchronization.

One snag, the Console.In.ReadLineAsync() method blocks without returning a Task, at least on .NET Core/Windows, so an additional Task.Run() was required.

@nblumhardt nblumhardt requested a review from KodrAus February 21, 2018 03:33

var text = Content(next.Location);
while (text.HasValue)
while (text.HasValue || !text.Remainder.IsAtEnd)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was necessary to deal with the change from While() to WithoutAny() a few lines earlier. The former created zero-length matches, while the latter doesn't.

We can revert to the earlier approach when something like Span.MatchedBy(Character.Except('"').Many()) can be expressed in Superpower.

@nblumhardt nblumhardt merged commit 68d7750 into datalust:feat/plain Feb 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants