New Preview

README.md

@@ -1,4 +1,4 @@
-# `seqcli` [![Build status](https://ci.appveyor.com/api/projects/status/sc3iacxwxqqfjgdh?svg=true)](https://ci.appveyor.com/project/datalust/seqcli) [![GitHub release](https://img.shields.io/github/release/datalust/seqcli.svg)](https://github.com/datalust/seqcli/releases)
+# `seqcli` [![Build status](https://ci.appveyor.com/api/projects/status/sc3iacxwxqqfjgdh/branch/dev?svg=true)](https://ci.appveyor.com/project/datalust/seqcli/branch/dev) [![GitHub release](https://img.shields.io/github/release/datalust/seqcli.svg)](https://github.com/datalust/seqcli/releases)
 
 The [Seq](https://getseq.net) client command-line app. Supports logging (`seqcli log`), searching (`search`), tailing (`tail`), querying (`query`) and [JSON log file](https://github.com/serilog/serilog-formatting-compact) ingestion (`ingest`).
 
@@ -55,14 +55,16 @@ Send JSON log events from a file or `STDIN`.
 Example:
 
 ```
-seqcli ingest -i events.clef --filter="@Level <> 'Debug'" -p Environment=Test
+seqcli ingest -i events.clef --json --filter="@Level <> 'Debug'" -p Environment=Test
 ```
 
 | Option | Description |
 | ------ | ----------- |
 | `-i`, `--input=VALUE` | CLEF file to ingest; if not specified, `STDIN` will be used |
 |       `--invalid-data=VALUE` | Specify how invalid data is handled: fail (default) or ignore |
 | `-p`, `--property=VALUE1=VALUE2` | Specify event properties, e.g. `-p Customer=C123 -p Environment=Production` |
+| `-x`, `--extract=VALUE` | An extraction pattern to apply to plain-text logs (ignored when `--json` is specified) |
+|       `--json` | Read the events as JSON (the default assumes plain text) |
 | `-f`, `--filter=VALUE` | Filter expression to select a subset of events |
 | `-s`, `--server=VALUE` | The URL of the Seq server; by default the `connection.serverUrl` value will be used |
 | `-a`, `--apikey=VALUE` | The API key to use when connecting to the server; by default `config.apiKey` value will be used |
@@ -147,3 +149,72 @@ Stream log events matching a filter.
 ### `version`
 
 Print the current executable version.
+
+## Extraction patterns
+
+The `seqcli ingest` command can be used for parsing plain text logs into structured log events.
+
+```shell
+seqcli ingest -x "{@t:timestamp} [{@l:ident}] {@m:*}{:n}{@x:*}"
+```
+
+The `-x` argument above is an _extraction pattern_ that will parse events like:
+
+```
+2018-02-21 13:29:00.123 +10:00 [ERR] The operation failed
+System.DivideByZeroException: Attempt to divide by zero
+  at SomeClass.SomeMethod()
+```
+
+### Syntax
+
+Extraction patterns have a simple high-level syntax:
+
+ * Text that appears in the pattern is matched literally - so a pattern like `Hello, world!` will match logging statements that are made up of this greeting only,
+ * Text between `{curly braces}` is a _match expression_ that identifies a part of the event to be extracted, and
+ * Literal curly braces are escaped by doubling, so `{{` will match the literal text `{`, and `}}` matches `}`.
+
+Match expressions have the form:
+
+```
+{name:matcher}
+```
+
+Both the name and matcher are optional, but either one or the other must be specified. Hence `{@t:timestamp}` specifies a name of `@t` and value `timestamp`, `{IPAddress}` specifies a name only, and `{:n}` a value only (in this case the built-in newline matcher).
+
+The _name_ is the property name to be extracted; there are four built-in property names that get special handling:
+
+ * `@t` - the event's timestamp
+ * `@m` - the textual message associated with the event
+ * `@l` - the event's level
+ * `@x` - the exception or backtrace associated with the event
+
+Other property names are attached to the event payload, so `{Elapsed:dec}` will extract a property called `Elapsed`, using the `dec` decimal matcher.
+
+Match expressions with no name are consumed from the input, but are not added to the event payload.
+
+### Matchers
+
+Matchers identify chunks of the input event.
+
+Different matchers are needed so that a piece of text like `200OK` can be separated into separate properties, i.e. `{StatusCode:nat}{Status:alpha}`. Here, the `nat` (natural number) matcher also coerces the result into a numeric value, so that it is attached to the event payload numerically as `200` instead of as the text `"200"`.
+
+There are three kinds of matchers:
+
+ * Matchers like `alpha` and `nat` are built-in _named_ matchers.
+ * The special matchers `*`, `**` and so-on, are _non-greedy content_ matchers; these will match any text up until the next pattern element matches (`*`), the next two elements match, and so-on. We saw this in action with the `{@m:*}{:n}` elements in the example - the message is all of the text up until the next newline.
+ * More complex _compound_ matchers are described using a sub-expression. These are prefixed with an equals sign `=`, like `{Phone:={:nat}-{:nat}-{:nat}}`. This will extract chunks of text like `123-456-7890` into the `Phone` property.
+
+### Processing
+
+Extraction patterns are processed from left to right. When the first non-matching pattern is encountered, extraction stops; any remaining text that couldn't be matched will be attached to the resulting event in an `@unmatched` property.
+
+Multi-line events are handled by looking for lines that start with the first element of the extraction pattern to be used. This works well if the first line of each event begins with something unambiguous like an `iso8601dt` timestamp; if the lines begin with less specific syntax, the first few elements of the extraction pattern might be grouped to identify the start of events more accurately:
+
+```
+{:=[{@t} {@l}]} {@m:*}
+```
+
+Here the literal text `[`, a timestamp token, adjacent space ` `, level and closing `]` are all grouped so that they constitute a single logical pattern element to identify the start of events.
+
+When logs are streamed into `seqcli ingest` in real time, a 10 ms deadline is applied, within which any trailing lines that make up the event must be received.

src/SeqCli/Cli/CommandAttribute.cs

@@ -20,6 +20,7 @@ namespace SeqCli.Cli
     public class CommandAttribute : Attribute, ICommandMetadata
     {
         public string Name { get; }
+        public string SubCommand { get; }
         public string HelpText { get; }
 
         public string Example { get; set; }
@@ -29,5 +30,10 @@ public CommandAttribute(string name, string helpText)
             Name = name;
             HelpText = helpText;
         }
+
+        public CommandAttribute(string name, string subCommand, string helpText) : this(name, helpText)
+        {
+            SubCommand = subCommand;
+        }
     }
 }

src/SeqCli/Cli/CommandLineHost.cs

@@ -38,10 +38,15 @@ public async Task<int> Run(string[] args)
             if (args.Length > 0)
             {
                 var norm = args[0].ToLowerInvariant();
-                var cmd = _availableCommands.SingleOrDefault(c => c.Metadata.Name == norm);
+                var subCommandNorm = args.Length > 1 && !args[1].Contains("-") ? args[1].ToLowerInvariant() : null;
+
+                var cmd = _availableCommands.SingleOrDefault(c =>
+                    c.Metadata.Name == norm && (c.Metadata.SubCommand == subCommandNorm || c.Metadata.SubCommand == null));
+
                 if (cmd != null)
                 {
-                    return await cmd.Value.Value.Invoke(args.Skip(1).ToArray());
+                    var amountToSkip = cmd.Metadata.SubCommand == null ? 1 : 2;
+                    return await cmd.Value.Value.Invoke(args.Skip(amountToSkip).ToArray());
                 }
             }
 

src/SeqCli/Cli/CommandMetadata.cs

@@ -17,6 +17,7 @@ namespace SeqCli.Cli
     public class CommandMetadata : ICommandMetadata
     {
         public string Name { get; set; }
+        public string SubCommand { get; set; }
         public string HelpText { get; set; }
         public string Example { get; set; }
     }

src/SeqCli/Cli/Commands/ApiKey/ListCommand.cs

@@ -0,0 +1,42 @@
+using System;
+using System.Threading.Tasks;
+using SeqCli.Cli.Features;
+using SeqCli.Config;
+using SeqCli.Connection;
+using Serilog;
+
+namespace SeqCli.Cli.Commands.ApiKey
+{
+    [Command("apikey", "list", "List API keys on the server", Example="seqcli apikey list")]
+    class ListCommand : Command
+    {
+        readonly SeqConnectionFactory _connectionFactory;
+
+        readonly ConnectionFeature _connection;
+        readonly OutputFormatFeature _output;
+
+        public ListCommand(SeqConnectionFactory connectionFactory, SeqCliConfig config)
+        {
+            if (config == null) throw new ArgumentNullException(nameof(config));
+            _connectionFactory = connectionFactory ?? throw new ArgumentNullException(nameof(connectionFactory));
+
+            _output = Enable(new OutputFormatFeature(config.Output));
+            _connection = Enable<ConnectionFeature>();
+        }
+
+        protected override async Task<int> Run()
+        {
+            var connection = _connectionFactory.Connect(_connection);
+
+            var apiKeys = await connection.ApiKeys.ListAsync();
+            Log.Debug("Retrieved ApiKeys {@ApiKeys}", apiKeys);
+
+            foreach (var apiKey in apiKeys)
+            {
+                _output.WriteEntity(apiKey);
+            }
+
+            return 0;
+        }
+    }
+}

src/SeqCli/Cli/Commands/ApiKey/RemoveCommand.cs

@@ -0,0 +1,61 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Seq.Api.Model.Inputs;
+using SeqCli.Cli.Features;
+using SeqCli.Connection;
+
+namespace SeqCli.Cli.Commands.ApiKey
+{
+    [Command("apikey", "remove", "Remove an API key from the server",
+        Example="seqcli apikey remove -t TestApiKey")]
+    class RemoveCommand : Command
+    {
+        readonly SeqConnectionFactory _connectionFactory;
+        readonly ConnectionFeature _connection;
+        string _title;
+        string _id;
+
+        public RemoveCommand(SeqConnectionFactory connectionFactory)
+        {
+            _connectionFactory = connectionFactory;
+            _connection = Enable<ConnectionFeature>();
+            Options.Add(
+                "t=|title=",
+                "Remove API Keys with the specified title",
+                (t) => _title = t);
+
+            Options.Add(
+                "i=|id=",
+                "Remove API Keys with the specified Id",
+                (t) => _id = t);
+        }
+
+        protected override async Task<int> Run()
+        {
+            if (_title != default && _id != default)
+            {
+                Console.WriteLine("You can only specify \"title\" or \"id\" not both");
+                return -1;
+            }
+
+            var connection = _connectionFactory.Connect(_connection);
+
+            var apiKeys = await connection.ApiKeys.ListAsync();
+            var apiKeyToRemove = apiKeys.Where(ak => ak.Title == _title || ak.Id == _id).ToList();
+            if (!apiKeyToRemove.Any())
+            {
+                Console.WriteLine($"\"{_title}\" API Key doesn't exist");
+                return -1;
+            }
+
+            foreach (var apiKeyEntity in apiKeyToRemove)
+            {
+                await connection.ApiKeys.RemoveAsync(apiKeyEntity);
+            }
+            return 0;
+        }
+    }
+}

src/SeqCli/Cli/Commands/HelpCommand.cs

@@ -37,13 +37,29 @@ protected override Task<int> Run(string[] unrecognised)
         {
             var ea = Assembly.GetEntryAssembly();
             var name = ea.GetName().Name;
-
+
+            if (_markdown)
+            {
+                if (unrecognised.Length != 0)
+                    return base.Run(unrecognised);
+
+                PrintMarkdownHelp(name);
+                return Task.FromResult(0);
+            }
+
+            string topLevelCommand = null;
             if (unrecognised.Length > 0)
             {
-                var target = unrecognised[0].ToLowerInvariant();
-                var cmd = _availableCommands.SingleOrDefault(c => c.Metadata.Name == target);
-                if (cmd != null)
+                topLevelCommand = unrecognised[0].ToLowerInvariant();
+                var subCommand = unrecognised.Length > 1 && !unrecognised[1].Contains("-") ? unrecognised[1] : null;
+                var cmds = _availableCommands.Where(c => c.Metadata.Name == topLevelCommand &&
+                                                         (subCommand == null || subCommand == c.Metadata.SubCommand)).ToArray();
+                if (cmds.Length == 0)
+                    return base.Run(unrecognised);
+
+                if (cmds.Length == 1)
                 {
+                    var cmd = cmds.Single();
                     var argHelp = cmd.Value.Value.HasArgs ? " [<args>]" : "";
                     Console.WriteLine(name + " " + cmd.Metadata.Name + argHelp);
                     Console.WriteLine();
@@ -52,19 +68,13 @@ protected override Task<int> Run(string[] unrecognised)
                     cmd.Value.Value.PrintUsage();
                     return Task.FromResult(0);
                 }
-
-                return base.Run(unrecognised);
             }
 
-            if (_markdown)
-            {
-                PrintMarkdownHelp(name);
-            }
+            if (topLevelCommand != null)
+                PrintHelp(name, topLevelCommand);
             else
-            {
                 PrintHelp(name);
-            }
-
+
             return Task.FromResult(0);
         }
 
@@ -126,17 +136,40 @@ void PrintHelp(string executableName)
             Console.WriteLine();
             Console.WriteLine("Available commands are:");
 
-            foreach (var availableCommand in _availableCommands)
+            var printedGroups = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+            foreach (var avail in _availableCommands)
             {
-                Printing.Define(
-                    "  " + availableCommand.Metadata.Name,
-                    availableCommand.Metadata.HelpText,
-                    13,
-                    Console.Out);
+                if (avail.Metadata.SubCommand != null)
+                {
+                    if (!printedGroups.Contains(avail.Metadata.Name))
+                    {
+                        Printing.Define($"  {avail.Metadata.Name}", "<sub-command>", 13, Console.Out);
+                        printedGroups.Add(avail.Metadata.Name);
+                    }
+                }
+                else
+                {
+                    Printing.Define($"  {avail.Metadata.Name}", avail.Metadata.HelpText, 13, Console.Out);
+                }
             }
 
             Console.WriteLine();
             Console.WriteLine($"Type `{executableName} help <command>` for detailed help");
         }
+
+        void PrintHelp(string executableName, string topLevelCommand)
+        {
+            Console.WriteLine($"Usage: {executableName} {topLevelCommand} <sub-command> [<args>]");
+            Console.WriteLine();
+            Console.WriteLine("Available sub-commands are:");
+
+            foreach (var avail in _availableCommands.Where(c => c.Metadata.Name == topLevelCommand))
+            {
+                Printing.Define($"  {avail.Metadata.SubCommand}", avail.Metadata.HelpText, 13, Console.Out);
+            }
+
+            Console.WriteLine();
+            Console.WriteLine($"Type `{executableName} help {topLevelCommand} <sub-command>` for detailed help");
+        }
     }
 }