Updating and moving the grammar to a separate file

[mtdowling]

This commit simplifies the JMESPath grammar and moved the grammar to a
separate downloadable file.

This is WIP PR. I'm finding other places in the grammar that I think could be improved.

[mtdowling]

Actually, "||" and "|" are still ambiguous. What's the desired precedence here?

I found more ambiguities with pipes and ors: a | b | c was ambiguous, so I've updated the grammar. I also noticed an issue with my grammar where projections were consuming pipes and ors. I've made a change to address this.

I've also adding slicing and grouping to the grammar.

[jamesls]

I had an updated grammar somewhere that had part of the precedence baked into the grammar. It wasn't complete but it had the or/pipe's worked out. I'll see if I can find that branch. We should double check that it's consistent with this.

The desired precedence is that a || b | b is parsed as ((a || b) | c). I believe we have a compliance test somewhere that verifies this. I confirmed that this is how the python lib parses this expression.

>>> pprint(jmespath.compile('a || b | c').parsed)
{'children': [{'children': [{'children': [], 'type': 'field', 'value': 'a'},
                            {'children': [], 'type': 'field', 'value': 'b'}],
               'type': 'or_expression'},
              {'children': [], 'type': 'field', 'value': 'c'}],
 'type': 'pipe'}

[mtdowling]

Good catch. I've pushed a commit that ensures that or expressions bind more tightly than pipe expressions.

[mtdowling]

I've now shortened the majority of the rule names (e.g., expression is now expr). This shortening of the rule names seems to be in line with most other grammars I've seen in the wild (i.e., http://www.lua.org/manual/5.2/manual.html#9 and https://docs.python.org/3/reference/grammar.html).

I updated the or and and to have an explicit precedence so that and binds more tightly than or.

I updated subexpr to now split out into object-subexpr and array-subexpr to better distinguish between object type access using a "." and array style access using a "[". This change fixes a bug that is present in the current grammar that allows you to use dot style object access off of a multi-list (i.e., this is no longer allowed because you cannot access an array as an object: [a, b].c). I think splitting these rules out will help us to very granularly specify what and how subexpressions descend into data.

I think this grammar update is pretty close to being a fully functional and non-ambiguous replacement to the current grammar. I'm using this grammar verbatim in the Clojure implementation, so you can use that and play around with how it parses to see how everything works. I'll continue to flesh out the Clojure implementation and try to get it fully compliant with the test suite to ensure that the grammar is backwards compatible.

[mtdowling]

I pushed a change that allows for insignificant whitespace built into the grammar (similar to the JSON ABNF). This makes the grammar much more robust and does not require implementation details to be part of parsing (i.e., we no longer need to say "whitespace is insignificant except inside of quotes and literals).

[jamesls]

I'm starting to pick up work again on JEP-9 (the improved filters with ands, nots, parens, etc) and I want to incorporate this grammar as part of that work. I'm starting to take a look at this.

[jamesls]

Shouldn't have the *. It should just be optional instead of zero or more.

[mtdowling]

Ah yeah, should be ["-"] 1*DIGIT

[mtdowling]

I'll send an updated PR shortly

[jamesls]

What was the motivation for this change? I would prefer to have the JSON grammar in here. Otherwise this will allow invalid strings such as:

`[`

which is invalid.

[mtdowling]

I thought [ was valid (currently) because JMESPath allows you to pass in unquoted strings here. It will try to parse and see that it failed and then parse it as "[".

[mtdowling]

Actually, I've got an updated copy of the grammar that includes JSON and fixes a bunch of stuff from the clojure implementation... Updating coming.

[mtdowling]

I've updated the grammar to include the changes from jmespath.clj. This includes adding JSON back into the grammar and better whitespace control.

[jamesls]

Just curious, why is this called terminal? I'm thinking of terminals as the leaf nodes in the parsed tree, i.e it can't be replaced with anything. But everything on the RHS can be broken down into more nodes.

[mtdowling]

Maybe it isn't a good name. I named it this because it owns the entire side. Do you have a suggestion?

[mtdowling]

I've updated the grammar to support JEP 12 and added support to the Clojure implementation for raw string literals.

[jamesls]

Just a heads up, I synced with this latest version that includes raw string literals and abnfstress caught this. The ["'"] means that the closing single quote is optional so strings such as '\' would be valid. This should be raw-string-escape = escape "'"