Refactor Cmd2.parseline()

[kotfu]

To ensure consistency in parsing, we need to do a bit more refactoring:

• refactor tab completion to use StatementParser instead of parseline()
• refactor Cmd2.parseline() to use StatementParser.parse()

These refactoring tasks involve a complex issue related to submenu support, namely submenu instances don't have access to the self.command_parser attribute.

[tleonhardt]

@kmvanbrunt Can you please make sure that tab-completion is working correctly after all of the massive changes involved with moving from pyparsing to shlex? That could either be part of this Issue or a separate one. It is just something that is important, should be done, and you are the expert on.

[kmvanbrunt]

@tleonhardt
Work has been hectic the last two weeks. I'll start hammering on the new stuff this weekend.

@kotfu
Can a terminator be used in non-multiline commands?
Can I do something like: load file ; !ls output.txt

[kotfu]

I emulated the pyparsing based logic when I switched to shlex. I'm not in love with it, and am open to changes if we think it would be better.

A terminator can be used on non-multiline commands, and it will terminate the command. Stuff after the terminator is not interpreted as an additional command. Output redirection must be after a terminator (if there is a terminator present).

As a result, load file ; !ls output.txt is exactly the same as typing load file. Which is not exactly intuitive, and if tab completion works once you are passed the terminator, then it makes the user think that what they are typing will actually do something (which it won't).

In addition, the redirection tokens ('>', '>>', and '|') are treated as arguments, not redirection, if they appear before a terminator. Therefore say hello > alice; does not redirect any output, but say hello > alice does.

[kmvanbrunt]

This is actually a pretty big issue. I think we need to come up with requirements for how tab completion is supposed to behave. Up until now tab completion has always used parseline() because it never accounted for the advanced parsing features of cmd2.

Do we care about supporting comments and multiline commands when tab completing? If not, there isn't much to do. The only real decision in this case is agreeing on how commands are delimited. cmd states the following:

Interpreters constructed with this class obey the following conventions:
...
2. A command is parsed out of each line by collecting the prefix composed of characters in the identchars member.

parseline() obeys this unless default_to_shell is on. However, the new shlex parser delimits commands using a space. If we continue doing this, then we are no longer adhering to cmd standards.

If we do want tab completion to account for comments and multiline commands, then let's figure out how we want it to work.

[kmvanbrunt]

Wow @kotfu, you've been busy. I guess that answers my identchars question :)
This also means we now support unicode commands.

[kotfu]

Yes, we now support unicode commands, but we break backwards compatibility with cmd in the standard library. I thought long and hard about this, and I think it's the right move. Since we merged in the ply branch with the shlex-based parsing code, cmd2 has ignored identchars. However, I hate breaking backwards compatibility and I could easily be persuaded to go back to supporting identchars, but it makes unicode command names impractical. I tried to do it and it means you have to generate an enormous string containing every possible valid unicode code point. And you have to scan that string for every character in the command entered by the user.

I don't want to squelch a meaningful debate on the identchars topic. If anybody feels like this needs further discussion, let's have it.

[kmvanbrunt]

I'm OK with it. I just wanted to make sure everyone knew we were moving away from cmd conventions.

[kotfu]

I don't think tab completion needs to support comments. Comments are useful in scripts and transcripts, but nobody types them interactively to document for themselves what the command they are typing is going to do.

It would be nice if multiline commands supported tab completion. I'll have to do some testing and see if tab completion is working on multiline commands now.