[Dustin L. Howett (DHowett)]

This pull request adds support for IRM. I chose a brute-force strategy:
every time we insert text into a row, back up the characters and
attributes and stamp them down at the end of the inserted range.

Pros:

• Easy
• Fast implementation
• Relies on preexisting DBCS lead/trail calculation and pre-formed
  CharRowCells

Cons:

• Applies brute force
• Expensive in terms of allocations, especially when inserting only a
  single character. Most output inserted is going to be on the order
  of single characters. 120 single characters, inserted one by one,
  with a memcpy() after each one.

I consider the cons to be acceptable enough for this scenario. It's rare,
and it seems like the most common use is in interactive shells where we
need to operate at "user speed". We can always optimize it later.

I chose to give SCREEN_INFORMATION knowledge of whether it was in
insert mode, rather than the text buffer, adapter or any other
component. When insert mode is on, the text buffer is instructed to
insert--a request it passes directly on to ROW. I've tried to keep
branch misprediction to a minimum (with some optimizer hints) and reduce
default-case allocations¹.

¹ by using unique_ptr<...> to store copied CharRow and ATTR_ROW,
s.t. there is no cost in the normal case except 2*sizeof(void*) bytes
on the stack in ROW::WriteCells

Closes #1947

Validation

• Tests for IRM must be added
• Tests for cool new ATTR_ROW stuff added, pass

[Dustin L. Howett (DHowett)]

CAVEAT: we should be able to insert attributes, but... nobody has asked for this, so why bother?

[paul vixie (vixie)]

if by attribute you mean blink, underline, bold, double wide, or double height, then it's what the ANSI terminal protocol requires, and waiting for somebody to ask for it is probably the wrong approach.

if by attribute you mean something internal that doesn't impact ANSI compatibility, then i'd agree, "why bother?"

see also: https://invisible-island.net/vttest/vttest.html

[Dustin L. Howett (DHowett)]

Nope, this is an implementation detail.

This writer supports text, attributes (ANSI as well as Windows console-specific flags), and combined text+attributes. Attribute-only writing changes the style of text without changing the text itself, and is used to support some Win32 console APIs such as FillConsoleOutputAttribute.

Since there’s no need for us to do an insert operation on attributes only without simultaneously manipulating the text stored in the buffer, we can avoid the complexity. It would produce an interesting effect, but I am not aware of any VT or Console requirement to do so.

[Dustin L. Howett (DHowett)]

DESIGN QUESTION: Should insertion be a property on the OutputCellIterator instead of forced down into the row?

[Dustin L. Howett (DHowett)]

NOTE: Memory profiler shows this causes 0 allocations, but it does incur a minor stack cost.

[Dustin L. Howett (DHowett)]

NOTE: This did nothing in my testing, but it makes me feel better. The optimizer actually deleted this function and TextBuffer::Write (and ...::Insert (!)) because it saw right through our trick. It lifted the boolean load from SCREEN_INFORMATION::_insertMode directly into WriteCharsLegacy and passed it down into the fully-realized ROW::Write(..., insert) (!)

[paul vixie (vixie)]

in case my earlier comment on one of the individual changes isn't noticed there, i'll repeat something here:

i hope that this PR will be gated by successful completion of the vttest suite, including its insert/delete character/line tests.

https://invisible-island.net/vttest/vttest.html

[Dustin L. Howett (DHowett)]

vttest (paul vixie (@vixie))

of course! 😄

[Dustin L. Howett (DHowett)]

Sorry, now that I'm at a computer I can explain a bit better.

Attributes and attribute insertion

The console buffer implementation (shared with Windows Terminal) stores text and "attributes" separately. Attributes are everything you'd expect--graphic renditions as well as cell size (potentially; pending #8664)--and some things you wouldn't, like whether there's an OSC 8 hyperlink or what "grid lines"² are enabled.

Today, the Write method where the bulk of the changes were made supports overstriking text with text, attributes with attributes, or entire cells with text+attributes. This is the backbone for FillConsoleOutputCharacter (writes text only) and FillConsoleOutputAttribute (writes attributes only), and it also handles every other write operation (which is usually both text and attributes simultaneously.)

The caveat noted above is for the following scenario:

ORIGINAL
| Hello World                |
| RED  | GREEN               |
0                           29

Insert attributes only: BLUE (length=5); position 0

NEW
| Hello World                |
|BLUE | RED  | GREEN         |
0                           29

Since it only changes the attributes, " Hello" would remain, but the colors would shift underneath it such that " Hell" was in blue and "o World" was in red. Nothing in the literature suggests that we'll need to support it. Every change to the buffer coming from VT is accompanied by text.

² The Win32 console supports drawing borders around every cell; the bottom border is distinct from underline, the top border is recycled for overline, and the left/right borders are used in some CJK composition scenarios.

vttest

edit: thanks for reviewing!

[Dustin L. Howett (DHowett)]

This implementation will absolutely hork UnicodeStorage. It tracks non-dbcs high unicode by row/column index. Shuffling the backing store for the row but not updating UnicodeStorage will make it blast into outer space.

We are likely overdue for a UnicodeStorage redesign.

[paul vixie (vixie)]

i don't have a Windows software build environment, so i'm not in a position to test this PR. is there a .EXE i can try?

[Dustin L. Howett (DHowett)]

paul vixie (@vixie) sorry, we ended up running into some of our longstanding code rot and this work is, as a result, a bit delayed. I'm keeping it on the docket for 2021 such that we can finally put it to bed, and will keep this thread updated.

Once we're in a place where what you test is close to what we want to ship, I'll share a build of conhost/openconsole (without any of the terminal bits) to test with.
d

[Marcel Hellwig (hellow554)]

What happened to that poor bot?

[Mike Griese (zadjii-msft)]

Marcel Hellwig (@hellow554) This branch is pretty stale, I'm betting the spellcheck rules in this branch don't quite work for the updated version of the bot

[WSLUser]

Any chance this sequence will be updated with the latest changes? I think this would be a useful sequence to support.

[Dustin L. Howett (DHowett)]

So, the significant blocker for IRM is that we didn't have a good way to shift our backing store character-by-character to handle insertion (due to some legacy baggage that we had accumulated over time ("UnicodeStorage", which is a hash map of coordinate positions to non-UCS-2 code units)). That's being actively worked on in #8000. Once that lands, we can dust this off.

[Mike Griese (zadjii-msft)]

Dustin L. Howett (@DHowett) since the implementation of this is likely to change aggressively after #8000 lands, should we close out this draft for now?

[Dustin L. Howett (DHowett)]

Nah, you're totally right. I can't land this in its current form, and it's just clogging the list up until it can land. I'll keep the branch.

[paul vixie (vixie)]

Mike Griese wrote on 2022-07-05 07:08:

since the implementation of this is likely to change aggressively after #8000 lands, should we close out this draft for now?

if Terminal would stop identifying itself as "xterm" and instead answer as "vt100", then we could postpone implementing IRM and SM/RM until after #8000 lands. Terminal isn't "xterm" and that problem can be fixed several ways.

…

-- P Vixie

[paul vixie (vixie)]

given this has been open for more than two years and that it's blocked on #8000 which has also been open for two years, can i ask that the TERM variable be set to something other than "xterm-256color" please? that's false, since xterm has insert mode, and this terminal emulator does not. if you won't change it to "vt220" then at least add a configuration setting for TERM. you can revert this when and if this terminal emulator ever actually becomes xterm compatible.