Problem reading large data on stdin (v4.0.1)

[vascolp]

Hi!

I am trying to convert my RemoteBlaBla application to pybricks v4.0.1.
RemoteBlaBla involves a lot of communication between a PC or android device and the hub. The hub sits there listening in stdin.
Almost everything seems to be working perfectly except when the hub has to read a large ammount of data.
Something like
msg=stdin.buffer.read(128)

This was never a problem with v3 but now it seems to be a problem with things bigger than around 20 bytes.
The PC side gets a (129, 'GATT Protocol Error: Application-specific Error 0x81')
So apparently the hub is busy (0x81) and is not able to read all that data...

I don't have a simple test program to share with you yet (I have a complex test program...), but I know you have changed something around stdin.
I was wondering if you could give me some hints about this before starting to create a simple test program which is not that simple...

Thanks and best regards!
VascoLP

[vascolp]

So here is a sample code. I would say something is not ok.
This sample code is based on this sample.
Basically it is the same thing but with some trash added to the communication between pc and hub.
In pybricks v3 everything works fine with big trash sizes, like 128 bytes.
In v4, if the total message is bigger than 20 (so trash size>17), the hub program fails.

Hub program:

    from pybricks.pupdevices import Motor
    from pybricks.parameters import Port
    from pybricks.tools import wait
     
    from usys import stdin, stdout
    from uselect import poll
     
    motor = Motor(Port.A)
     
    keyboard = poll()
    keyboard.register(stdin)
     
    while True:
     
        stdout.buffer.write(b"rdy")
     
        while not keyboard.poll(0):
            wait(10)
     
        # Read three bytes plush extra trash
        trash_size=17
        cmd = stdin.buffer.read(3+trash_size)
        cmd=cmd[0:3]
     
        # Decide what to do based on the command.
        if cmd == b"fwd":
            motor.dc(50)
        elif cmd == b"rev":
            motor.dc(-50)
        elif cmd == b"bye":
            break
        else:
            motor.stop()

PC program:

    import sys
    import asyncio
    from contextlib import suppress
    from bleak import BleakScanner, BleakClient
     
    PYBRICKS_COMMAND_EVENT_CHAR_UUID = "c5f50002-8280-46da-89f4-6d8051e4aeef"
    async def main():
        if len(sys.argv) <2:
            print('pc.py <hub name>')
            return
        hub_name = sys.argv[1]
     
        main_task = asyncio.current_task()
     
        def handle_disconnect(_):
            print("Hub was disconnected.")
            if not main_task.done():
                main_task.cancel()
     
        ready_event = asyncio.Event()
     
        def handle_rx(_, data: bytearray):
            if data[0] == 0x01:  # "write stdout" event (0x01)
                payload = data[1:]
     
                if payload == b"rdy":
                    ready_event.set()
                else:
                    print("Received:", payload)
     
        device = await BleakScanner.find_device_by_name(hub_name)
     
        if device is None:
            print(f"could not find hub with name: {hub_name}")
            return
     
        async with BleakClient(device, handle_disconnect) as client:
     
            await client.start_notify(PYBRICKS_COMMAND_EVENT_CHAR_UUID, handle_rx)
     
            async def send(data):
                await ready_event.wait()
                ready_event.clear()
                await client.write_gatt_char(
                    PYBRICKS_COMMAND_EVENT_CHAR_UUID,
                    b"\x06" + data,  # prepend "write stdin" command (0x06)
                    response=True
                )
     
            print("Start the program on the hub now with the button.")
     
            trash=b'a'*17   # On v4.0.1 it works up until 17... after that it fails
            for i in range(3):
                await send(b"fwd"+trash)
                await asyncio.sleep(1)
                await send(b"rev"+trash)
                await asyncio.sleep(1)
                print(".", end="", flush=True)
     
            await send(b"bye"+trash)
     
            print("done.")
     
    if __name__ == "__main__":
        with suppress(asyncio.CancelledError):
            asyncio.run(main())

[BertLindeman]

Hi,

I needed to shift both programs 4 bytes to the left.

Ran on primehub:

python issue_2788_pc.py "prime b2"
Start the program on the hub now with the button.
...Received: bytearray(b'\r\n')
done.
Hub was disconnected.

and on Technichub:

C:\Users\bert\py\pybricks\issue_2788_stdin>python issue_2788_pc.py "hubA v4.0.1"
Start the program on the hub now with the button.
...Received: bytearray(b'\r\n')
done.
Hub was disconnected.

and a cityhub:

python issue_2788_pc.py "city v4.1.0b2"
Start the program on the hub now with the button.
...Received: bytearray(b'\r\n')
done.
Hub was disconnected.

The movehub has no usys so cannot test there.
And EV3 has no bluetooth yet.

These tests ran with trashsize=17

Test with trashsize=128 on the PC is strange on the hub side I see the program runs on, but the PC side states rather fast:

python issue_2788_pc.py "city v4.1.0b2"
Start the program on the hub now with the button.
Hub was disconnected.

The same with trashsize=128 on both sides.

With added debug prints before anf after send:

python issue_2788_pc.py "city v4.1.0b2"
Start the program on the hub now with the button.
about to send fwd trashsize= 128
Hub was disconnected.

PC side gets (or invents) a disconnect on the send fwd, while the hub still is running.

Running win11 Python 3.11.4

[vascolp]

Sorry about the 4 bytes offset...
18 will fail as much as 128. My feeling is when the message is larger than 20 bytes it fails.
This PC program does not handle the error properly, but with a bigger program I got a 0x81 error from the hub which supposedly means busy. Hope it helps!

[BertLindeman]

Updated the test scripts to determine the boundaries for WRITE_STDIN payload size (with help of Claude, as I am not really good in this Bluetooth coding).

Setup:

• hub-side program reads a fixed 3-byte command followed by a 2-byte little-endian length prefix, then reads exactly that many trailing bytes.
• PC-side script bisects the largest "trashsize" that still succeeds, per hub/firmware combination.

Results:

Hub	Firmware	Protocol	MTU	max_char_size	Largest working trashsize	Boundary (total stdin payload)	Failure mode
Technic Hub	v3.6.1	1.4.0	158	155	149	154 (= max_char_size - 1)	0x0B Attribute Not Long
Technic Hub	v4.0.1	1.5.0	158	155	15	20	0x81 BUSY
City Hub	v4.1.0b2	1.6.0	158	155	15	20	0x81 BUSY
Prime Hub	v4.1.0b2	1.6.0	527	512	58	63	0x81 BUSY

Conclusions:

• On protocol 1.4.0 the boundary matches max_char_size - 1 exactly, and the failure mode is the documented ATT-level rejection for exceeding a single write (no long-write support). This is the expected/correct behaviour per the profile spec.
• Starting at protocol 1.5.0 the actual usable payload drops far below max_char_size, independent of MTU. Failure mode changes to the undocumented app-specific BUSY (0x81).
• The 1.5.0+ boundary is chip/hub-dependent, not derivable from max_char_size or MTU: ~20 bytes on City/Technic, ~63 bytes on Prime.
• max_char_size as read from the hub capabilities characteristic is no longer a reliable upper bound on 1.5.0+. Anything relying on it for WRITE_STDIN sizing will hit spurious BUSY errors well under the advertised limit.
• Reproducible with two small scripts:

hub-side length-prefixed stdin reader - issue_2788_stdin_bisect.py

from pybricks.pupdevices import Motor
from pybricks.parameters import Port
from pybricks.tools import wait

from usys import stdin, stdout
from uselect import poll

motor = Motor(Port.A)

keyboard = poll()
keyboard.register(stdin)

while True:

    stdout.buffer.write(b"rdy")

    while not keyboard.poll(0):
        wait(10)

    # Fixed 3-byte command.
    cmd = stdin.buffer.read(3)

    # 2-byte little-endian length prefix tells us exactly how much
    # trailing "trash" data to read, so PC can send any size per
    # message without hub/PC needing to agree on it in advance.
    lenb = stdin.buffer.read(2)
    trashlen = lenb[0] | (lenb[1] << 8)
    if trashlen:
        stdin.buffer.read(trashlen)

    if cmd == b"fwd":
        motor.dc(50)
    elif cmd == b"rev":
        motor.dc(-50)
    elif cmd == b"bye":
        break
    else:
        motor.stop()

PC-side bisector - issue_2788_pc_bisect2.py

import sys
import asyncio
from contextlib import suppress
from bleak import BleakScanner, BleakClient
from bleak.exc import BleakError
import struct

PYBRICKS_COMMAND_EVENT_CHAR_UUID = "c5f50002-8280-46da-89f4-6d8051e4aeef"
PYBRICKS_HUB_CAPABILITIES_UUID = "c5f50003-8280-46da-89f4-6d8051e4aeef"
SW_REV_UUID = "00002a28-0000-1000-8000-00805f9b34fb"

async def main():
    if len(sys.argv) < 2:
        print('pc.py <hub name>')
        return
    hub_name = sys.argv[1]

    main_task = asyncio.current_task()

    def handle_disconnect(_):
        print("Hub was disconnected.")
        if not main_task.done():
            main_task.cancel()

    ready_event = asyncio.Event()

    def handle_rx(_, data: bytearray):
        if data[0] == 0x01:
            payload = data[1:]
            if payload == b"rdy":
                ready_event.set()
            else:
                print("Received:", payload)

    device = await BleakScanner.find_device_by_name(hub_name)
    if device is None:
        print(f"could not find hub with name: {hub_name}")
        return

    async with BleakClient(device, handle_disconnect) as client:
        await client.start_notify(PYBRICKS_COMMAND_EVENT_CHAR_UUID, handle_rx)

        print("MTU:", client.mtu_size)
        sw_rev = (await client.read_gatt_char(SW_REV_UUID)).decode()
        print("Pybricks protocol:", sw_rev)

        caps = await client.read_gatt_char(PYBRICKS_HUB_CAPABILITIES_UUID)
        if len(caps) == 10:
            max_char_size, feature_flags, max_user_program_size = struct.unpack_from("<HII", caps)
        else:
            max_char_size, feature_flags, max_user_program_size, num_slots = struct.unpack_from("<HIIB", caps)
        print("max_char_size:", max_char_size)

        async def wait_ready(timeout=5.0):
            try:
                await asyncio.wait_for(ready_event.wait(), timeout=timeout)
                return True
            except asyncio.TimeoutError:
                return False

        async def try_send(trashsize, cmd=b"fwd"):
            if not await wait_ready():
                input(
                    f"\n  hub not responding.\n"
                    f"  Restart the hub program, then press Enter to continue..."
                )
                if not await wait_ready(timeout=30.0):
                    print("  still no response, aborting.")
                    return False
            ready_event.clear()
            trash = b'a' * trashsize
            data = cmd + struct.pack("<H", trashsize) + trash
            try:
                await client.write_gatt_char(
                    PYBRICKS_COMMAND_EVENT_CHAR_UUID,
                    b"\x06" + data,
                    response=True,
                )
                print(f"  trashsize={trashsize:4d}  total stdin payload={len(data):4d}  OK")
                return True
            except BleakError as e:
                print(f"  trashsize={trashsize:4d}  total stdin payload={len(data):4d}  FAIL: {e}")
                # Nothing reached the hub -- it's still waiting on the same "rdy"
                # cycle as before, so restore the ready signal instead of waiting
                # for a new notification that will never come.
                ready_event.set()
                return False

        print("Start the program on the hub now with the button.")
        if not await wait_ready(timeout=60.0):
            print("no response from hub, aborting.")
            return

        lo, hi = 0, 260
        last_good = 0
        while lo <= hi:
            mid = (lo + hi) // 2
            ok = await try_send(mid)
            await asyncio.sleep(0.3)
            if ok:
                last_good = mid
                lo = mid + 1
            else:
                hi = mid - 1

        print(f"\nLargest working trashsize: {last_good}  (total stdin payload: {last_good + 5} bytes)")
        # +5 = 3-byte cmd + 2-byte length prefix, trash bytes not included in that count

        await try_send(0, cmd=b"bye")
        print("done.")

if __name__ == "__main__":
    with suppress(asyncio.CancelledError):
        asyncio.run(main())

[dlech]

It looks like we broke something. I think @laurensvalk mentioned he was looking at something like this recently though. So maybe there is a fix in the works.

[laurensvalk]

Excellent analysis by @vascolp and @BertLindeman. Thanks! The fix is to make our stdin buffer match the advertised write size. We'll fix this in the firmware and Pybricks Code.

And we can take that as an opportunity to clear up the meaning of various internal buffer sizes.

[laurensvalk]

This is fixed in the latest Pybricks Beta and beta firmware. Now you can paste the following into the REPL and it should execute without missing characters:

data = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
print(data)
len(data)

[BertLindeman]

Tested and works on
Pybricks Beta v3.1.0-beta.4 with firmware v4.1.0b3

Great