flashing to darp5 breaks "7" key

[Thulinma]

Hey guys,

I hope this is the right place to report issues / ask for help. If not please do point me elsewhere!
I've also e-mailed firmware@system76.com about this a few days ago, but figured it might be good to put it here since it's been a few days and I'm not sure if the right people read that inbox. 🤔

Thanks for your time reading and/or helping me with this! I know the darp5 is an older model and will likely never be officially supported for the open firmware / open ec. The fact that I am able to do this at all is definitely making you guys look really good! Good enough that some of my colleagues/friends are planning to also get system76 machines in the near future. 💪

Background info:

My darp5 has been smoothly running coreboot for over a year now, and I've been keeping it up to date with the master branch of firmware-open. (As of last Monday, it was up to date to latest master, at least.) The only trouble I've had is that it often stalls for a while initially when cold-booting. A bit annoying, but since it only takes a few seconds I didn't really mind.
Wanting to also have open EC, and noticing darp5 in the supported models list of this repo, I figured I'd get the right parts and give it a try. (I also hoped it would fix the stalling issue, of course.)
Unfortunately my Arduino was not using the right USB controller (it was misadvertised in the listing), but changing the references in the source from ttyACM0 to ttyUSB0 made everything work without a hitch. Might be useful to mention in the docs, that. I'm sure you guys just haven't had a guinea pig to test that on. It works 😉!
I was able to connect everything up and flashed the EC successfully. (Next time I do this I'll take some pictures; might be useful for the documentation!) Afterwards I used the flash_internal method to flash it once more, just to make sure nothing went wrong because of the different Arduino USB controller. That also went fine, and the laptop remained (mostly) functional after both flashes.

The problems:

Once on the open EC code, my "7" key stopped functioning (and only the regular 7 key, the numpad 7 and all other keys still worked fine).
In addition, every ACPI-related call (battery status, changing display backlight brightness, changing fan speed, changing keyboard backlight color/brightness) would stall for a long time (on the order of minutes) and then sometimes succeed, sometimes fail. ⬅ This turned out to be caused by an old fan control daemon. See this comment for details.

Eventually it started to annoy me enough that I decided to flash back the original EC from the backup (again using the Arduino). That went fine as well, and all functionality was immediately restored to normal.
Strangely... the stalling boot problem is now completely gone without a trace and it boots up lightning fast every time 🤔. Oh well, a win is a win.

Any ideas regarding the ACPI calls stalling and the 7 key not working anymore? Especially the 7 key being the only broken key is absurdly strange to me. I'd love to go back to open EC if possible, and I don't mind testing/trying a few things. I'm comfortable with the flashing procedure now, and verified my backup works, so none of that should be an issue. 👍

[leviport]

Is your keyboard an ISO or ANSI layout? I vaguely recall offering ISO layout keyboards on darp5, but I'm not sure of it. My darp5 never had issues with the 7 key and mine is ANSI.

[Thulinma]

ANSI. So no difference there.
I also meant to include these messages from the kernel ring buffer, but forgot:

[   92.109231] ACPI Error: Timeout from EC hardware or EC device driver (20200925/evregion-274)
[   92.109266] ACPI Error: Aborting method \_SB.BAT0.UPBS due to previous error (AE_TIME) (20200925/psparse-529)
[   92.109271] ACPI Error: Aborting method \_SB.BAT0._BST due to previous error (AE_TIME) (20200925/psparse-529)
[   92.109278] ACPI Error: AE_TIME, Evaluating _BST (20200925/battery-576)
[   94.622543] ACPI Error: AE_TIME, Returned by Handler for [EmbeddedControl] (20200925/evregion-264)

This sequence loops roughly once per second (probably triggered by battery status polling) and has similar messages for the other stalling ACPI calls as well.

I'm not sure if this may be relevant or not, but I'm also running this as a system daemon for fan control: https://gitlab.com/francois.kneib/clevo-N151ZU-fan-controller
Is it possible that application somehow disturbs the EC...? (On the closed-source EC code it gives me very nice and precise control over the fan speed. I honestly forgot it was running until just now.)

[Thulinma]

Update: today I flashed the open EC again, and tried disabling the clevo fan controller daemon.
This indeed fixes the ACPI issues. Might be worth making a note somewhere that running the closed EC fan control software messes up the open EC!

I tested everything I could think of, and all functionality seems to work (fan RPM display, temperature readings, battery status, backlight and keyboard light control, etc).

... except my "7" key is still broken. 🤣
EDIT: Oh, and thunderbolt doesn't seem to work - it detects devices/docks being plugged in, but boltctl doesn't show them and reports thunderbolt security is set to "none" (shouldn't that be hardcoded to "user" on open firmware..?) Nevermind, thunderbolt works perfectly if I install the system76-power package, which I hadn't...

This is now usable enough that I can stick to using the numpad 7 key instead for now; so I'll stay on the open EC to help debug this. Anything I can do to track down what might be wrong..?
EDIT: I've now confirmed that the "7" key generates no output whatsoever when using "showkeys" on a text terminal. Other keys work as normal. So... definitely something in the EC, and not my configuration, I would say. 🙂

Here are some pictures of the flashing process, which I hereby put into the public domain for you to use however you see fit (e.g. some helpful imagery for in the flashing readme file, perhaps?):

Disconnecting the battery

Showing how the arduino is connected

Aligning the ribbon cable to pin 1

Grounding the two laptops using USB-C cable, connecting Arduino to the "host" laptop to flash the "guest" laptop.

[Thulinma]

Sorry for the spam, but I think this warranted another post:
I found and used the system76_ectool console command, which showed me that pressing 7 actually outputs:

KB 11, 2, 0 = 0x043D, 1
KB 11, 2, 0 = 0x043D, 0

That's the culprit! That 0x043D should've been 0x003D. Interestingly, pressing fn+7 works totally fine..!?
And, finally, the part that proves it: running system76_ectool keymap 0 11 2 0x3D restores the 7 key to perfect working order. 🤔

I'm going to guess that for some reason, my keymap was not set correctly after the flash...? Perhaps zero-bytes don't always get written or something like that..? Either way, this seems fixed for now (I'm guessing/hoping from reading the code that keymaps persist across resets/flashes).

Leaving this open so it will be seen by somebody, but feel free to close after reading. 👍

EDIT: Apparently, it does not persist. I ended up adding a line to the firmware to hardcode the key for now (if (i == 11 && j == 2){key = 0x3D;} in kbscan.c). Ugly, but it works. 🤷

[crawfxrd]

AFAIK this was fixed around when system76/firmware-open#260 was merged. Will look into it more if it's still found on the release PR.

[Thulinma]

I'm currently running on a May 2022 build of the firmware and have not checked if the issue went away... (Still contains my patch)
I'll upgrade to latest with my one line patch removed and will report back! Hopefully today, if not within the next few days. 🤞

[Thulinma]

Okay, took me way longer than that to actually test this (almost a year..? oh man 👀), but I can happily confirm that the key no longer breaks in the current latest version. 👍