Autonomous QA for a drone flight cockpit, built for the FlytBase AI hackathon (kit).
- Mutation agent (
qa/mutator.ts): an LLM reads the cockpit source and the product brief and proposes realistic bugs as search/replace patches, each with a sealed answer key. - QA harness (
qa/): Playwright drives the cockpit, an independent socket.io client records drone telemetry as ground truth, and seven deterministic checks (K1–K7) compare the screen against it. No LLM calls in the test loop, so every verdict repeats. - Blind scoring (
qa/mutants.ts): the answer key is opened only after a run, to mark each bug caught, missed or a false alarm.
11 of 13 injected bugs caught (8/8 hand-written, 3/5 agent-generated), with 0 false alarms across 3 clean or harmless builds. The two misses are explained in the report: both need a check that the unmodified kit can't pass.
The harness also found real bugs in the original kit. The worst: after the link dropped, the cockpit kept showing "connected, in flight, 10 m/s" on 12-second-old data while the drone kept flying.
See qa/README.md. The recordings and findings behind the report are in qa/recordings.