Repository navigation
fix(v1): a real body refuses a LeRobot checkpoint that does not say its norm mode - #4804
rmncardoso wants to merge 2 commits into
Conversation
…ts norm mode
A checkpoint stores no norm mode, so Policy("<ckpt>") took the
registry's so101 label, so101-range-gripper-percent-v1, while the
pinned LeRobot defaults SO arms to degrees (use_degrees=True since
0.4.4). On the fake Feetech bus a degrees checkpoint's +90 deg landed
+20.4 deg off on shoulder_lift and +72.0 deg off on the full-turn
wrist_roll. On a real body LeRobotPolicy.attach now refuses a
checkpoint given neither norm= nor semantics=, before LeRobot is
imported and before anything reaches the bus, and the error names both
options and both modes. An empty one says no mode either:
semantics="" fell back to the registry's label and norm={} to radians.
A simulated body keeps the registry default.
yinsong1986
left a comment
There was a problem hiding this comment.
Summary
Adds a fail-closed guard at the top of LeRobotPolicy.attach: when the robot is a hardware body (Robot.is_real, i.e. port=/driver= given, which also covers Mesh.connect() robots) and the policy was built with neither a non-empty norm= nor a non-empty semantics=, it raises ValueError naming both options and both SO norm modes. The check sits before lerobot() is imported, before server_config resolves (and for a Hub id, downloads) the checkpoint, and before bind reaches units/requirements, so nothing touches the bus. A simulated body keeps the registry default, so existing sim tests and docs are unchanged. The _kw dict always carries both keys, every attach caller (run/loop.py, run/lockstep.py via bind) passes a Robot, and Description.load is not cached, so the early-return on contract.description is robot.description cannot let a sim-attached policy slip onto a different real body. No new public API, wire format or persisted schema; the behaviour change on hardware is an explicit refusal with the migration spelled out in the message, not a silent default flip. Changelog fragment is well-formed (one ###, no ##).
What's good
- Guard placement is the cheapest possible point: zero network, zero torch import, zero bus frames on refusal, and the default-env test proves it without the
lerobotextra. tests/test_units_lerobot_modes.pycovers the three undeclared shapes ({},semantics="",norm={}) and assertss.frames == []; the LeRobot-env test asserts bothcontract is Noneand_runner is Noneon refusal, then that each declared form actually drives aGoal_Positionsync write.- Scope discipline: the served path (
server_config/ZenohPolicy) is explicitly left alone and called out as a question rather than quietly widened.
Verification suggestions
- The LeRobot-env test needs the
remoteextra:hatch run lerobot:test tests/policy/test_lerobot.py tests/test_units_lerobot_modes.py(expect 17 passed). I could only run the default-env side here (tests/test_units_lerobot_modes.py: 7 passed;tests/test_layout.py,tests/test_public_surface.py,tests/test_package.py: green;ruff check/ruff format --check/mypyclean on the three changed source/test files). python scripts/assemble_changelog.py --checkflags only the pre-existing0000-...placeholder ondev, not this PR's4804-...fragment.
|
@yinsong1986 your approval is newer than the last push, |
Fix to part 4 of #4749 · roadmap #3818
In one paragraph
A LeRobot checkpoint stores no norm mode, so
Policy("<checkpoint>")takes the registry's label,so101-range-gripper-percent-v1for an SO-101 (registry/robots.json:401-404,policy/lerobot/contract.py:138). LeRobot has recorded SO arms in degrees by default since v0.4.4, the pinned 53feaef3 included. Read as range, a degrees checkpoint commanded to +90° landed up to 72° off on a hardware so101 over the repo's fake Feetech bus ([1] row 1). #4749 says a deployment's units are checked "before a session exists"; a checkpoint's were guessed, and a degrees default would misread range checkpoints instead ([1] rows 3, 19). NowLeRobotPolicy.attachrefuses, on a real body, a checkpoint given neithernorm=norsemantics=(or an empty one), before LeRobot is imported, the checkpoint fetched or the bus touched, naming both options and modes. A simulated body keeps the registry's default. AMesh.connect()robot is real even for a simulated device ([2]), and a spec string alone cannot declare a mode ([1] row 14).Measured
pytest tests/test_units_lerobot_modes.py, default env (the required check)ImportError: it loads LeRobot), 4 passedpytest tests/test_units_lerobot_modes.py tests/policy/test_lerobot.py, LeRobot envMissing 'type' field in config.json, 1DID NOT RAISE), 13 passednorm=ValueError, 0 bus framessemantics/norm,units=,start(), a Hub id,from_spec, sim first,twin=Truenorm=degrees,semantics=range, a simulated so101Mesh.connect(), nonorm=: a fake-bus so101, a simulated so101ValueError, 0 bus writespytest, whole suiteruff check,ruff format --check,mypy,lint-importsgit merge-treewith #4784, #4790, #4794-#4803, each; thentests/test_layout.pyand the second rowNot in this PR
The served path:
server_config()withoutnorm=still takes the registry's label, which aZenohPolicyclient then uses (policy/lerobot/remote.py:67).Questions
Should
server_configrefuse an undeclared checkpoint too, as LeRobot'sServerConfigdoes? Simulated arms then neednorm=: 8 of 10tests/policy/test_lerobot.pytests fail (Evidence).Evidence
How it was run. macOS 26.7.1 arm64, Python 3.12.10, pytest 9.1.1, on a shared machine (load average 17 to 22), so no timing is claimed. The default env has no LeRobot, like CI's required check (
hatch run test); the LeRobot env has LeRobot 53feaef3, the commitpyproject.tomlpins, and torch 2.11.0. No hardware:FakeSerial(tests/test_drivers.py) is six STS3215 servos in memory, camera frames are synthetic,HF_HUB_OFFLINE=1andSTRANDS_ROBOTS_LIVEwas unset. dev is agit archiveof cc15d9a'sstrands_robots/,tests/,tests_integ/andpyproject.toml; "dev with this PR's tests" is the same plus this PR's two test files. This PR's change is one commit, dc83ef3, on cc15d9a; the changelog fragment is a second commit. Everypytestbelow isnice -n 10 pytest -q -p no:cacheprovider --timeout=300 --basetemp=<a scratch dir> ..., run from the tree's root.Red first.
pytest tests/test_units_lerobot_modes.py -rA: dev with this PR's tests3 failed, 4 passed. Each case oftest_a_real_arm_refuses_a_checkpoint_that_does_not_say_its_norm_modeendsImportError: LeRobot policies need the lerobot extra: ...atpolicy/lerobot/contract.py:30, so dev went on to load LeRobot. This PR:7 passed. The test needs no LeRobot, so the required check, which skipstests/policy, covers the refusal.pytest tests/test_units_lerobot_modes.py tests/policy/test_lerobot.py -rA: dev with this PR's tests4 failed, 13 passed. The three cases failRegex pattern did not matchonMissing 'type' field in config.json of <tmp>/...(dev read the checkpoint), andtest_a_real_arm_runs_a_checkpoint_only_once_its_norm_mode_is_declaredfailsDID NOT RAISE ValueError(tests/policy/test_lerobot.py:64). The guardtest_a_local_checkpoint_drives_the_arm_through_run(a simulated arm keeps the registry's label) passes on both. This PR:17 passed.Code reading at cc15d9a and LeRobot 53feaef3, unless a GET is quoted.
"lerobot_norm": {"*": "range_m100_100", "gripper": "range_0_100"}(registry/robots.json:401-404), whichsemantics_formaps toso101-range-gripper-percent-v1throughunits/semantics.json.server_configtakessemantics or semantics_for(description, norm)(policy/lerobot/contract.py:138), sosemantics=""gets the registry's label too, andnorm={}getsradians-v1(units/semantics.py:36-38).use_degrees: bool = True(src/lerobot/robots/so_follower/config_so_follower.py:42), andnorm_mode_body = MotorNormMode.DEGREES if config.use_degrees else MotorNormMode.RANGE_M100_100(so_follower.py:72). By GET at each of LeRobot's 11 tags (gh api "repos/huggingface/lerobot/tags?per_page=100"),gh api "repos/huggingface/lerobot/contents/<path>?ref=<tag>" -H "Accept: application/vnd.github.raw" | grep -n "use_degrees: bool"givesFalsefrom v0.3.2 to v0.4.2 (robots/so101_follower/config_so101_follower.py:41) and at v0.4.3 (robots/so_follower/config_so_follower.py:43), andTruefrom v0.4.4 (:43) to v0.6.1 (:42). LeRobot's ownServerConfigsays "Explicit feature conventions are required when checkpoint metadata is incomplete." (src/lerobot/remote_inference/configs.py:75) and refuses an emptysemantics(:87-88).runchecks its arguments and then binds, before anything else (robot/run/loop.py:48-52), andbindcallsattach(robot)before units, requirements and reset (robot/run/bind.py:64-65; units at:71). The check is the first step ofattachafter its same-robot return (policy/lerobot/local.py:53-58here), beforelerobot()(:59) and beforeserver_configresolves the checkpoint (:62; a Hub id is downloaded there,contract.py:137). EachRobotloads its ownDescription(robot/robot.py:54,description/description.py:102-116), so a policy attached to a simulated arm is checked again on a real one ([1] row 17).is_realisport=ordriver=given (robot/robot.py:56), the flag the operator gate reads (robot/gate.py:31), andMesh.connect()returnsRobot(name, driver=body, twin=False, id=device_id)(mesh/mesh.py:196).make(spec)with no keywords (policy/callable.py:38,policy/factory.py:76-80), and a checkpoint spec takes no query (factory.py:10,:58-63), sofrom_spec, or a string passed torun, cannot declare a mode.range_m100_100spreads -100..100 over a joint's calibrated sweep of R counts, and degrees put 360 on 4095 counts, so +v degrees read as range lands near v · R · 360 / (200 · 4095) degrees. With the probe's sweeps (R = 2534, 2789, 2234, 2425, 4095, shoulder_pan to wrist_roll) that is +10.2, +20.3, −1.6, +5.9, +72.0 off at +90; [1] row 1 reads back the integer goal counts as +10.2, +20.4, −1.6, +6.0, +72.0. The error is negative below R = 2275.[1] A hardware so101 on the fake bus, LeRobot env, from a checkout:
HF_HUB_OFFLINE=1 PYTHONPATH=.:tests:tests/policy python units_probe.py | grep -v "^Loading weights". Onlyact_chunk(LeRobot'sPolicyRunner.predicton one frame,local.py:82-90) is replaced, by a fixed action; the checkpoint still loads, so each row is what binding, units, the calibrated conversion and the driver write for that action. The goal counts are read back with LeRobot's ownFeetechMotorsBus._normalize, on a bus that is never connected. LeRobot's download is replaced by a stand-in that records the ask and fetches nothing.dev cc15d9a:
this PR:
Row 10 is the same on both:
norm=must match asemantics.jsonentry exactly, gripper included, and a partial one is refused atcontract.py:138, before the weights load or the bus is touched, but after:137has read the checkpoint's config (a Hub id is downloaded there). Rows 15 and 16 are the same on both:units="lerobot"takes the registry's range mode over a declared degreesnorm=, andsemantics=wins overnorm=. Row 19 is the same on both: a range recording run with a degreesnorm=, as a degrees default would run every range checkpoint, lands -9.2, -16.6, +1.7, -5.6 and -40.0 off.[2]
Mesh.connect(), LeRobot env, from a checkout:HF_HUB_OFFLINE=1 PYTHONPATH=.:tests:tests/policy python units_mesh.py | grep -v "^Loading weights". OnlyGoal_Positionwrites are counted: the device's state stream reads its bus on its own schedule.dev cc15d9a:
this PR:
The question's alternative. This PR plus a refusal in
server_configbefore the checkpoint is read (ahead ofcontract.py:137):pytest tests/policy/test_lerobot.py -rA, LeRobot env:8 failed, 2 passed. The two that pass aretest_a_real_arm_runs_a_checkpoint_only_once_its_norm_mode_is_declaredandtest_unknown_semantics_are_refused_naming_the_table; the in-process sim run (test_a_local_checkpoint_drives_the_arm_through_run),test_server_config_is_built_from_the_robot_and_the_checkpointand the six tests that serve the checkpoint fail.Merges.
git merge-tree --write-treeof dc83ef3 with #4794 (fe59b12), #4795 (0d37932), #4796 (2805bfa), #4797 (06e97cf), #4798 (b72cb68), #4799 (1e0a3a3), #4800 (14b2bb1), #4801 (3c2a427), #4802 (6c6ba4c), #4803 (f439675), #4790 (b4b0da9) and #4784 (73f8102), one at a time, heads by GET at 22:50 UTC: no conflict, and none of them touches this PR's three files. On each merged tree (agit archiveof it),pytest tests/test_layout.py:4 passed, the largest files at 250 lines, none of them this PR's (local.pyhas 94); and, LeRobot env,pytest tests/test_units_lerobot_modes.py tests/policy/test_lerobot.py:17 passed.Checks, from the checkout at dc83ef3, default env unless named.
ruff check --no-cache strands_robots tests tests_integ:All checks passed!.ruff format --no-cache --check strands_robots tests tests_integ:188 files already formatted.mypy strands_robots:Success: no issues found in 123 source files.lint-imports --no-cache:Contracts: 8 kept, 0 broken.local.pygrows from 87 to 94 lines,tests/policy/test_lerobot.pyfrom 170 to 193 andtests/test_units_lerobot_modes.pyfrom 48 to 62;pytest tests/test_layout.py:4 passed.pytest -q -p no:cacheprovider --timeout=300 -rs --deselect tests/mesh/test_mesh_multiprocess.py::test_two_devices_and_a_controller_discover_each_other_zero_config_then_a_kill_leaves:647 passed, 32 skipped, 1 deselected. The deselected test opens a barezenoh://, which scouts by multicast on the LAN (mesh/transports/zenoh.py:3). Every skip is an environment one: Isaac 16, Newton 5, LeRobot 10, the opt-in live model 1.pytest tests/policy tests/test_policy_factory.py tests/test_units_lerobot_modes.py tests/test_units_lerobot_calibrated.py tests/recording/test_record_checkpoint_units.py: dev36 passed, this PR40 passed. Only this PR's tests run a LeRobot checkpoint on a hardwareRobot: the fake-bus tests intests/test_policy_factory.pyandtests/test_units_lerobot_calibrated.pyuse their own policies, andtests/recordingruns checkpoints on simulated arms (code reading).Docs (code reading). No example on dev or in #4781 (f70a53a) runs
Policy("<checkpoint>")in-process on a hardware robot:docs/index.md:43runsPolicy("lerobot/smolvla_base")onRobot("so101"), andnotebooks/02_policies.ipynbits checkpoints onRobot("so101"), both simulated. #4781's "The unit conversion is automatic: the policy says what it speaks, the robot converts." (docs/robots.md:31-32) holds on a real arm oncenorm=orsemantics=says what the policy speaks.Refs #3818