User Story #81 » README.md
Black-box screen capture — instrumentation for the #69b-i walk
Turns the phone screen into a machine-readable data channel so a plain screen recording of a walk —
captured while your usual mofe-trace ndjson is recording — carries the one thing the ndjson can't:
the raw IMU stream (for odometry / observer position), plus a tEngineMicros sync key so the two align.
One carried-phone recording then yields, for #69b-i: authoritative ranges (ndjson) + IMU-odometry position
(overlay), with no change to the on-device ndjson format.
What's in this drop
| file | layer | status |
|---|---|---|
0001-…-black-box-codec.patch |
:common (diag/BlackBoxCodec,BlackBoxSchema + test) |
✅ jvmTest-verified on 9976b41
|
0002-…-wire-format-golden.patch |
:common (cross-language golden) |
✅ jvmTest-verified; locks Kotlin ⇄ python |
androidApp-overlay-72.patch |
androidApp (the on-screen overlay) |
⚠ at-risk — not compilable here; device confirms |
mofe_blackbox_decode.py |
offline agent tool | ✅ --selftest passes end-to-end through real H.264
|
Verification done in-sandbox: the codec survives real H.264 (libx264 crf 23 and 28: 30/30 frames exact,
fail-closed) rendered exactly as the overlay draws it (0.04·min margin, non-square cells, 3 fiducials) and
decoded via the resolution-exact path; and the on-device Kotlin encoder is byte-identical to the python
decoder (golden test). Unprovable here (device only): that the androidApp overlay compiles and renders,
and the real phone-screen-recorder round-trip.
Capture protocol (the walk)
You already produce the ndjson via the mofe-trace.on sentinel. With this build, that same sentinel also
draws the black-box overlay — so your capture procedure barely changes:
-
Mesh up + converged — the 4 anchors as surveyed (A1/A2 near wall, A3 across the room, A4 the red cup),
mesh at QUORUM. -
On the observer Pixel (
c158fc), enable trace as you always do — createmofe-trace.onin the app's
external files dir (…/Android/data/<pkg>/files/mofe-trace.on). This now arms both the ndjson recorder
and the overlay. Launch the app → the screen fills with a black/white grid + 3 white corner squares
(that's the overlay; it's not meant to be looked at — the screen is the data channel now). - Start the phone's screen recorder.
-
Carry the phone on a walk — ~60–90 s, natural pace, holding it in a consistent orientation. Cover
varied bearings: through the room interior and close to the walls, especially near the A3 corner
(the biased anchor) and along its wall — that spread is what lets the observer ranges fix A3's
wall-tangential error. -
(Optional but ideal) have a second phone/camera film the walk against the tiled floor — the tile
grid is a metric reference I can use to extract your absolute position per frame (removes IMU drift as a
confound). No floor marks needed. -
Stop the screen recorder; remove
mofe-trace.on. -
Send me two raw files (I do all the computation): the screen-recording
.mp4, and the
files/mofe-trace/mofe-trace-c158fc-*.ndjson. Plus the room video if you shot one.
What I do with it
mofe_blackbox_decode.py capture.mp4 --ndjson trace.ndjson --out records.jsonl →
decode each video frame (resolution-exact grid registration) → per-frame {tEngineMicros, frameIdx, raw-IMU buffer, pose} → merge with the ndjson ranges on the shared engine clock → integrate IMU odometry (or the
tile-floor trajectory from the room video) for independent observer position → feed those as observerConstraints
into the leader refine on the biased 4-anchor graph → report whether A3 pulls back toward its surveyed
truth vs the un-augmented solve. That's the go/no-go for building #69b-i.
Notes
- The overlay replaces the visible scene during capture — expected; it's instrumentation, not a view.
- If the screen recorder crops (rare), decode with
--fiducial; a full-screen recording uses the exact
resolution path automatically. -
mofe_blackbox_decode.py --selftestreproduces the H.264 verification on any machine with numpy + ffmpeg
(pip install imageio-ffmpegbundles one). - Everything here is on branch
claude/mofe-aiming-calibration-y2ckrl(pushed). Wire format is locked by
BlackBoxWireFormatTest, so the on-device encoder and this decoder can't drift.
- « Previous
- 1
- 2
- 3
- Next »