Project

General

Profile

User Story #81 » README.md

knight8241, 08/07/2026 18:33

 

Black-box screen capture — instrumentation for the #69b-i walk

Turns the phone screen into a machine-readable data channel so a plain screen recording of a walk —
captured while your usual mofe-trace ndjson is recording — carries the one thing the ndjson can't:
the raw IMU stream (for odometry / observer position), plus a tEngineMicros sync key so the two align.
One carried-phone recording then yields, for #69b-i: authoritative ranges (ndjson) + IMU-odometry position
(overlay), with no change to the on-device ndjson format.

What's in this drop

file layer status
0001-…-black-box-codec.patch :common (diag/BlackBoxCodec,BlackBoxSchema + test) ✅ jvmTest-verified on 9976b41
0002-…-wire-format-golden.patch :common (cross-language golden) ✅ jvmTest-verified; locks Kotlin ⇄ python
androidApp-overlay-72.patch androidApp (the on-screen overlay) ⚠ at-risk — not compilable here; device confirms
mofe_blackbox_decode.py offline agent tool --selftest passes end-to-end through real H.264

Verification done in-sandbox: the codec survives real H.264 (libx264 crf 23 and 28: 30/30 frames exact,
fail-closed
) rendered exactly as the overlay draws it (0.04·min margin, non-square cells, 3 fiducials) and
decoded via the resolution-exact path; and the on-device Kotlin encoder is byte-identical to the python
decoder (golden test). Unprovable here (device only): that the androidApp overlay compiles and renders,
and the real phone-screen-recorder round-trip.

Capture protocol (the walk)

You already produce the ndjson via the mofe-trace.on sentinel. With this build, that same sentinel also
draws the black-box overlay
— so your capture procedure barely changes:

  1. Mesh up + converged — the 4 anchors as surveyed (A1/A2 near wall, A3 across the room, A4 the red cup),
    mesh at QUORUM.
  2. On the observer Pixel (c158fc), enable trace as you always do — create mofe-trace.on in the app's
    external files dir (…/Android/data/<pkg>/files/mofe-trace.on). This now arms both the ndjson recorder
    and the overlay. Launch the app → the screen fills with a black/white grid + 3 white corner squares
    (that's the overlay; it's not meant to be looked at — the screen is the data channel now).
  3. Start the phone's screen recorder.
  4. Carry the phone on a walk — ~60–90 s, natural pace, holding it in a consistent orientation. Cover
    varied bearings: through the room interior and close to the walls, especially near the A3 corner
    (the biased anchor) and along its wall — that spread is what lets the observer ranges fix A3's
    wall-tangential error.
  5. (Optional but ideal) have a second phone/camera film the walk against the tiled floor — the tile
    grid is a metric reference I can use to extract your absolute position per frame (removes IMU drift as a
    confound). No floor marks needed.
  6. Stop the screen recorder; remove mofe-trace.on.
  7. Send me two raw files (I do all the computation): the screen-recording .mp4, and the
    files/mofe-trace/mofe-trace-c158fc-*.ndjson. Plus the room video if you shot one.

What I do with it

mofe_blackbox_decode.py capture.mp4 --ndjson trace.ndjson --out records.jsonl
decode each video frame (resolution-exact grid registration) → per-frame {tEngineMicros, frameIdx, raw-IMU buffer, pose}merge with the ndjson ranges on the shared engine clock → integrate IMU odometry (or the
tile-floor trajectory from the room video) for independent observer position → feed those as observerConstraints
into the leader refine on the biased 4-anchor graph → report whether A3 pulls back toward its surveyed
truth vs the un-augmented solve
. That's the go/no-go for building #69b-i.

Notes

  • The overlay replaces the visible scene during capture — expected; it's instrumentation, not a view.
  • If the screen recorder crops (rare), decode with --fiducial; a full-screen recording uses the exact
    resolution path automatically.
  • mofe_blackbox_decode.py --selftest reproduces the H.264 verification on any machine with numpy + ffmpeg
    (pip install imageio-ffmpeg bundles one).
  • Everything here is on branch claude/mofe-aiming-calibration-y2ckrl (pushed). Wire format is locked by
    BlackBoxWireFormatTest, so the on-device encoder and this decoder can't drift.
(3-3/3)