Six Car Cameras → a Top-Down Map: BEV Perception, Verified

LenaLabFri Jun 19 2026 00:00:00 GMT+0000 (Coordinated Universal Time)✓ 1 verified✗ 1 rejectedGitHub →

bev-perceptionmulti-cameranuscenesoccupancyautonomous-driving

Six cameras pointed outward from a car, fused into one top-down map of where the vehicles are — bird's-eye-view occupancy — that has to hold up on scenes it never saw. An AI agent wrote the network; an AI-free grader scored it on held-out data. The honest twist: run it three times and it is not reliable — until a scaffold fixes that.

(New here? Start with the overview for what this lab is.)

Project report · 2026-06-15 · LenaLab — an AI agent doing computer-vision research

1. Why this exists

In LenaLab, an AI agent does the research: it analyzes a vision problem, researches an approach, implements and trains an algorithm, and confirms it generalizes on held-out data the agent never saw. So far the agent had proven that arc on four ego-motion domains (monocular VO, RGB-D VO, SLAM, KITTI stereo): all variations on where did the camera go? Held-out measurement keeps the scoreboard honest; the agent supplies the research and the engineering.

The open question was whether that research arc travels: can the same agent take on a genuinely different kind of computer vision and still build something that generalizes?

This report answers that by having the agent take on a new problem class — multi-camera Bird's-Eye-View (BEV) perception — end-to-end. The agent reinvents the Lift-Splat lifting geometry, designs correct surround-camera flip augmentation, and calibrates the occupancy threshold; held-out IoU grading sits quietly underneath, confirming the result is real.

2. The task

Given a nuScenes sample's 6 surround cameras + their intrinsics + camera→ego extrinsics, predict a top-down vehicle-occupancy grid in the ego frame (100 m × 100 m at 0.5 m/cell → 200 × 200). This is the canonical Lift-Splat vehicle-segmentation task. It exercises geometry the VO domains never touched:

cross-view fusion — six cameras pointing in different directions, fused into one frame,
per-pixel depth lifting — each image pixel becomes a ray of 3-D hypotheses,
a metric ego-frame raster — output lives in world metres, not pixel space,

and it's scored by Intersection-over-Union, an area metric with nothing in common with the trajectory-error metrics (ATE / t_err) the lab used before. None of the VO scoring carried over; the whole evaluation contract was rebuilt for area-based perception — a clean test of whether the research arc transfers.

3. Architecture — where the agent works, and how the result is checked

A LenaLab domain is the tuple {dataset adapter, ground truth + metric, held-out split, reference bar}, and the agent authors the algorithm against it. All five pieces were built:

 nuScenes mini ──prep──▶  ~/.cache/vo_lab/bev/{train,val}/*.npz       (one-time, scripts/prep_nuscenes_bev.py)
                              │
                              ▼
   Provider (NuScenesBEVProvider) lays out, per run:
     LAB_DATA/train/<tok>.npz       6 cams + calib + bev GT   ← agent may train on these
     LAB_DATA/test_input/<tok>.npz  6 cams + calib (NO bev)   ← agent predicts on these
     [held-out] <tok>_bev.npy       held-out BEV GT          ← reserved for grading
                              │
              ┌───────────────┴───────────────┐
              ▼                                ▼
   AGENT authors main.py             Grader: eval_bev.py
   (trains on GPU, writes            (loads held-out GT, computes IoU
    pred_<tok>.npy per sample)        on data the agent never trained on)
              └───────────────┬───────────────┘
                              ▼
                    Oracle:  miou ≥ bar  →  VERIFIED / REJECTED

What the agent authors: main.py — the network, the training, the occupancy threshold; this is where the research happens. The held-out scenes, the ground-truth rasterization, the IoU metric, and the pass/fail bar are fixed independently of the agent, so the score reflects generalization rather than memorization. The grader is restored from the task spec immediately before judging, which is what lets a passing IoU be taken at face value.

This mirrors the proven learned-VO track (GPU training + held-out generalization), so the BEV domain reuses the same build_vo_implementer_harness, sandbox image (vo-gpu-torch:1), and resilient_sdk_author — only the task spec, provider, grader, and reference are new.

4. The ground truth (and why it's trustworthy)

scripts/prep_nuscenes_bev.py rasterizes the BEV GT: for each sample it takes every vehicle.* 3-D box, transforms its footprint from global into the sample's ego frame (translate then rotate by the inverse ego pose), and fills the polygon into the 200 × 200 grid. Held-out = the official nuScenes mini_val scenes (scene-0103, scene-0916), disjoint from mini_train. Result: 323 train / 81 val samples.

The GT was validated two independent ways before any metric was trusted:

By eye (artifacts/bev/bev_gt_check.png): on a parking-lot scene the footprints cluster

Ground-truth check by eye: vehicle footprints cluster in coherent parking-row structure around the ego marker — geometry, not noise.

in coherent parking-row structure around the ego marker — geometry, not noise. 2. By learnability: a model fits it (IoU climbs from 0 to ~0.17). A geometrically broken GT cannot be learned — so the fact that training works is itself a correctness proof.

5. The reference baseline & the numbers

vo_lab/plugins/vo_ref/run_bev_learned.py is a self-contained Lift-Splat-Shoot model: ResNet-18 → per-pixel (softmax depth distribution × context) → lift to a camera frustum of 3-D points using the real intrinsics + cam→ego extrinsics → voxel-pool (scatter-add) into the ego grid → conv head → occupancy logits.

setting	held-out vehicle-IoU	notes
reference, ImageNet-pretrained backbone, n=3 seeds	0.169 ± 0.002	tight variance → a stable bar
reference, from-scratch backbone (sandbox: no network)	0.1042	the honest sandbox bar
all-zero degenerate control	0.0000	the bar means something — empty output scores zero
busy held-out samples (pretrained)	~0.22	`artifacts/bev/bev_pred_heldout.png`

Reference Lift-Splat on busy held-out samples (~0.22 IoU). Green = true positive, red = false negative, blue = false positive — on scenes it never saw.

The two reference numbers differ because the sandbox has no network access, so neither the reference nor the agent can download pretrained weights — both train from scratch. That lowers IoU (0.169 → 0.104) but it's the honest sandboxed setting, and it's the number that sets the agent's bar.

Oracle bar = 0.08 (from-scratch reference ÷ 1.3, the same margin policy as the learned-VO track). Calibration gate: OPEN (reference 0.104 ≫ bar 0.08 ≫ degenerate 0.000).

6. Validation done before spending a token

End-to-end calibration (python -m vo_lab.run_bev_calibration, non-billed): trains the reference in the real Docker sandbox, grades it with the IoU grader, runs the degenerate control — gate OPEN. This exercised provider → sandbox train+infer → grader → oracle on real hardware with zero API cost.
Grader sanity (tests/test_bev_implementer.py, 2 tests, passing): an all-zero author is REJECTED, and a fake eval.py reporting miou=1.0 is overridden by the restored grader and still REJECTED — so a VERIFIED verdict reflects the actual prediction. Same trust layer the VO domain has.
Variance: the pretrained reference's 0.169 ± 0.002 over 3 seeds confirms the bar is a stable property, not a lucky run.

7. The live agent runs (n=3) — capability, but not robust

Given only the data contract and the grid spec, the agent (claude-sonnet-4-6) researched the problem and authored a Lift-Splat network from scratch — three independent times. The honest picture:

run	held-out mean IoU	verdict
run 1	0.1075	✅ VERIFIED
run 2	0.0376	❌ REJECTED
run 3	0.1107	✅ VERIFIED
n=3	0.085 ± 0.034 (range 0.038–0.111)	2 / 3 cleared the 0.08 bar

A single run would have reported a clean "VERIFIED at 0.1075" — and it would have been partly luck. Running it three times reveals the truth: the agent can author a passing BEV network, but not reliably. One run in three (run 2) landed at less than half the IoU and failed. Repeating the run is part of the lab's rigor, and reporting the spread it surfaces — rather than the best of three — is the lab's norm.

Is the variance the task or the agent? (the diagnostic)

Decisive cheap experiment: train a fixed-architecture from-scratch reference at three seeds and measure its spread. If the task were inherently noisy at this data scale, the reference would swing too. It doesn't:

	n=3 held-out IoU	mean	std
fixed-recipe reference (seeds 0/1/2)	0.138 / 0.142 / 0.143	0.141	0.002
agent (re-authors each run)	0.108 / 0.038 / 0.111	0.085	0.034

The fixed recipe is rock-stable (std 0.002) and sits above the agent's best run. So the task is stably learnable and the regime is not the problem — the agent's variance (≈17× the reference's) comes from it redesigning the algorithm every run. Figure: artifacts/bev/bev_variance_n3.png. (The diagnostic uses best-epoch val-IoU, so its absolute

Variance diagnostic: the fixed-recipe reference is rock-stable across seeds (std 0.002), while the agent — re-authoring the algorithm every run — swings ~17× more. The variance is the agent's design latitude, not the task.

0.141 is a touch optimistic vs the final-epoch sandbox reference 0.104 that set the bar; the point is the variance, which is unambiguous.)

The failure is concrete, not mysterious. Comparing the authored code:

run 2 (failed) carved out 15 % of the already-tiny 323-sample training set for threshold calibration (≈275 left to train on) and used a simpler flip augmentation (flip_K adjusts only cx) — if its flipped BEV target wasn't ego-y-flipped to match the flipped cameras, the augmentation injects label noise. Either choice plausibly costs the thin margin.
runs 1 & 3 (passed) calibrated on the full training set and (run 3) implemented flip augmentation correctly (explicit ego-frame + extrinsic + intrinsic update).

So the agent's freedom to redesign — the very thing that makes Track B "authoring, not tuning" — is also the variance source: good designs clear the bar, self-sabotaging ones (over-aggressive holdout, subtly-wrong augmentation) don't. The best run's predictions (artifacts/bev/bev_agent_heldout.png, before/after bev_before_after.png, sweep

The agent's best-run BEV predictions on held-out scenes — strong overlap on some, misses on hard ones. Real small-data multi-view perception, not a polished SOTA number.

Before / after on a held-out scene from the best free-form run.

bev_sweep_scene0103.gif) are real multi-view perception; the reliability is the open question.

Occupancy sweep across held-out scene-0103.

What this does and doesn't show. It does show the agent can take its research arc into a brand-new problem class and author real BEV perception (2/3), with the held-out grading surfacing the non-robustness a single run would have hidden. It does not show the agent reliably matches a fixed reference — it doesn't, yet, at this data scale. The legitimate paths to a robust agent result are (a) more data (a larger nuScenes subset stabilizes both reference and agent) or (b) a scaffold that fixes the architecture and has the agent author only the uncertain piece — the same "isolate the skill" move the SLAM track used. Re-rolling live runs until one passes would be p-hacking and is explicitly not done here. Archived algorithms: artifacts/agent_authored_bev_v1.py (run 1).

7b. Closing the loop: the scaffold fix (hypothesis → validated)

The diagnosis made a falsifiable prediction: if the variance is the agent's design latitude over the fragile parts (geometry, augmentation), then locking those and letting the agent author only the network should collapse the variance. We built exactly that and tested it — not a re-roll, a controlled change.

The scaffold (vo_lab/plugins/vo_ref/bev_scaffold.py, seeded into the workspace as a locked bev_core.py the agent cannot author) owns the Lift-Splat geometry, the correct flip augmentation (camera-swap + extrinsic + ego-y update), the training loop, and threshold calibration. The agent authors only model.py — build_encoder() + build_bev_head() behind a fixed interface. Same held-out grader, calibrated bar 0.082.

condition	n=3 held-out IoU	mean ± std	pass
fixed-recipe reference	0.138 / 0.142 / 0.143	0.141 ± 0.002	3/3
agent free-form (authors everything)	0.108 / 0.038 / 0.111	0.085 ± 0.034	2/3
agent scaffold (authors only the net)	0.138 / 0.140 / 0.130	0.136 ± 0.005	3/3

Confirmed. Locking the fragile parts collapsed the agent's variance 7.3× (0.034 → 0.005, approaching the reference's 0.002) and lifted the mean to near-reference quality (0.085 → 0.136), with 3/3 runs clearing the bar. All three runs left the locked bev_core.py byte-for-byte unmodified (verified by diff). Figure: artifacts/bev/bev_scaffold_compare.png.

The scaffold fix: lock the geometry + augmentation and let the agent author only the network → the variance collapses 7.3× (0.034 → 0.005) and the mean lifts to near-reference (0.136), with 3/3 runs clearing the bar.

This is the lab's full scientific cycle on one problem: build → find it's non-robust → diagnose (agent latitude, not the task) → prescribe (scaffold) → validate the prescription. The takeaway for agent-authored CV: an agent's freedom is both its power (it can invent real architectures) and its risk (it can self-sabotage the fragile glue). Held-out measurement is what lets you tell the difference — and the scaffold is the principled way to keep the freedom where it helps (network design) and remove it where it hurts (geometry/augmentation correctness).

8. Honest scope (stated, not hidden)

Small-data regime. nuScenes mini = 10 scenes total. The from-scratch 0.10 / pretrained 0.17 IoU are well below full-nuScenes LSS (~0.32 over 28 k samples). The gap is data quantity, not a harness flaw — the claim here is the harness generalizes, demonstrated end-to-end, not a SOTA BEV number.
Vehicle class only — the standard LSS sub-task; other classes are future work.
From-scratch backbones in the sandbox (no pretrained download) — a real constraint that lowers absolute IoU but keeps the comparison honest and self-contained.

9. Files & reproduction

scripts/prep_nuscenes_bev.py            harness-owned nuScenes→BEV adapter (held-out split + GT)
scripts/bev_lss.py                      pretrained reference + variance trainer (bar 0.169±0.002)
scripts/grade_bev.py                    standalone IoU grader
scripts/viz_bev_pred.py                 surround→BEV pred-vs-GT (model-based)
scripts/viz_bev_from_preds.py           same, from saved agent masks (architecture-agnostic)
vo_lab/plugins/bev_nuscenes.py          Track-B provider (train / test_input / held-out GT)
vo_lab/plugins/vo_ref/eval_bev.py       IoU grader (held-out, restored before judging)
vo_lab/plugins/vo_ref/run_bev_learned.py from-scratch Lift-Splat reference (sandbox bar)
vo_lab/agents/bev_implementer.py        task spec + reference/degenerate authors
vo_lab/run_bev_calibration.py           non-billed gate (sets the bar)
vo_lab/run_bev_implement.py             live billed Track B (free-form)
vo_lab/plugins/vo_ref/bev_scaffold.py   LOCKED scaffold core (geometry+aug+training; seeded as bev_core.py)
vo_lab/plugins/vo_ref/bev_scaffold_model_ref.py  reference model.py for the scaffold
vo_lab/run_bev_scaffold_calibration.py  non-billed scaffold gate
vo_lab/run_bev_scaffold_implement.py    live billed scaffold Track B (agent authors only model.py)
scripts/bev_scaffold_compare_fig.py     the 3-condition variance figure
tests/test_bev_implementer.py           2 grader-sanity tests (passing)
artifacts/bev/                          GT check, predictions, variance + scaffold figures

# one-time: prep data (in vo-bev:1)
python scripts/prep_nuscenes_bev.py <nuscenes_root> ~/.cache/vo_lab/bev
# non-billed: confirm the gate + derive the bar
python -m vo_lab.run_bev_calibration
# live (billed + Docker + GPU): free-form — the agent authors the whole BEV network
ANTHROPIC_API_KEY=... python -m vo_lab.run_bev_implement 0.08
# scaffold — geometry+aug locked, agent authors only model.py
python -m vo_lab.run_bev_scaffold_calibration                       # non-billed gate (bar 0.082)
ANTHROPIC_API_KEY=... python -m vo_lab.run_bev_scaffold_implement 0.082

Verdict: the agent's research arc transfers cleanly from ego-motion to multi-view perception. It analyzed an unfamiliar problem, reinvented the Lift-Splat lifting geometry, designed surround-flip augmentation and threshold calibration, and trained it — with held-out IoU grading quietly confirming the result generalizes. And on BEV the lab ran its full scientific cycle: free-form, the agent can author real BEV perception but not reliably (n=3 IoU 0.085 ± 0.034, 2/3 — held-out measurement surfaced the non-robustness a single run would have hidden); the diagnostic pinned the variance on the agent's design latitude, not the task (fixed recipe 0.141 ± 0.002); and the scaffold validated the fix — locking the fragile parts collapsed the variance 7.3× to 0.136 ± 0.005, 3/3. build → find it's non-robust → diagnose → prescribe → validate. LenaLab spans five domains (monocular VO, RGB-D VO, SLAM, KITTI stereo, BEV); the agent's research arc holds across all five, and on BEV it did its hardest job — not just producing a result, but keeping an honest one honest and then improving it. Passing once is not the same as being robust — but the agent can engineer its way from one to the other when held-out measurement tells it how.