Off the Road: an AI Agent Maps a Warehouse Floor From 19 Cameras

Nineteen cameras bolted to a warehouse ceiling. The task: fuse them into a single top-down map of where every person, forklift, and robot is standing — a map, not a recognizable image of anyone. An AI agent wrote the algorithm; a separate program with no AI in it scored the result on a stretch of time the agent never saw. It beat the baseline I'd written by about 2×.

This is the seventh distinct problem LenaLab's agent has taken on — and the first that isn't autonomous driving. (New here? Start with the overview for what this lab is and how the grading works.)

"A map, not a camera": the fixed overhead cameras (left) collapse into a top-down floor-occupancy map (right; dots = people, forklifts, robots) — what's where, not who. The agent learned this and was graded on unseen-time frames in the same space.

Why a warehouse, and why "trust on day one"

A vertical-expert exercise asked a simple question: where in real life would a cheap top-down "what's where" map actually matter? The answer kept landing on static multi-camera spaces — warehouses, retail floors, eldercare — with two recurring insights:

• "A map, not a camera." An occupancy map is privacy-preserving by construction: it says a forklift is here, never who is here.
• "Trust on day one." Every space is physically unique, so the unlock is a model that self-verifies on a brand-new space rather than one tuned on a leaderboard.

This domain makes the second one literal. The held-out split is the "trust on day one" claim: train on the first 70% of a space's timeline, grade on the last 30% (unseen time, same space). That is exactly the deployable promise — stand up and verify a model for THIS space from a short window. (Cross-space generalization — a brand-new warehouse — is the harder stretch, deferred and honestly noted.)

How it ran (the concrete bits)

Data	NVIDIA Physical AI Smart Spaces (AI City Warehouse_000): 19 fixed 1080p overhead cameras, a shared world frame, 3D box annotations. 2.9 GB video → 481 MB cache.
Target	A 206×203 top-down floor grid @ 0.5 m world cells, scored by floor IoU. Occupied cells are sparse (~0.5%), so absolute IoU is lower than a dense task — expected.
Split	210 train / 90 val, by time (first 70% / last 30% of the same space). The grader owns the held-out answer key; the agent never sees it.
Hardware	A rented cloud RTX 4090 (24 GB) — the lab's first run off the local machine.
Training	From scratch (no pretrained weights in the sandbox), ~120 epochs, ~14 minutes of GPU. Whole run cost about $2–3.

A real geometry finding: the driving model is wrong here

The first instinct was to reuse the Lift-Splat geometry that worked for the driving BEV and 3D-occupancy domains (learn a per-pixel depth, lift each pixel to a ray of 3D points, pool them into the grid). On these static overhead cameras it failed: the learned-depth frustum scattered thinly across a 100×100 m warehouse — only ~3% of lifted points landed in-grid — and the model learned the occupancy rate but not where (validation IoU ≈ 0).

The fix is the geometry the setting actually wants: Inverse Perspective Mapping (IPM). Because the cameras are static and we're predicting a flat ground plane, you can project each floor cell (at a few height planes) straight through the dataset's verified camera matrix and sample the image features right there. Coverage went 3% → 98.8%. The honest lesson of the domain: transfer the method, not the geometry.

That hand-written IPM reference, trained at three seeds, lands at a stable ~0.22 floor IoU — a real learned baseline, and the bar the agent has to beat.

The agent out-researches the baseline

Given the same contract, a sandboxed agent (Claude, on the cloud GPU) authored a floor-occupancy network from scratch, trained it, and was graded on the unseen-time frames:

run	held-out floor IoU	verdict
1	0.4378	✅ VERIFIED
2	0.3942	✅ VERIFIED
reference (IPM, 3 seeds)	~0.22	—

Seeing it work, on held-out time. Four panels: camera 0, the ground-truth floor occupancy, the agent's prediction, and a true-positive (correct) vs false-positive (false alarm) overlay. The map tracks people, forklifts, and robots across the warehouse floor — never their faces.

Both runs clear the bar, and both land ~1.8–2× the reference I wrote — on data the agent never saw. The reason it won isn't tuning; it's that the agent noticed the cameras are static and built a method around that fact (recovered from its own main.py):

• Temporal background subtraction — a median background image, differenced per frame, makes the moving agents pop. The input becomes 6-channel (3 RGB + 3 background-difference).
• Multi-plane IPM at four heights, fused mean + max across the 19 cameras.
• Focal loss (α=0.97, γ=2) matched to the ~0.4% positive rate.
• Adaptive top-K inference — predict the per-frame expected number of occupied cells and take that many, so it's robust to probability-scale drift on unseen frames instead of a fixed threshold.

That is the agent reasoning like a researcher: it identified a property of the setting (static cameras) and engineered for it — which is precisely why it beat a generic geometric baseline.

The harder question: one model for every warehouse?

The per-space result is the deployable claim, but it begs a bolder one: could one model serve every warehouse, with no per-space training? We tried it honestly — and it doesn't, yet.

Train on 4, then 12 warehouses and test an entirely unseen one, and held-out IoU collapses to ~0.04–0.05 (versus ~0.2 on the warehouses it was trained on). Neither more training scenes nor domain randomization closes the gap: each space is its own camera-rig distribution. A four-expert panel plus a three-step experiment ladder pinned the bottleneck as distribution shift, not architecture — and showed that per-space adaptation recovers almost all of it:

adaptation to the new space	held-out IoU
frozen cross-space model (no adaptation)	~0.05
few-shot fine-tune on the new space	0.36
trained from scratch on the new space	0.42

So the honest conclusion is the one the harness was built to verify: per-space is the deployable unit — exactly the "trust on day one" claim.

Pushing per-space to production

With per-space as the target, the result climbed past the agent's free-form baseline:

model	held-out floor IoU
IPM reference (hand-written)	~0.22	baseline
agent free-form	~0.39	~2× reference
pretrained ResNet18, stride-8	0.340	a pretrained backbone alone underperforms — resolution matters
+ finer stride-4 features	0.391	matched the agent baseline — finer features were the gap
+ 3× denser frames (630)	0.442	+13% over the agent baseline — the production model

Recipe: an ImageNet-pretrained ResNet18 (its first conv inflated to 6 channels = RGB + background-diff) at stride-4, four-plane IPM fused mean+max, focal loss, discriminative learning rates, 120 epochs. The checkpoint and raw predictions are kept, so the demo below stays re-renderable.

The production model (held-out IoU 0.442) on 48 unseen-time frames. Four panels: camera 0, the ground-truth floor occupancy, the model's probability map, and a true-positive / false-negative / false-positive overlay — with per-frame IoU ticking along (mean 0.448).

Honest scope

• Per-space, by design. Box-derived agent occupancy (people/forklifts/robots); the absolute IoU is for this space, on unseen time. Cross-space generalization (a brand-new warehouse) was attempted, not deferred — and it's hard (~0.05 unseen): each space is its own distribution, so per-space adaptation is the honest deployable unit, as shown above.
• No n=3 / no scaffold here yet. The variance study from the BEV and occupancy domains isn't repeated — this is two free-form runs, both verified. The claim is that the loop transfers to a new problem class and the agent beats the baseline on fresh data, not a variance result or a SOTA number.
• The first run's main.py was lost (the pod auto-terminated before salvage); the supervisor was then fixed to salvage artifacts before terminating, and run 2's model is captured. Recorded, not hidden.

The takeaway: the lab's analyze → research → implement → train → verify loop carries off the road. On static multi-camera warehouse perception — a fresh, non-driving problem — a from-scratch agent beat the geometric baseline by ~2× on unseen time, by inventing static-camera-specific techniques. Seven problem classes now, and the first that isn't a car.

---

The full picture across all seven problems is in the overview. Code: github.com/WhaSukGO/LenaLab.