LenaLab: What an AI Agent Built — and How We Know It's Real
An AI agent wrote the computer-vision algorithms. A separate program — no AI in it, fully deterministic — graded them, on data the agent never saw and couldn't touch. Over three weeks that one loop ran across seven different problems: reading a car's path from dashcam video, closing a kilometre-long SLAM loop, turning six car cameras into a top-down map of the road, and — off the road — mapping a warehouse floor from 19 ceiling cameras.
This is the tour: what it built, how it trained, where it broke, and why the numbers hold up.
The agent's visual-odometry front-end tracking features across a KITTI street — the flow trails it uses to recover the car's motion. Everything below is graded on data like this that the agent never trained on.
The one idea everything rests on
Most "AI writes code" demos stop at "it ran." LenaLab's rule is stricter: "it ran" is never success — only "it generalized" counts. So the system splits into two halves that don't trust each other:
- The solver — the AI agent. It looks at the data, researches an approach, writes the algorithm, and trains it on a GPU when needed. This is where the creativity lives.
- The verifier — the harness. It owns the ground truth, the metric, and the pass/fail bar. It runs the agent's code in a sandbox and grades it on a held-out split the agent was never given. It is deterministic and contains no AI, so it can't be talked into a good score.
The agent is judged by something it cannot influence, on data it cannot see. That's the whole trick — and it's the only reason the numbers below are worth anything.
The seven problems
Each one is a full contract: a data adapter (raw data → tensors + a secret held-out answer key), a from-scratch reference that sets the bar, an un-cheatable grader, and the live run where the agent authors the algorithm.
| # | Problem | What the agent wrote | Held-out result | |
|---|---|---|---|---|
| 1 | RGB-D visual odometry | SIFT→PnP RANSAC + optical flow, metric depth | 0.033 m trajectory error on an unseen scene — beats the classical reference (0.057) | ✅ |
| 2 | Monocular VO | optical-flow + wide-baseline keyframes | 0.052 m (unseen) | ✅ |
| 3 | KITTI stereo VO | stereo depth → ORB → PnP, outdoor driving | 2.08% translation error (unseen sequences) | ✅ |
| 4 | Learned VO (trained on GPU) | a pose CNN fed optical flow, from scratch | beats the learned reference ~1.7× | ✅ |
| 5 | BEV perception | Lift-Splat: 6 cams → top-down vehicle occupancy | 0.136 IoU, 3/3 runs (after a variance fix) | ✅ |
| 6 | 3D occupancy | Lift-Splat-to-3D: 6 cams → a 200×200×12 voxel grid | 0.079 IoU, 3/3 runs | ✅ |
| 7 | Warehouse floor map (off-road) | IPM + temporal background subtraction, 19 static cams | 0.39–0.44 IoU — ~2× the reference I wrote | ✅ |
Plus a stereo DROID-SLAM benchmark (0.03–0.20% error on kilometre-scale KITTI loops) and an autonomous research committee — a multi-agent program that improved its own held-out accuracy 26% over a lineage of experiments, i.e. the lab steering itself, not just one-shot coding.
BEV (problem 5), read left-to-right: the six surround cameras → the reference's guess (BEFORE) → the
agent's (AFTER) → the held-out ground truth. Green = correct, red = missed, blue = false alarm. On scenes
it never saw, the agent roughly doubles the reference's accuracy.
The SLAM benchmark: stereo DROID-SLAM reconstructs a kilometre-scale KITTI driving loop as a dense
point cloud; the camera trajectory (red) threads the road corridor and closes the loop.
How it actually trained — hardware, data, and method
No hand-waving. Every domain, exactly what ran:
| Problem | Hardware | Dataset (size · train/val) | Training | Held-out result |
|---|---|---|---|---|
| RGB-D VO | CPU (sandbox) | TUM RGB-D · 3.1 GB · unseen fr1_desk | none — pure geometry | 0.033 m |
| Monocular VO | CPU | TUM RGB-D · 3.1 GB | none — pure geometry | 0.052 m |
| KITTI stereo VO | CPU | KITTI odometry · 32 GB · seq 00 dev, 05/07 held-out | none — pure geometry | 2.08% |
| Learned VO | GPU (RTX 3080, 16 GB) | KITTI multi-sequence | from scratch, n=3 seeds | beats learned ref |
| BEV | GPU (3080 / 4090) | nuScenes mini · 4 GB → 227 MB cache · 323/81 | from scratch, ~24 epochs | 0.136 IoU |
| 3D occupancy | GPU | nuScenes mini · 226 MB cache · 323/81 | from scratch, ~24 ep, 0.57 GB VRAM | 0.079 IoU |
| Warehouse floor | cloud RTX 4090, 24 GB (rented) | NVIDIA Smart Spaces · 19 cams, 2.9 GB video → 481 MB · 210/90 | from scratch, ~14 min GPU | 0.39–0.44 IoU |
| SLAM benchmark | GPU | KITTI loops · 32 GB | pretrained DROID (no training) | 0.03–0.20% |
Two honest design choices behind those numbers: the classical front-ends train nothing (they're geometry, on CPU), and the learned models train from scratch — the sandbox has no internet, so no pretrained weights. That makes the absolute numbers modest by design; the datasets are deliberately tiny (nuScenes mini is 10 scenes; the warehouse is one space). The claim is "the method generalizes," not "a record score." The whole cloud run for problem 7 cost about $2–3.
The agent's BEV occupancy swept frame-by-frame across a full held-out scene — its prediction tracking
the vehicles it was never shown.
Problem 6 in 3D: six cameras → a held-out voxel grid. Height-coloured, with true positives, misses, and
false alarms marked — real camera-to-3D perception in a tiny-data regime.
The headline finding: an agent's freedom is its variance
The most interesting result isn't a single score — it's a method failure, caught and fixed.
When the agent authors a perception network completely freely, the results are high-variance: it can write an excellent network or quietly sabotage the fragile geometry/augmentation glue. Run three times, BEV and occupancy passed only ~2/3, swinging wildly. A single run would have reported a clean "VERIFIED" — and it would have been partly luck. Running it three times is what caught the truth.
The fix: lock the fragile parts in a scaffold and let the agent author only the network. The variance collapsed ~6–7× to near-reference reliability — 3/3 passing — and the same effect replicated in 3D. The lab working as a method: build → find it's not robust → diagnose → fix → validate → replicate.
Free-form agent runs scatter (high variance, left); locking the geometry in a scaffold collapses the
spread to near-reference reliability with all three runs clearing the bar. The same finding replicated
on the harder 3D occupancy task.
Off the road — and the agent out-researches the baseline
The seventh problem left autonomous driving entirely: a static warehouse with 19 fixed ceiling cameras, predicting a privacy-preserving top-down floor map ("a map, not a camera"), graded on a later, unseen slice of time in the same space — the deployable "trust it on day one" claim.
A real lesson surfaced: the driving Lift-Splat geometry simply failed here (only ~3% of its lifted points landed in the warehouse). The fix was the geometry the setting wants — Inverse Perspective Mapping — which took coverage to 98.8%. Transfer the method, not the geometry. Then the agent, given that contract, noticed the cameras were static and invented techniques the reference lacked (temporal background subtraction, multi-plane IPM, adaptive top-K) — and beat the hand-written baseline ~2×.
"A map, not a camera": the fixed warehouse cameras (left) collapse into a top-down floor-occupancy map
(right; dots = people, forklifts, robots) — what's where, not who. The agent learned and verified this
on unseen-time frames.
The automation that made it runnable
The research only counts if the plumbing is honest, so a fair amount went into the harness:
- Cloud execution. A
localjob mode lets the whole lab run on a rented GPU (RunPod) without nested Docker — validated end-to-end on an RTX 4090. - A pod supervisor that writes a live status file, is GPU-aware, salvages the agent's artifacts on completion, and auto-terminates the pod on done/error/stall/max-wall — closing the cost and silent-hang gaps that plain "it finished" notifications miss.
- Anti-tamper grading — 25 tests, including red-team cases proving that a malicious grader claiming a perfect score is overridden by the harness-owned one. All pass.
Why the numbers are trustworthy
- Held-out, every time. Every headline number is on data the agent never trained on.
- A retraction on the record. An early "VERIFIED 0.1075" BEV claim was a single lucky run; re-run at n=3 it was 0.085 ± 0.034 (2/3). Corrected, not buried.
- Honest negatives kept. A 412 m SLAM divergence, a self-sabotaging BEV run, a from-scratch loop-closure failure, a C++ IMU-fusion dead-end — all on the record.
- Failures surfaced. A 3-hour training hang and a lost agent model (a pod auto-terminated before salvage) were both written up and fixed in the harness, not hidden.
Honest scope (what this is not)
The datasets are small and the models train from scratch, so absolute numbers sit below full-scale literature — the claim is "the agent's research loop works and generalizes," not state of the art. The warehouse result is per-space self-verification (unseen time), not a brand-new warehouse. And the variance study was done for BEV and occupancy, not yet repeated off-road.
Bottom line: an AI agent doing end-to-end computer-vision research — analyze, build, train, and prove it generalizes — across seven problem classes, with a method (and a record) honest enough to catch its own mistakes.
Want the depth? Each problem has its own write-up — the BEV perception, 3D occupancy, and warehouse floor map reports, and the full build-by-build chronicle. Code: github.com/WhaSukGO/LenaLab.