When Simplex Met Images: Reading a License Plate From 31 Shaky Frames

CompositeThu Jun 11 2026 00:00:00 GMT+0000 (Coordinated Universal Time)

image-registrationsuper-resolutionclassical-cvoptimization

You have 31 blurry dashcam frames. Look at any one of them alone and even a human can't read it. But if all 31 captured the same plate from slightly different positions, that "slightly" becomes information.

bicubic vs SR — one frame upscaled (left) vs 31 frames registered and fused (right)

On the left, a single LR frame upscaled 4× with bicubic interpolation. The characters smear together and are hard to read. On the right, the same sequence's 31 frames registered to one another and accumulated onto an HR grid. Same data, same resolution, same camera. The only difference is how the tiny differences across many frames were gathered.

This article is about the Simplex optimizer at the heart of that "how" — why a 60-year-old algorithm is still useful in imaging, and how one simple trick makes it applicable to images at all.

1. Simplex — finding the valley without derivatives

The Simplex (Nelder–Mead) algorithm is a derivative-free optimization method proposed in 1965. The key idea is that it uses no gradient at all.

For an N-dimensional problem it builds a shape (a simplex) from N+1 points, and by comparing function values it reflects · expands · contracts · shrinks that shape. Picture an amoeba rolling downhill toward the valley.

initial simplex  →  reflect one vertex  →  if better, expand
                                           if worse, contract / shrink

Why is it attractive?

You don't need the function's derivative — it works directly on black-box functions
The implementation is tiny (scipy.optimize.minimize(method="Nelder-Mead"), one line)
In low dimensions (usually ≤10) it converges surprisingly fast

Where is it used?

Medical image registration — rigid alignment of CT/MR/X-ray is a Simplex classic
Hyperparameter tuning
Calibrating simulation outputs where no derivative is defined
Statistical model parameter estimation

The starting point for this article was a few years ago. At a former job I was working on license-plate OCR, got drawn into image registration, and that's where I first met Simplex. One piece of intuition I picked up then has stuck with me ever since — "if the cost function isn't smooth, Simplex can't solve it, so you have to blur the image to make it smooth."

That single line is the whole article.

2. Simplex's constraint — "the function must be smooth"

Because it compares points without derivatives, its limits are clear-cut.

Constraint	In plain terms
Low dimensions	As dimensions grow, the number of simplex vertices grows as `N+1`, and every step must compare the value at all of them. The practical ceiling is around 10 dimensions
Local optima	Because it rolls into a valley, a bad start traps it in a false one
No convergence guarantee	It's a heuristic, not something a theorem guarantees
Fragile on discontinuities / plateaus	Where the function is flat, Simplex can't tell "which side is lower"

The last row is decisive. The cost function we manipulate when registering images — the difference between two images (SSD) — is, unfortunately, almost flat.

3. Why image registration is hard for Simplex

Let's define the problem of aligning two images.

reference frame R, moved frame M
parameters θ = (tx, ty, angle, scale)
cost: C(θ) = mean( (R - warp(M, θ))² )

If you nudge θ by 0.01 px at a time and plot C, what shape comes out?

An image is a pixel grid. While shifting tx by less than 0.5 px, bilinear interpolation blends the colors a little, but the cost ends up being a piecewise-bilinear sawtooth plus rough plateaus created by texture. Add a repeating pattern like a license plate and you get several false valleys of equal depth.

On a cost surface like that, Simplex can't reach the true valley. It stops at a nearby plateau, "deciding" that this must be the bottom.

4. Scale-space — blur the image to smooth the cost

The fix is surprisingly simple. Blur the image first.

Suppose we blur R and M each with a Gaussian of width σ before measuring the cost.

large σ → fine texture disappears, only big structure remains → a smooth bowl-shaped cost surface
small σ → fine detail survives → a rough cost surface, but the true optimum is more precise

This is where the idea of a scale-space schedule comes in.

σ = 4.0  →  roll toward the true valley in a big smooth bowl (coarse)
σ = 2.0  →  move to a smaller bowl, more precise
σ = 1.0  →  refine the detail
σ = 0.0  →  finish at native resolution with sub-pixel accuracy (fine)

At each stage you run Simplex to convergence and hand its result to the next stage as the starting point. This is coarse-to-fine registration.

for sigma in (4.0, 2.0, 1.0, 0.0):
    R_blur = gaussian(R, sigma)
    def cost(theta):
        M_warped = warp(M, theta)
        return ssd(R_blur, gaussian(M_warped, sigma))
    theta = nelder_mead(cost, x0=theta)   # carry each stage's result into the next as x0

It turns a C⁰ cost into a C∞ one — building an environment where Simplex can actually work. This is exactly why a 60-year-old algorithm is still alive in imaging.

5. What to use it for — multi-frame super-resolution

Once sub-pixel registration is possible, an interesting application opens up.

Say you have N LR frames of the same scene shot from slightly different positions — the camera drifted a little from hand shake or vehicle vibration. Each frame's sensor saw "sub-pixel information from a different position."

If you can align each frame onto an HR grid with sub-pixel accuracy, then N different measurements land inside a single pixel. Average them, and you recover information no single frame holds.

The full pipeline has three stages.

(1) Sub-pixel registration

Gaussian schedule from above. One caveat: for strongly self-similar content like license plates, a cold-start Simplex easily falls into false valleys, so we first establish an integer-pixel prior with phase correlation and start from there.

(2) Shift-and-add (Gaussian splat)

Scatter the registered LR samples onto the HR grid. Spread a Gaussian weight around the sub-pixel location each LR pixel occupies. A naive nearest-neighbor splat produces stippling, so the splat σ is scaled up to match the upscale factor.

(3) Iterative Back-Projection (IBP)

Take the initial HR estimate and simulate "what would it look like dropped back down to LR?" Back-project the difference (residual) between that simulation and the real LR frame onto the HR estimate to correct it. A few iterations and it sharpens up.

6. How well does it actually work?

I validated it on Korean license-plate dashcam sequences (the RLPR dataset, 31 frames per sample).

A single sequence

Easy sequence — left: bicubic ↑4×, right: multi-frame SR + IBP

Same camera, same resolution, same 4×. The only difference is whether 31 frames were gathered.

On easy sequences the digits "813 3489" come back clearly (the single Korean character in the middle is mosaicked in the RLPR dataset itself for privacy).

Let's also look at a hard sequence (more shake, more noise).

Hard sequence

Bicubic carries forward the limits of a single LR frame, but multi-frame SR gathers sub-pixel information from many frames to reconstruct the character outlines.

Quantitative evaluation over 50 samples

I applied the same pipeline to 50 samples in a batch and measured PSNR against a pseudo-GT.

50-sample grid — LR↑NN / Bicubic / SR / GT

	mean Δ	median Δ	win-rate	best	worst
SSD, no-norm	+0.19	+0.21	60%	+1.67	−1.80
SSD, norm	+0.03	+0.10	64%	+0.92	−1.10
NCC, no-norm	+0.27	+0.21	64%	+2.50	−2.11
NCC, norm	−0.01	+0.09	60%	+0.93	−2.36

This table is interesting for two reasons.

First, changing the cost function itself is more effective than normalizing the input. Dashcam frames have heavy lighting changes, and a plain SSD mistakes a brightened frame for "misaligned" and throws it out in outlier rejection. Matching each frame's mean/std to the reference (norm) helps, but it only matches first-order statistics.

Second, stacking normalization on top of NCC actually makes it worse. NCC is by definition brightness/contrast invariant, so pre-normalizing the input removes the same information twice. The second normalization only adds noise.

7. How big is the implementation?

The core is under 200 lines.

Simplex registration (registration.py) — walk the scale-space schedule, Nelder-Mead at each stage.

def register_similarity_simplex(ref, mov,
                                blur_sigmas=(4.0, 2.0, 1.0, 0.0),
                                x0=(0.0, 0.0, 0.0, 1.0),
                                cost="ssd"):
    params = np.array(x0, dtype=np.float64)
    cost_fn = _cost_fn(cost)         # "ssd" or "ncc" (=1-ZNCC)
    for sigma in blur_sigmas:
        R = gaussian(ref, sigma)
        def cost_at(p):
            warped = warp_similarity(mov, *p)
            return cost_fn(R, gaussian(warped, sigma))
        res = minimize(cost_at, params, method="Nelder-Mead", ...)
        params = res.x
    return params

It's all in one line of scipy; all we did was layer a Gaussian schedule and an NCC option on top.

Shift-and-add (reconstruction.py) — splat each LR sample onto the HR grid's (2r+1)² neighborhood with a Gaussian.

for lr, (tx, ty, ang) in zip(lr_frames, transforms_hr):
    x_ref, y_ref = forward_warp(lr_grid, tx, ty, ang)   # sub-pixel HR coords
    for dy, dx in neighborhood(splat_radius):
        w = exp(-((xi - x_ref)**2 + (yi - y_ref)**2) / (2 * sigma**2))
        accum[yi, xi] += lr_value * w
        weight[yi, xi] += w
hr = accum / weight

IBP refinement — forward-simulate (warp → blur → downsample), then back-project the residual.

All you need is OpenCV's warpAffine, GaussianBlur, resize, and invertAffineTransform. No CUDA, no deep-learning framework.

8. Where else does this trick work?

What this article really wants to convey isn't super-resolution itself, but the generality of the idea: smooth a discontinuous cost surface with a Gaussian blur so a derivative-free optimizer can get through it.

Areas where the same pattern works:

Medical image registration — rigid/affine CT/MR alignment. Already a classic
Satellite / aerial image stitching — sub-pixel registration of multiple passes for better mosaic precision
Multi-frame averaging in microscopy — drift correction and photon-limited noise reduction
Lucky imaging in astronomy — aligning many short exposures distorted by turbulence
Video stabilization / rolling-shutter correction — fine inter-frame registration via Simplex
OCR preprocessing — registering multiple captures of the same document via SR for legibility (this article's case)
General calibration with a non-differentiable cost — fitting parameters of simulation outputs

The point is that when a cost is "non-smooth + low-dim + black-box," before reaching for a gradient-based method, ask once: "can I make the cost smooth?" For images the answer was Gaussian blur. Other fields have other answers — kernel smoothing, a surrogate model, the cooling schedule of simulated annealing.

An algorithm's own limits are often solved outside the algorithm — in its input. That's why a 60-year-old Simplex can read a dashcam license plate.

9. Closing

The lessons I drew directly from this work, in two lines:

"This algorithm doesn't work" is not the answer. Often you can reshape the problem to satisfy the algorithm's assumptions — here, smoothing the cost surface with a Gaussian blur was one such reshaping.
Simple comparison experiments break your intuition often. Without the 4-way SSD-vs-NCC, norm-vs-no-norm comparison over 50 samples, the conclusion "a second normalization hurts" would have stayed a guess forever.

The full code lives in whasuk/3_image_registration. It didn't work on the first try — a sign-convention bug, false minima from self-similarity, splat stippling, and outlier-rejection threshold tuning were all found and fixed one step at a time. Follow the git log and you can read that order exactly.

Thanks for reading.