99.6% ML inference cost reduction for global geospatial workloads.

§ 01 — Program context and constraintsAn inherited notebook. A $26,200/month billing surprise.

This engagement was a federally-funded geospatial intelligence program requiring automated detection of heavy construction activity from multispectral satellite imagery. The scope was planetary: geographic coverage 10–20× the size of Moscow, with imagery revisits every two weeks spanning 5–10 year historical windows per area of interest.

The program's primary evaluation metric was cost per unit of processed area — not accuracy alone, not throughput alone. Cost efficiency was a first-class scored deliverable, validated by the government client across 50+ independent release cycles throughout the program lifecycle. Budget overruns were not recoverable.

The baseline inference system was inherited from a university research partner and delivered as Jupyter notebooks. My mandate was to transform that research artifact into a production-grade pipeline. What the notebooks contained was a set of compounding engineering defects that, left unaddressed, would have made the program economically unviable.

§ 02 — Workload sizing111 million chips. Every per-chip inefficiency multiplied.

Understanding the cost problem requires understanding the scale. Every per-chip inefficiency multiplied across 111 million chips.

The model itself was a mid-sized PyTorch CNN operating on multispectral 4-band imagery at 2m resolution — a pure inference workload with no training in the production loop. The problem was not model complexity. It was everything built around the model.

§ 03 — Baseline architecture and cost$32/hr GPU instances. Regardless of AOI size.

The baseline system used high-end GPU compute instances — multi-A100 configurations. Cost figures use current AWS on-demand public rates as a conservative proxy; actual contract rates were negotiated. The structural cost drivers and proportional reduction are accurate regardless of the specific rate applied.

AOI sizes varied enormously across the program — some regions covered tens of thousands of square kilometers, others covered areas as small as a single 100×100m construction site. Both triggered identical instance configurations. Small AOIs generated near-zero GPU utilization before teardown, billing GPU-hours for a handful of forward passes.

§ 04 — Root cause analysis — five defectsEach independently fixable. Together, multiplicative.

Each defect below was independently fixable. Together they were multiplicative: the waste from each compounded the savings from the one before it. Addressing all five was necessary to reach the final reduction.

# Before: single-sample loop
for chip_path in chip_list:
    results.append(model(preprocess(load_chip(chip_path))))

# After: batched queue
def batched_inference(chip_queue, model, batch_size=64):
    for batch in iter_batches(chip_queue, batch_size):
        tensors = torch.stack([preprocess(c) for c in batch])
        with torch.no_grad():
            yield from zip(batch, model(tensors))

# Research workaround
band_4      = load_imagery(scene)           # shape: (4, H, W)
band_8_fake = torch.cat([band_4, band_4], dim=0)
# bands 0-3 == bands 4-7 — perfectly correlated

def build_inference_queue(chip_paths, mask_paths, threshold=0.5):
    return [
        (chip, mask)
        for c, m in zip(chip_paths, mask_paths)
        for chip, mask in [(load_chip(c), load_mask(m))]
        if 1.0 - mask.mean() >= threshold
    ]  # ~35% smaller queue

§ 05 — The production rebuildBuilt from scratch. Stage-separated. CPU-only inference.

The research partner provided model architecture, training data, and Jupyter notebooks. The production system was built from scratch. Ownership breakdown:

• Fully mine: classification pipeline end-to-end — chip extraction, mask filtering, queue management, batched inference, aggregation, all Python code, all Docker images, Airflow DAG
• Architecture lead: change detection stage — overall pipeline architecture and integration contracts
• Directed, not implemented: model retrain — I set acceptance metrics and provided training dataset; university team executed
• Inherited: research model weights and original architecture (university partner)

Pipeline architecture

Airflow DAG

with DAG('inference_pipeline', schedule_interval='@biweekly') as dag:
    preprocess   = DockerOperator(task_id='raster_preprocessing',
                                   image='inference/preprocess:slim')
    filter_queue = PythonOperator(task_id='mask_filter_chip_queue',
                                   python_callable=build_inference_queue)
    inference    = DockerOperator(task_id='batched_inference',
                                   image='inference/model:slim')
    aggregate    = PythonOperator(task_id='result_aggregation',
                                   python_callable=aggregate_regional_outputs)
    preprocess >> filter_queue >> inference >> aggregate

Docker image slimming

FROM pytorch/pytorch:2.x-cuda-runtime AS base
# runtime CUDA libs only — no cudnn-dev, no compiler
COPY requirements-inference.txt .
RUN pip install --no-cache-dir -r requirements-inference.txt
# pinned deps: torch, rasterio, numpy, boto3
COPY src/inference/ /app/
CMD ["python", "-m", "inference.worker"]
# ~400–600MB vs 2–3GB baseline · ~75% reduction across 50+ pull cycles

Before / after

§ 06 — Results$26,200 → $90. No new algorithms. No changed requirements.

The pipeline processed the full program workload within the government client's cost and accuracy constraints across 50+ independently evaluated release cycles. Cost efficiency held through the full evaluation period without architectural rework.

§ 07 — LessonsFive things that apply to the next one.

"The first question when inheriting research code should not be 'does it produce correct outputs' but 'what will this cost at the volume we actually need.'"

Research code is not a cost baseline

Jupyter notebooks are validation artifacts, not infrastructure. The cost structure of research code — single-sample loops, monolithic compute, no stage separation — is appropriate for a lab environment and catastrophic at production scale.

Workarounds compound silently

The band duplication hack was a single line of code that simultaneously degraded model quality, doubled memory bandwidth consumption, and compressed batch size headroom. Each consequence was invisible until you looked. Research workarounds tend to solve the immediate problem (the notebook runs) while embedding structural costs that multiply at scale.

Right-sizing compute is not a tradeoff

Running CPU-bound preprocessing on a multi-A100 instance is not a "good enough for now" decision — it is a billing error. Compute class mismatches between task type and instance type produce no benefit in exchange for the excess cost. The fix is always separation.

Fan-out patterns require size awareness

Uniform infrastructure per logical unit only makes sense when logical units are uniform in size. When AOIs range from 50,000 km² to 100m×100m, the appropriate abstraction is a shared queue where cost is proportional to actual chip volume — not a per-region instance where cost is proportional to region count regardless of size.

Batch size is free throughput at scale

The transition from batch=1 to batch=64 required approximately 20 lines of code and delivered a 35× throughput improvement. Against 111 million chips, that 35× translated directly into compute hours billed.