InferenceLab

InferenceLab is a public engineering platform for measuring what AI inference actually costs on real hardware. It runs vision, language, speech, image and video models across local, browser, edge and remote runtimes, and reports latency, throughput, memory, utilization, energy and output quality.

The distinguishing choice is decomposition. Most tooling reports one end-to-end figure, which is the least useful number available: it cannot tell you whether a slow response came from decoding the input, from the model itself, or from moving data across a network. InferenceLab measures those separately.

• Open source, developed in the open
• Active development — every capability is labelled by maturity
• Designed to be reproducible on a reader's own hardware

Models are usually evaluated in a setting that has little in common with deployment. Batch inference on a data-centre GPU says nothing useful about a single-request latency budget on an integrated GPU, in a browser tab, or on a battery-powered device.

Teams therefore discover late — often after integration — that a chosen model does not fit the target. The cost of that discovery is a redesign, not a tuning pass. And because the measurement is end-to-end, the redesign starts without knowing which stage to change.

• Public benchmarks rarely report the deployment conditions that matter
• Runtime support varies sharply between local, browser, edge and remote targets
• An end-to-end timing hides which stage caused a regression
• Energy and device utilization are seldom reported at all, yet often decide feasibility
• Cross-modality comparison is usually impossible because each modality has its own harness

• Decompose inference into input decoding, device execution and network transport, and measure each stage independently
• Run comparable workloads across vision, language, speech, image and video models
• Execute the same workload on local, browser, edge and remote runtimes without changing application code
• Detect what a host genuinely supports before loading a model
• Report latency, throughput, memory, utilization, energy and quality together, since a gain in one is usually a loss in another
• Make every measurement reproducible on the reader's own hardware

The system is a staged pipeline with two gates: a capability gate before any model is loaded, and an instrumentation boundary between every stage. The first makes the platform trustworthy — nothing downstream runs on an assumption. The second is what makes the measurements attributable to a stage rather than to the whole.

1. 01

   Input

   Camera, file, stream or text, depending on modality

2. 02

   Decode and Preprocess

   Measured separately — often the dominant cost for video and speech

3. 03

   Capability Detection

   Probes CPU, GPU, WebGPU and WebAssembly support before any download

4. 04

   Model Registry and Selection

   Declarative registry of models, modalities and runtime requirements

5. 05

   Execution: Local / Browser / Edge / Remote

   Target chosen from detected capability

6. 06

   Transport

   Measured for remote targets — serialisation, network, deserialisation

7. 07

   Measurement and Reports

   Per-stage latency, throughput, memory, utilization, energy and quality

Each modality has its own decode and preprocess path but shares the same instrumentation contract, so a stage timing means the same thing whether the input was a video frame or an audio buffer. Execution is the only stage that touches the model; decode and transport are deliberately kept outside it, because conflating them is what makes end-to-end numbers unactionable.

• Model export to ONNX with graph-level optimization passes
• Execution-provider selection driven by detected hardware rather than configuration
• Decode and preprocessing tuned per runtime, and measured so the tuning can be justified
• Throttling and back-pressure so the UI thread is never blocked
• Buffer reuse across requests to avoid per-request allocation churn
• Optional quantized variants where a model provides them, compared rather than assumed better
• Instrumentation designed to stay cheap enough not to distort what it measures

• Frontend deploys as a static bundle
• Inference service deploys as a container with optional GPU access
• Deployed at visionedge.c-maron.space — the hostname predates the rename to InferenceLab

• Capability detection runs before model download, so unsupported models are never fetched
• Automatic rollback to the nearest supported model, surfaced in the interface rather than hidden
• Explicit degraded mode when the remote inference service is unreachable
• Camera permission and device-loss states handled as first-class UI states
• Per-stage error boundaries so a failing stage does not stop the others
• Structured logging of stage timings and failures for observability
• A metric the host cannot provide is recorded as unavailable, so a missing energy reading never silently becomes a zero

Benchmarks are only meaningful alongside their conditions, so every published measurement is intended to carry the hardware, model version, runtime, execution provider, input characteristics and warm-up policy that produced it.

Because the platform reports per-stage numbers, the methodology also has to state where each stage boundary sits. A decode cost attributed to execution would make two runtimes look different when only the input pipeline changed.

• Fixed input sets rather than live capture, for repeatability
• Warm-up runs discarded before timing begins
• Latency reported as a distribution, not a single mean
• Throughput measured per stage as well as end to end
• Peak memory recorded per execution target
• Device utilization sampled during execution, not averaged over the whole run
• Energy measured where the host exposes a real interface, and reported as unavailable where it does not
• Output quality evaluated per modality on a fixed, published evaluation set

The measurement harness is being finalised. No latency, throughput, memory, utilization, energy or quality figures are published for this project yet.

Benchmarking in progress. Results will be published with the hardware, model, runtime and measurement conditions that produced them.

• Capability detection is worth building before anything else; every later assumption depends on it.
• A declarative registry removes an entire class of change: adding a model becomes configuration rather than code.
• Fallback behaviour must be visible. A silent downgrade produces benchmark numbers that quietly describe a different model.
• One end-to-end number is the least useful measurement available. Splitting decode, execution and transport was the change that made the output actionable.
• Stages whose latency differs by orders of magnitude should never share one timing boundary — treating them as a single stage was a design error corrected early.
• A metric that is unavailable has to look different from a metric that is zero, or the reports quietly mislead.

• Complete the per-stage decomposition across every modality
• Add the language, speech and video paths to the shared instrumentation contract
• Energy and utilization measurement where the host exposes real interfaces
• A dedicated edge-device runtime, where the power budget matters most
• Publish the reproducible benchmark suite with full measurement conditions
• Structured benchmark reports exportable as JSON
• Contribution guidelines and a documented model-addition process