ML Inference Estimator

What is an ML inference calculator?

This ML inference estimator shows typical inference speed for popular small machine-learning models on edge hardware: Raspberry Pi 4, Raspberry Pi 5, and Raspberry Pi 5 with a Coral Edge TPU accelerator. Inference is the process of using a trained model to make predictions on new data. The figures come from a fixed table of approximate benchmark results, so treat them as a guide for choosing hardware rather than a measurement of your own setup.

Inference speed depends on model size, numeric precision (FP32, FP16, INT8) and the hardware it runs on: the CPU, a GPU or a dedicated accelerator such as the Coral Edge TPU, which only runs 8-bit quantised TensorFlow Lite models. Large language models and speech models are limited mainly by memory and memory bandwidth, which is why they run slowly — or not at all — on small single-board computers.

How to use this tool

Pick a model (image classification, object detection, a small language model or speech recognition) and a device. The tool shows the approximate latency per inference and the resulting throughput: frames per second for vision models, tokens per second for the language model, and speed relative to real time for Whisper (above 1× means audio is transcribed faster than it plays). A bar chart compares all devices for the selected model.

Key concepts

• Parameters — the learned weights in a model. More parameters generally mean better quality but higher resource requirements.
• Precision — FP32 uses 4 bytes per parameter, FP16/BF16 uses 2 bytes, INT8 uses 1 byte. Lower precision reduces memory and increases speed with minimal quality loss.
• Batch size — processing multiple inputs simultaneously improves throughput but requires more memory.
• KV cache — for transformer models, the key-value cache grows with sequence length and consumes significant additional memory.

Common inference hardware

Consumer GPUs: RTX 3090 (24GB), RTX 4090 (24GB) — good for models up to 13B parameters in INT8. Professional: A100 (40/80GB), H100 (80GB) — required for larger models. Cloud options: AWS, GCP, and Azure offer GPU instances by the hour. For CPU inference: possible for smaller models but 10-100x slower than GPU. Apple M-series chips offer unified memory that can run surprisingly large models.

Worked example

A language model with 7 billion parameters stored in 16-bit precision needs 7,000,000,000 × 2 bytes = 14 GB just to hold its weights. Quantising to 4-bit cuts that to about 3.5 GB, letting it fit on a consumer GPU. If the GPU delivers 300 GB/s of memory bandwidth, reading 3.5 GB per token caps throughput near 85 tokens per second.

Common mistakes

A common mistake is judging inference speed by compute (FLOPS) alone; for large models, memory bandwidth usually limits token generation, not raw math. Another is ignoring the difference between latency (time to first token) and throughput (tokens per second across a batch). Forgetting that batching many requests together raises throughput at the cost of per-request latency is also common.

Frequently asked questions

How much GPU memory do I need for a 7B parameter model?

At FP16 precision: 7B * 2 bytes = 14GB for weights alone, plus 2-4GB for KV cache and overhead, totaling about 16-18GB. At INT8: 7B * 1 byte = 7GB for weights, about 10-12GB total. At INT4 (GPTQ/AWQ quantization): about 4-5GB for weights, 6-8GB total. A 24GB consumer GPU (RTX 3090/4090) can comfortably run 7B models in most precisions.

What is quantization and how much does it help?

Quantization reduces the precision of model weights from FP32 (4 bytes) to FP16 (2 bytes), INT8 (1 byte), or INT4 (0.5 bytes). This reduces memory requirements proportionally and often speeds up inference. INT8 quantization typically preserves 99%+ of model quality while halving memory compared to FP16. INT4 saves even more memory with slightly more quality degradation. It is the most practical way to run large models on consumer hardware.

Why does quantisation speed up inference?

Quantisation stores weights in fewer bits — 8-bit or 4-bit instead of 16-bit — so there is less data to move from memory for each token. Since large-model inference is usually memory-bandwidth bound, halving the bytes per weight roughly doubles the token rate, with a small and often acceptable loss in accuracy.