NVIDIA A10G (24 GiB)

NVIDIA · Dedicated memory

Installed memory
24 GiB
Usable per device
24 GiB
Bandwidth
600 GB/s
Largest fitting model
34.39B
Power
300 W

Inference configuration

One sequence per batch, a 512-token prompt and 128 generated tokens. Multi-GPU estimates use pipeline parallelism over PCIe 4.

Open in calculator

Decode speed by model

Compare up to four checkpoints at the settings above. Bars show generated tokens per second.

Llama 3.1 8B Instruct78.2 tok/s
DeepHermes-3-Llama-3-3B-Preview192 tok/s
DeepHermes-3-Llama-3-8B-Preview78.2 tok/s

Model performance matrix

Reference rates use the published 32K-cache benchmark when the calculator matches it. Other speeds use its roofline estimator; TTFT covers prompt processing.

#QuantizationRate source
1SmolLM2-135M0.13B4,080.90.006 sFitsQ4Roofline
2SmolLM2-135M-Instruct0.13B4,080.90.006 sFitsQ4Roofline
3SmolLM2-360M0.36B1,592.10.017 sFitsQ4Roofline
4SmolLM2-360M-Instruct0.36B1,592.10.017 sFitsQ4Roofline
5Qwen2-0.5B0.49B1,260.80.022 sFitsQ4Roofline
6Qwen2-0.5B-Instruct0.49B1,260.80.022 sFitsQ4Roofline
7Qwen2.5-0.5B0.49B1,260.80.022 sFitsQ4Roofline
8Qwen2.5-0.5B-Instruct0.49B1,260.80.022 sFitsQ4Roofline
9Qwen2.5-Coder-0.5B0.49B1,260.80.022 sFitsQ4Roofline
10Qwen2.5-Coder-0.5B-Instruct0.49B1,260.80.022 sFitsQ4Roofline
1 / 33

Inference capacity

Llama 3.1 8B Instruct · Fits · 6.52 / 24 GiB

Selection notes

Largest catalog checkpoint inside this memory budget · Yi-34B-200K Use the remaining memory as headroom before increasing concurrency.

Fine-tuning capacity

Llama 3.1 8B Instruct · 4,096 tokens · batch 1 · calculator defaults

QLoRA

8.4 GiB

Fits

LoRA

20.8 GiB

Fits

Full fine-tuning

154.1 GiB

Over memory

Frequently asked questions

How is usable memory determined?

The calculator uses the device profile’s available fraction · 100% × 24 GiB. System use can reduce the amount available at runtime.

Why is a speed missing?

Speed needs a fitting checkpoint and either matching calibration or both bandwidth and a comparable dense BF16 peak.