aceDoes Lower Precision Always Save Energy? Benchmarking Energy–Performance Trade-Offs in Weight-Quantized Large Language Model Inference on a GPU
Large language model (LLM) inference is an increasing component of AI energy demand, while the energy effects of weight quantization depend on workload phase and implementation. We benchmarked four LLM families—Gemma 4 E4B IT, Qwen3.5 4B, Nemotron 3 Nano 4B, and Ministral 3 3B Instruct—across Q4_K_M, Q6_K, Q8_0, and BF16 weight representations on a single NVIDIA GeForce RTX 4090 using llama.cpp/CUDA. Five workloads separated prefill from autoregressive generation, and GPU energy was estimated from NVML telemetry…
On one RTX 4090 using llama.cpp/CUDA, 4-bit Q4_K_M weights cut dynamic energy per token during generation by 63.6–85.7% versus BF16 and raised throughput across all four tested model families.
Findings are limited to single-request execution on one RTX 4090 with llama.cpp/CUDA, output quality was not assessed, and the 50 ms NVML sampling limits interpretation of short prefill measurements.
Evidence
- Peer-reviewedEnergies2026-10-16
How should this claim be treated?
Truvace Impact Record TRV-2026-1741, v1: “Does Lower Precision Always Save Energy? Benchmarking Energy–Performance Trade-Offs in Weight-Quantized Large Language Model Inference on a GPU.” Truvace, 2026-10-17. /record/TRV-2026-1741 (accessed at citation time). sha256 82bef67fd20c5f7f…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-1741 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.