Truace
Subscribe

Does Lower Precision Always Save Energy? Benchmarking Energy–Performance Trade-Offs in Weight-Quantized Large Language Model Inference on a GPU

Large language model (LLM) inference is an increasing component of AI energy demand, while the energy effects of weight quantization depend on workload phase and implementation. We benchmarked four LLM families—Gemma 4 E4B IT, Qwen3.5 4B, Nemotron 3 Nano 4B, and Ministral 3 3B Instruct—across Q4_K_M, Q6_K, Q8_0, and BF16 weight representations on a single NVIDIA GeForce RTX 4090 using llama.cpp/CUDA. Five workloads separated prefill from autoregressive generation, and GPU energy was estimated from NVML telemetry…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1741· cite & verify
Does Lower Precision Always Save Energy? Benchmarking Energy–Performance Trade-Offs in Weight-Quantized Large Language Model Inference on a GPU
"Leistungsanalyse NVIDIA GeForce RTX 4090 (极客湾Geekerwan) 14" by 极客湾Geekerwan is licensed under CC BY 3.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/3.0/.

In brief

Researchers benchmarked four small language model families at four weight precisions (Q4_K_M, Q6_K, Q8_0, BF16) on a single consumer GPU. They ran five workloads that separated prompt prefill from token-by-token generation, and estimated GPU energy from NVML telemetry.

The results suggest lower precision does not automatically save energy in every phase, but it clearly helps during generation, where inference energy tends to accumulate. The findings come from a narrow setup, and output quality was not evaluated, so the efficiency gains cannot yet be weighed against any accuracy loss.

Main points

  1. Generation scenarios showed significant effects of model family, precision, and their interaction on the primary endpoints.
  2. In the 512-token prefill workload, no statistically significant differences were detected under the measurement protocol.
  3. In the 1024-token prefill workload, precision affected total energy and mean GPU power.
  4. Ministral 3 3B Instruct–Q4_K_M ranked first in the TOPSIS multi-criteria ranking.
  5. Nemotron 3 Nano 4B–Q4_K_M required less GPU memory than the other configurations.

The gain

On one RTX 4090 using llama.cpp/CUDA, 4-bit Q4_K_M weights cut dynamic energy per token during generation by 63.6–85.7% versus BF16 and raised throughput across all four tested model families.

The rundown

The study tested Gemma 4 E4B IT, Qwen3.5 4B, Nemotron 3 Nano 4B and Ministral 3 3B Instruct on one NVIDIA GeForce RTX 4090 with llama.cpp and CUDA. Energy was estimated from NVML telemetry across five workloads.

Prefill results were mixed: the shorter 512-token case showed no significant differences, while the 1024-token case showed precision effects on total energy and mean GPU power. This is why the authors question whether lower precision always saves energy.

For generation, the 4-bit configuration was the most efficient against BF16 in every family, and a TOPSIS ranking placed Ministral 3 3B Instruct–Q4_K_M first, with Nemotron 3 Nano 4B–Q4_K_M using the least GPU memory.

Sources

  1. Peer-reviewedEnergies2026-10-16

The debate