Truace
Subscribe
52/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Resizing AI chatbot caches by the hour cut carbon use by 15.1% in tests on France's grid

Study title: Cache Your Prompt When It's Green - Carbon-aware Caching for Large Language Model Serving

Evidence type
Benchmark or lab evaluation
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2628cite or check it

Study title: Cache Your Prompt When It's Green - Carbon-aware Caching for Large Language Model Serving

Researchers at the University of Waterloo and Purdue built GreenCache, a system that decides how much SSD storage a language-model service should use to hold reusable context. In tests on France's grid, it cut average carbon emissions by 15.1% when serving Llama-3 70B, with a peak reduction of 25.3%, compared with always using the full cache.

Read the study abstract

As large language models (LLMs) become widely used, their environmental impact — especially carbon emission — has attracted more attention. Prior studies focus on compute-related carbon emissions. In this paper, we find that storage is another key contributor. LLM caching, which saves and reuses KV caches for repeated context, reduces operational carbon by avoiding redundant computation. However, this benefit comes at the cost of embodied carbon from high-capacity, high-speed SSDs. As LLMs scale, the embodied ca…

The gain

In tests replaying real request and grid traces, GreenCache cut average carbon emissions of serving Llama-3 70B by 15.1% on France's grid, up to 25.3%, versus a full-size cache, while keeping over 90% of requests within latency targets.

In brief

The work targets a gap in earlier research, which focused on compute emissions. Caching saves computation, but the SSDs add embodied carbon from manufacturing; in the team's setup, SSDs made up 76.6% of the server's embodied carbon. The balance shifts with the grid: a 16 TB cache cut emissions 7.5% in the MISO grid but raised them 16.5% in France.

Gains were smaller where electricity is dirtier. On California's CISO grid, reductions reached up to 6.91% for Llama-3 70B. The results come from simulated one-day traces on one four-GPU server, not a live production service.

Main points

  1. GreenCache resizes the cache hourly using predicted carbon intensity and request load, solving an integer linear program that requires at least 90% of requests to meet latency targets.
  2. Against a full 16 TB cache, it cut carbon by an average 12.6% for multi-turn conversation and 9.4% and 5.6% for document comprehension at two skew levels on Llama-3 70B.
  3. With Llama-3 8B, reductions were 10.8%, 7.6% and 9.7% for the same three scenarios.
  4. In France, GreenCache used 15 TB less SSD than the full cache at 12 AM, producing the 25.3% peak reduction.
  5. A fixed 16 TB cache raised emissions 16.5% in low-carbon France (33 gCO2e/kWh) but cut them 7.5% in MISO (485 gCO2e/kWh).
  6. Its carbon-aware replacement policy reached up to 9% higher cache hit rates than the default least-recently-used policy at smaller cache sizes.
  7. Prediction errors had little effect: carbon-intensity errors cost 0.0064% of savings, load errors 0.20% and profiling errors 0.79% on average.
  8. The solver took 7.03 seconds per resizing decision, small against the hourly schedule.

The full story

The team tested two tasks: multi-turn chat using the ShareGPT dataset and document question answering using TriviaQA. Request rates came from a 24-hour Azure trace scaled to fit the server, and grid carbon intensity from the CarbonCast dataset for four grids: France, Finland, Spain and California (CISO). Experiments ran on a system with four Nvidia L40 GPUs and up to 16 TB of SSD, built on the open-source LMCache system.

Savings varied by grid and task. Over a full day, GreenCache cut emissions by 6.9 to 20.4% in multi-turn conversation and 3.2 to 16.2% in document comprehension on Llama-3 70B. Low-carbon grids such as France and Finland saw 14.4 to 20.3% savings because SSD manufacturing emissions weigh more there. Without any cache, carbon sometimes came out lower, but that setup repeatedly broke the latency targets.

The system also depends on forecasts. It predicts load with a seasonal time-series model and carbon intensity with the EnsembleCI predictor, looking up to 24 hours ahead so new cache space can warm up. The load predictor's average error was 4.3%, and carbon-intensity errors ranged from 6.8% to 15.3% across the four grids.

Longer resizing intervals sharply reduced savings, since the cache must stay large enough for peak demand during the whole interval. The effect was stronger for document comprehension, which needs larger caches.

Sources

1 on file
  1. ACM SIGMETRICS Performance Evaluation Reviewdoi:10.1145/3820171.3806850
    Peer-reviewed2026-05-26

Talk it over

Four questions to start onWho gets this benefit, and who was left out of the study or story?

Start with a question

Resizing AI chatbot caches by the hour cut carbon use by 15.1% in tests on France's gridhttps://truvace.com/item/cache-your-prompt-when-its-green-carbon-aware-caching-for-large-language-model-s

  1. Who gets this benefit, and who was left out of the study or story?

    The gain In tests replaying real request and grid traces, GreenCache cut average carbon emissions of serving Llama-3 70B by 15.1% on France's grid, up to 25.3%, versus a full-size cache, while keeping over 90% of requests within latency targets.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a benchmark or lab test, from ACM SIGMETRICS Performance Evaluation Review.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The evaluation used simulated one-day traces on a single four-GPU server with a 16 TB SSD cap, scaled-down Azure request rates and one date's carbon intensity, so real production savings may differ.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Climate