Accuracy

[pmk7]

Hi,

This study below claims there is an energy consumption error rate of around 20% for CodeCarbon, is this a known limitation due to the exclusion of other system components when measuring energy consumption?

https://dl.acm.org/doi/10.1145/3665314.3670809

[LuisBlanche]

Hi we have ongoing work to assess the accuracy of codecarbon. Keep in mind that codecarbon uses approximations in a lot of cases depending on the platform and the setup. We try to use the most precise measure tools when they are available but e sometimes have to rely on modeling to provide an estimate. That being said, the errors are not systematic, meaning we cannot simply say that there is a 20% under estimation for every measure, because depending on the system and the installed tools, we can also sometimes overestimate. For instance the current setting for RAM power measurement is a static 3W / 8GB of RAM which we know is leading to overestimated emissions for big clusters with a lot of RAM where economies of scale make them less consuming.

[cianc]

https://arxiv.org/pdf/2509.22092 is another more recent paper claiming similar levels of under-estimation from codecarbon. The author claims this comes from PSU, cooling and peripheral overhead. This actually contradicts the paper listed above which claims codecarbon is overestimating power usage.
Has there been any discussion about somehow estimating these factors or adding a parameter for user estimation?

[inimaz]

Related to your comment @cianc , the author of that paper posted on Linkedin a summary. See Codecarbon's reply here

In that line, we proposed to set the default value of the PUE to 1.2 to compensate for the missing 20%. Or to create a different parameter experimental_compensation that we keep updating with future studies. What do you think?

[cianc]

Oh nice! That's a really helpful conversation.

My 2 cents:

• PUE is supposed to cover non-server power usage, and doesn't apply to on-server power and cooling IMO. I don't think we should conflate these two things.
• Agree that we should have a parameter to account for the parts of server power consumption that aren't CPU/RAM/GPU. I like more descriptive names, maybe non-compute-power-multiplier, but I am bad at naming things. This could default to 1.2
• This change needs to be announced well in advance since it will cause a step change in reported values. Probably a good idea to default it to 1.0 for a while, with a big warning that it will be flip in the future.

Has there been any discussion with the authors of the paper that claim codecarbon is overestimating? That paper says "This disparity can be attributed to these tools ignoring, overestimating, or underestimating the power consumed by components other than CPU and GPU, including memory, network, fans, and storage" - AFAIK the only one of those codecarbon accounts for is RAM. Could that really cause such a large 10-20% overestimation? It would be good to know how they configured codecarbon and instrumented their code.

[cianc]

Maybe this new multiplier is a good for v4 since we're keeping track of other breaking changes: #982

[inimaz]

Indeed! We can even set this as you suggested to 1 in this current version, and update it to 1.X in the future breaking change.