Skip to content

chore(ai): Track reason for missing cost in ai cost calculation #5611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shellmayr wants to merge 6 commits into
base: master
Choose a base branch
from
+38 −11
Open

chore(ai): Track reason for missing cost in ai cost calculation #5611

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
89a5198
chore(ai): track reason for missing cost in ai cost calculation
shellmayr Feb 4, 2026
e9d2f70
add changelog
shellmayr Feb 5, 2026
0e978b2
Merge branch 'master' into shellmayr/chore/track-reason-for-no-cost-i…
shellmayr Feb 5, 2026
9b5ea29
add span pipeline
shellmayr Feb 5, 2026
a86743e
add missing metric emission when model id is not present
shellmayr Feb 6, 2026
9b2d09e
add missing metric emission when model id is not present
shellmayr Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
- Add `gen_ai.cost_calculation.result` metric to track AI cost calculation outcomes by integration and platform. ([#5560](https://github.com/getsentry/relay/pull/5560))
- Normalizes and validates trace metric names. ([#5589](https://github.com/getsentry/relay/pull/5589))
- Add manual category to cost calculation metric origin tag ([#5603](https://github.com/getsentry/relay/pull/5603))
- Differentiate between reasons for missing cost calculation([#5611](https://github.com/getsentry/relay/pull/5611))
Copy link
Member

@Dav1dde Dav1dde Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to skip the changelog for changes which aren't "outside" visible


## 26.1.0

Expand Down
20 changes: 14 additions & 6 deletions relay-event-normalization/src/eap/ai.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use relay_protocol::Annotated;

use crate::ModelCosts;
use crate::span::ai;
use crate::statsd::{map_origin_to_integration, platform_tag};
use crate::statsd::{Counters, map_origin_to_integration, platform_tag};

/// Normalizes AI attributes.
///
Expand Down Expand Up @@ -120,7 +120,18 @@ fn normalize_ai_costs(attributes: &mut Attributes, model_costs: Option<&ModelCos
.and_then(|v| v.as_str())
.and_then(|model| model_costs?.cost_per_token(model));

let Some(model_cost) = model_cost else { return };
let integration = map_origin_to_integration(origin);
let platform_tag = platform_tag(platform);

let Some(model_cost) = model_cost else {
relay_statsd::metric!(
counter(Counters::GenAiCostCalculationResult) += 1,
result = "calculation_no_model_cost_available",
integration = integration,
platform = platform_tag,
);
return;
};
Copy link

@cursor cursor bot Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EAP path missing distinct "no model ID" metric

Low Severity

The EAP path emits "calculation_no_model_cost_available" when no model ID attribute exists (GEN_AI_REQUEST_MODEL/GEN_AI_RESPONSE_MODEL), but the span path emits "calculation_no_model_id_available" for the same scenario. The PR introduces "calculation_no_model_id_available" as a distinct metric (documented in statsd.rs and used in normalize/span/ai.rs), but the EAP path doesn't distinguish this case. This inconsistency makes telemetry data harder to interpret since the same condition produces different metric values.

Fix in Cursor Fix in Web


let get_tokens = |key| {
attributes
Expand All @@ -137,10 +148,7 @@ fn normalize_ai_costs(attributes: &mut Attributes, model_costs: Option<&ModelCos
output_reasoning_tokens: get_tokens(GEN_AI_USAGE_OUTPUT_REASONING_TOKENS),
};

let integration = map_origin_to_integration(origin);
let platform = platform_tag(platform);

let Some(costs) = ai::calculate_costs(model_cost, tokens, integration, platform) else {
let Some(costs) = ai::calculate_costs(model_cost, tokens, integration, platform_tag) else {
return;
};

Expand Down
24 changes: 20 additions & 4 deletions relay-event-normalization/src/normalize/span/ai.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ pub fn calculate_costs(
if !tokens.has_usage() {
relay_statsd::metric!(
counter(Counters::GenAiCostCalculationResult) += 1,
result = "calculation_none",
result = "calculation_no_tokens",
integration = integration,
platform = platform,
);
Expand Down Expand Up @@ -189,11 +189,20 @@ fn extract_ai_model_cost_data(
origin: Option<&str>,
platform: Option<&str>,
) {
let Some(model_cost) = model_cost else { return };

let used_tokens = UsedTokens::from_span_data(&*data);
let integration = map_origin_to_integration(origin);
let platform = platform_tag(platform);

let Some(model_cost) = model_cost else {
relay_statsd::metric!(
counter(Counters::GenAiCostCalculationResult) += 1,
result = "calculation_no_model_cost_available",
integration = integration,
platform = platform,
);
return;
};

let used_tokens = UsedTokens::from_span_data(&*data);
let Some(costs) = calculate_costs(model_cost, used_tokens, integration, platform) else {
return;
};
Expand Down Expand Up @@ -288,6 +297,13 @@ fn extract_ai_data(
origin,
platform,
)
} else {
relay_statsd::metric!(
counter(Counters::GenAiCostCalculationResult) += 1,
result = "calculation_no_model_id_available",
integration = map_origin_to_integration(origin),
platform = platform_tag(platform),
);
Copy link

@cursor cursor bot Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing metric when model costs config is unavailable

Medium Severity

When model_costs is None in enrich_ai_span_data, no cost calculation metric is emitted. The code silently skips the extract_ai_data call, meaning no tracking occurs for this failure case. This is inconsistent with the EAP path in eap/ai.rs where the "calculation_no_model_cost_available" metric is correctly emitted when model_costs is None. This defeats the PR's purpose of differentiating between reasons for missing cost calculation.

Fix in Cursor Fix in Web

}
}

Expand Down
4 changes: 3 additions & 1 deletion relay-event-normalization/src/statsd.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ pub enum Counters {
/// `calculation_negative`,
/// `calculation_zero`,
/// `calculation_positive`,
/// `calculation_none`
/// `calculation_no_tokens`
/// `calculation_no_model_cost_available`
/// `calculation_no_model_id_available`
/// - `integration`: The integration used for the cost calculation.
/// - `platform`: The platform used for the cost calculation.
GenAiCostCalculationResult,
Expand Down
Loading