Skip to content

Conversation

@shashjar
Copy link
Member

@shashjar shashjar commented Dec 12, 2025

Using an EndpointTraceItemTable RPC query to fetch the count of occurrences for a specific group in the current hour.

Tested locally to ensure that the count aggregation is coming back correctly from EAP:

sentry.utils.snuba_rpc: Running a EndpointTraceItemTable RPC query (rpc_query={'meta': {'organizationId': '1', 'cogsCategory': 'issues', 'referrer': 'issues.escalating', 'projectIds': ['2'], 'startTimestamp': '2025-12-12T00:00:00Z', 'endTimestamp': '2025-12-12T00:01:01.209821Z', 'traceItemType': 'TRACE_ITEM_TYPE_OCCURRENCE'}, 'columns': [{'aggregation': {'aggregate': 'FUNCTION_COUNT', 'key': {'type': 'TYPE_INT', 'name': 'group_id'}}, 'label': 'count'}], 'filter': {'comparisonFilter': {'key': {'type': 'TYPE_INT', 'name': 'group_id'}, 'op': 'OP_EQUALS', 'value': {'valInt': '3'}}}, 'limit': 1} referrer='issues.escalating' organization_id=1 trace_item_type=7)

sentry.utils.snuba_rpc: Table RPC query response (rpc_rows=1 page_token=offset: 1
 meta=request_id: "ce96e6d07c544b2eaec75707ff21a5a4"
query_info {
  stats {
    progress_bytes: 1674
  }
  metadata {
  }
}
downsampled_storage_meta {
}
)

sentry.issues.escalating.escalating: 
get_group_hourly_count_eap: raw response (group_id=3 raw_response='column_values {\n  attribute_name: "count"\n  results {\n    val_double: 6\n  }\n}\npage_token {\n  offset: 1\n}\nmeta {\n  request_id: "ce96e6d07c544b2eaec75707ff21a5a4"\n  query_info {\n    stats {\n      progress_bytes: 1674\n    }\n    metadata {\n    }\n  }\n  downsampled_storage_meta {\n  }\n}\n')

sentry.issues.escalating.escalating: 
is_escalating: old_count=6, new_count=6

To roll this out, we can:

  1. Merge in this PR (no change in behavior in any region)
  2. Flip the occurrences.eap-reads.enabled option on for the S4S region only
  3. At this point, we'll be calculating both values in S4S but sticking with the Snuba value as the source of truth
  4. We should see the issues.escalating.eap_mismatch metric start getting recorded and then converge to 0 over time (as EAP gets up-to-speed on all the groups within retention)
  5. At that point (01/21/2026, 90 days after 10/23/2025), we should be able to remove the legacy read path and use EAP only

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Dec 12, 2025
return int(hourly_count)


# TODO: probably some stuff to abstract out into EAP utils between here and src/sentry/eventstream/eap.py (and likely other read paths as well)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we wait until we've worked on a couple more read paths to consolidate this logic? I feel like it's hard at this point to say exactly what kind of abstraction we want.

@codecov
Copy link

codecov bot commented Dec 12, 2025

Codecov Report

❌ Patch coverage is 97.72727% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/issues/escalating/escalating.py 97.67% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master   #104830   +/-   ##
========================================
  Coverage   80.49%    80.49%           
========================================
  Files        9370      9370           
  Lines      402213    402255   +42     
  Branches    25837     25837           
========================================
+ Hits       323744    323788   +44     
+ Misses      78021     78019    -2     
  Partials      448       448           

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants