feat: Measure system errors using a counter

[morgan-wowk]

TL;DR

Added OpenTelemetry metrics instrumentation to track execution system errors in the orchestrator.

Business value

We will be able to track the rate of system errors and respond to high or increasing rates.

Future iterations

In the future we will emit metrics for state transitions in general, and use measurement attributes to create dimensions on status, then we will have the option to deprecate to deprecate this measurement specific to system errors.

What changed?

• Created a new metrics module (cloud_pipelines_backend/instrumentation/metrics.py) that defines an orchestrator meter and an execution_system_errors counter instrument
• Integrated the metrics counter into the record_system_error_exception function in orchestrator_sql.py to increment the counter when system errors occur
• Added comprehensive OpenTelemetry strategy documentation (otel_strategy.md) covering best practices for meters, instruments, temporality, and aggregation

How to test?

• Cherry pick the OTel stack from https://app.graphite.com/github/pr/TangleML/tangle/101/feat-Add-development-observability-stack-for-testing-OpenTelemetry-with-Jaeger%2C-and-Prometheus
• Run the stack (docker-compose up -d)
• Run export TANGLE_OTEL_TRACE_EXPORTER_ENDPOINT="http://localhost:4317" && export TANGLE_OTEL_TRACE_EXPORTER_PROTOCOL="grpc" && export TANGLE_OTEL_METRIC_EXPORTER_ENDPOINT="http://localhost:4317" && export TANGLE_OTEL_METRIC_EXPORTER_PROTOCOL="grpc"
• Trigger execution nodes that result in SYSTEM_ERROR status and verify that the execution.system_errors metric is incremented in your OpenTelemetry metrics backend. Add a line raise RuntimeError("Temporary") to the start of https://github.com/TangleML/tangle/blob/system-error-metric/cloud_pipelines_backend/orchestrator_sql.py#L209

Why make this change?

This enables monitoring and alerting on system errors in pipeline executions, providing better observability into the health and reliability of the orchestrator component. The metrics follow OpenTelemetry semantic conventions and provide a foundation for expanding observability coverage across the application.

[morgan-wowk]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: Measure system errors using a counter #127 👈 (View in Graphite)
• feat: Setup metric provider for manual and auto-instrumented measurements #126
• chore!: Refactor OTel exporters to be signal-specific #125
• feat: Set service version on OTel data #124
• chore: Refactor OTel concerns in preparation for metric provider #123
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.