Transform processor - memory usage continue to increase when transforming high cardinality Redis Calls

[manojksardana]

Hi All,

we are using open telemetry collector to receive traces from the application instrumented using otel SDK. To have optimal cardinality for the traces, we are trying to used transform processor which transform the traces based on some pattern and replace then with a generic URL. Post this replacement, the traces are send to spanmetric connector to convert the traces into metrics based on the few predefined dimensions.

Below is our definition of the pipelines

    pipelines:
      logs/k8s_events:
        receivers: [k8s_events]
        processors: [batch, resource/k8s_events]
        exporters: [loki/grafana]
      metrics:
        exporters:
          - logging
          - prometheusremotewrite/grafana
        processors:
          - transform/commerce
          - filter/metrics
          - memory_limiter
          - k8sattributes
          - attributes/k8s_cluster
          - batch
        receivers:
          - otlp
          - prometheus
      metrics/span:
          exporters:
          - logging
          - otlphttp/metrics
          processors:
          - transform/commerce
          #- filter/metrics2
          - k8sattributes
          - memory_limiter
          - batch
          receivers:
          - spanmetrics/http
      metrics/db:
          exporters:
          - logging
          - otlphttp/metrics
          processors:
          - transform/commerce
          - filter/metrics3
          - k8sattributes
          - memory_limiter
          - batch
          receivers:
          - spanmetrics/db
      traces:
        exporters:
          - logging        
          - otlp
          - spanmetrics/http
          - spanmetrics/db
        processors:
          - memory_limiter
          - k8sattributes
          - transform/commerce
          - filter/commerce
          - batch
        receivers:
          - otlp

Now as part of of the transform processor we have the following operations

    transform/commerce:
      error_mode: ignore
      metric_statements:
          - context: resource
            statements:
            - truncate_all(attributes, 2048)
          - context: datapoint
            statements:
            - truncate_all(attributes, 2048)
      trace_statements:
          - context: resource
            statements:  
            - truncate_all(attributes, 2048)
          - context: span
            statements:
            - truncate_all(attributes, 2048)
            - set(attributes["http.route"], attributes["url.path"])
            - replace_match(attributes["http.route"], "/wcs/resources/store/AU/order/*", "/wcs/resources/store/AU/order/{orderID}") 
            - replace_match .......
            - replace_match........
             **- set(name, attributes["db.system"]) where attributes["db.system"] == "redis"**  
          - set(name, Concat([attributes["http.request.method"], attributes["http.route"]], " ")) where attributes["url.scheme"] == "https" or attributes["url.scheme"] == "http" 

One of the command above (in bold) is converting all the high cardinality redis calls to give same span name called redis. As the number of calls are very high here, i expect all the calls get converted to same span name and get into same metric calculations by span metric later.

however this result into ever increasing memory and end up getting into OOM error. If I remove this call and add it to filter processor to filter it out instead of passing it to spanmetric as single span name, memory remain under control and consistent.

  filter/commerce:
      error_mode: ignore
      traces: 
        span:
          - IsMatch(attributes["db.system"], "redis")

I am not sure if its the high cardinality of redis call resulting into some memory leak at transform or spanmetric processor level, or there is something else wrong which i am not able to capture.

Any help here ?

Thanks
Manoj Sardana

[trask]

hi @manojksardana, I'd suggest asking in the https://github.com/open-telemetry/opentelemetry-collector-contrib repo where the spanmetricsconnector lives