Skip to content

Conversation

@Ziinc
Copy link
Contributor

@Ziinc Ziinc commented Dec 19, 2025

implements flatmap usage as suggested by @edgurgel 🙏

This PR also adds in much performance improvements around the processing of opentelemetry data at ingestion time.

This converts the underlying typing of the column to a timestamp instead of storing iso strings. This is a more appropriate way of storing time-related columns. This will require manual conversion of start_time and end_time columns of opentelemetry ingestion columns, as these columns will not automatically conver their data types.

Before:

##### With input mixed (5 each x 1kdp) #####
Name                           ips        average  deviation         median         99th %
handle_batch (merge)          8.35      119.75 ms    ±15.68%      118.93 ms      167.52 ms

Memory usage statistics:

Name                    Memory usage
handle_batch (merge)        64.28 MB

**All measurements for memory usage were the same**

Reduction count statistics:

Name                         average  deviation         median         99th %
handle_batch (merge)          4.94 M     ±0.84%         4.92 M         5.01 M

After:

##### With input mixed (5 each x 1kdp) #####
Name                   ips        average  deviation         median         99th %
handle_batch        285.18        3.51 ms    ±20.87%        3.33 ms        6.31 ms

Memory usage statistics:

Name            Memory usage
handle_batch         9.27 MB

**All measurements for memory usage were the same**

Reduction count statistics:

Name                 average  deviation         median         99th %
handle_batch        764.78 K     ±0.68%       763.99 K       808.70 K

##### With input sum (5 each x 1kdp) #####
Name                   ips        average  deviation         median         99th %
handle_batch        1.57 K      636.91 μs    ±18.29%      606.71 μs     1104.84 μs

Memory usage statistics:

Name            Memory usage
handle_batch         2.08 MB

@Ziinc Ziinc changed the base branch from main to perf/benchmark-otel-metrics January 7, 2026 18:41
@Ziinc Ziinc changed the title perf: flatmap for OtelMetric processor perf: OtelMetric processor, Otel improvements Jan 16, 2026
@Ziinc Ziinc changed the base branch from perf/benchmark-otel-metrics to main January 16, 2026 07:57
1609459200000000
"""
def to_microseconds(raw) when is_integer(raw) do
case Integer.digits(raw) |> Enum.count() do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From memory I remember benchmarking this and I think that doing to_integer + byte_size was quicker. For negative number you have to discount the -

I also remember being more predictable doing to_string |> byte_size as Enum.count is O(n) and byte_size is O(1)

May be worth benchmarking if this is called many many times by Logflare

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants