Skip to content

Comments

fix(getFLBPluginContext): use proper FLBPluginGetContext API and improve data parsing#109

Open
yuzone wants to merge 5 commits intoGoogleCloudPlatform:mainfrom
yuzone:fix/use-proper-flbplugin-getcontext-api
Open

fix(getFLBPluginContext): use proper FLBPluginGetContext API and improve data parsing#109
yuzone wants to merge 5 commits intoGoogleCloudPlatform:mainfrom
yuzone:fix/use-proper-flbplugin-getcontext-api

Conversation

@yuzone
Copy link

@yuzone yuzone commented Jan 23, 2026

Summary

This PR fixes plugin context retrieval by switching to the official Fluent Bit Go API (output.FLBPluginGetContext()).
It also improves record parsing by safely handling nil, recursively processing nested structures, and normalizing timestamps for fluentd→fluent-bit migration compatibility.

Background

  1. The previous implementation used direct pointer casting to retrieve the plugin context, which could lead to incorrect behavior. The proper output.FLBPluginGetContext() API should be used instead.

  2. The parseMap function did not handle nil input safely, potentially causing panics.

  3. Elements inside slices (nested maps, byte slices) were not being recursively processed by parseMap, which could cause issues when serializing data for BigQuery.

  4. When migrating from fluentd (fluent-plugins-nursery/fluent-plugin-bigquery) to fluent-bit, timestamp fields may be emitted as Unix seconds instead of the microseconds expected by the BigQuery Storage Write API, resulting in incorrect timestamp values (e.g., dates appearing as 1970-01-01).

Changes

  • getFLBPluginContext: Replace direct pointer casting with output.FLBPluginGetContext() to correctly retrieve the plugin context
  • parseMap: Add nil check to safely return nil for nil input
  • parseSlice: Add new function to recursively process slice elements (byte slice conversion, nested maps/slices), with nil check
  • parseMap: Update to call parseSlice for []interface{} values, enabling full recursive processing
  • convertTimestampFields: Add new function to convert timestamp fields from Unix seconds to microseconds for BigQuery compatibility (values less than year 2100 in seconds are converted)
  • maxUnixSeconds: Add constant for timestamp conversion threshold (4102444800 = 2100-01-01 00:00:00 UTC)

Test Plan

  • Added unit tests for parseMap covering nil input, empty maps, nested structures, and various data types
  • Added unit tests for parseSlice covering nil input, empty slices, byte slice conversion, nested maps, and mixed types
  • Added unit tests for convertTimestampFields covering:
    • Different numeric types (int64, int, uint64, float64)
    • Boundary values (maxUnixSeconds threshold)
    • Already-microsecond values (no conversion)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant