[BLOG] Variant Type in Apache Parquet for Semi-Structured Data

[aihuaxu]

I followed geospatial blog to create this PR.

This is to add this blog post of Variant in Parquet with the initial draft in this google doc.

• Closes Blog on Variant #147

[alamb]

Thank you @aihuaxu -- this looks (amazing) to me, though I am somewhat biased 😆

Sorry for the delay, I am out this week.

I left some some comments but I don't think any of them are actually required to publish this post. I also made some small proposed changes in here for your consideration:

• aihuaxu#1

I'll also post to the Parquet mailing list soliciting additional feedback

How about we shoot to publish this in about a week (Feb 26, 2026?)

[alamb]

I also left a note on the parquet dev list asking for additional feedback: https://lists.apache.org/thread/vms8ohk4onl8fom9n9zkql2ctgdbz3lo

[emkornfield]

It might be better to motivate why variant with an example here. event_data is mentioned below, so maybe we can talk about storing logged Event data, that has a structure that might evolve because new events are added, or fields are added or removed from a specific event type.

Then motivate this with a query. e.g. Select event[timestamps] where event[User] = 5. Then you can explain that if the events where just stored as JSON (there original format) we require partings the JSON to do any processing. BSON could be an improvement but is still potentially sub-optimal because you end up repeated storage of common keys.

[aihuaxu]

I updated a little bit. Please take a look.

[emkornfield]

Overall, looks good. Lets please check the status of Arrow C++ before publishing.

The other comments are not a blocker but I think this would be a more interesting read by putting the motivating examples first and discuss why existing types don't support the use-case well.

[emkornfield]

Thanks for the changes, mostly looks good. A few more non-required suggestions on making the blog more focused.

[wgtmac]

As we are using VARIANT(1) here, should we also update the spec to be consistent? https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#variant

[alamb]

That is a good point -- I double checked at the (1) is the "Variant Version":

https://github.com/apache/parquet-format/blob/38818fa0e7efd54b535001a4448030a40619c2a3/src/main/thrift/parquet.thrift#L409-L413

/**
 * Embedded Variant logical type annotation
 */
struct VariantType {
  // The version of the variant specification that the variant was
  // written with.
  1: optional i8 specification_version
}

I made a PR to update

• Correct VARIANT Logical Type annotation in Parquet examples parquet-format#555

[alamb]

I pushed a few small changes:

1. Adjusted the title to better mirror the geospatial blog title "Native Geospatial Types in Apache Parquet". New title is "Variant Type in Apache Parquet for Semi-Structured Data"
2. Removed the ## Introduction heading (left the content, just removed the heading)
3. Tweaked some table headings to look better visually
4. vertically aligned some comments in the examples

[alamb]

I also had to update this example when I tested using spark-sql:

spark-sql (default)> INSERT INTO sensor_readings VALUES
                   > (1, '2026-01-28 10:00:00',
                   >      PARSE_JSON('{"sensor_id": "T001", "temp": 72.5, "unit": "F", "battery": 95}')),
                   >     (2, '2026-01-28 10:00:05',
                   >      PARSE_JSON('{"sensor_id": "M001", "motion_detected": true, "confidence": 0.95, "zone": "entrance"}')),
                   >     (3, '2026-01-28 10:00:10',
                   >      PARSE_JSON('{"sensor_id": "C001", "image_url": "s3://bucket/img_001.jpg", "objects_detected": ["person", "vehicle"]}'));
[INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST] Cannot write incompatible data for the table `spark_catalog`.`default`.`sensor_readings`: Cannot safely cast `timestamp` "STRING" to "TIMESTAMP". SQLSTATE: KD000

So I added an explicit ::timestamp cast in 43fc111

spark-sql (default)> INSERT INTO sensor_readings VALUES
                   >     (1, '2026-01-28 10:00:00'::timestamp,
                   >      PARSE_JSON('{"sensor_id": "T001", "temp": 72.5, "unit": "F", "battery": 95}')),
                   >     (2, '2026-01-28 10:00:05'::timestamp,
                   >      PARSE_JSON('{"sensor_id": "M001", "motion_detected": true, "confidence": 0.95, "zone": "entrance"}')),
                   >     (3, '2026-01-28 10:00:10'::timestamp,
                   >      PARSE_JSON('{"sensor_id": "C001", "image_url": "s3://bucket/img_001.jpg", "objects_detected": ["person", "vehicle"]}'));
Time taken: 1.067 seconds

[alamb]

Ok, I think this PR has had enough review and all the outstanding comments have been addressed. I'll update the date and then publish it.

We can address any additional feedback as follow on PRs