Schema globbing

[pmetras]

This adds the ability to map table names to paths in a IMPORT FOREIGN SCHEMA command, with the option tables_map.

Its value is a space-separated list of key=value values with table names as keys and Parquet files paths as values. Like for filename, file globbing can be used, and like with Linux PATH variable, the colon : is used to separate multiple paths within a value.

import foreign schema "/path/to/directory"
from server parquet_srv
into public options (tables_map 'table1=/path/to/directory/2022/*/*.parquet:/path/to/directory/2023/*/*.parquet table2=/path/to/directory{2024,2025}/*/*.parquet')
;

As a security, the extension checks that the paths in the tables_map are within the external schema path, preventing accessing files elsewhere on the server. Also, if foreign schema limit to or except clauses are used specifiying table names, only these names will be considered in the tables_map.

Tables created through the tables_map options are identical as if they were created with a create foreign table command with a filename option. All other options from the import foreign schema are transmitted to the create foreign table.

Each table must have at least one Parquet file when created, to query Parquet field and map them to colomun names. Of course, files globbing is evaluated at query-time, so one can add files to a directory and have them considered in the query results.

This branch is based on code in #96, so working with PostgreSQL 17. It also adds support for UUID columns.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pandas@​3.0.0
	pyarrow@​23.0.0

View full report