[FEATURE] all uncompressed file sizes for different scale factors

[MrPowers]

I computed the file sizes for scale factor 1 as follows:

import pandas as pd

def get_mem(df):
    # Get memory usage in bytes
    memory_usage = df.memory_usage(deep=True).sum()
    
    print(f"Memory size: {memory_usage:,} bytes")
    print(f"Memory size: {memory_usage / (1024**2):.2f} MB")

df = pd.read_parquet('building.parquet')
get_mem(df)

Here are the file sizes for SF1:

Table	Size (sf1)
building	4.06 MB
customer	9.30 MB
driver	0.15 MB
trip	3547.67 MB
vehicle	0.02 MB
zone	1388.61 MB

Perhaps we can expose a helper to fetch the uncompressed data size and add this info to the documentation?

[prantogg]

Thanks @MrPowers!

I agree we should start by adding the uncompressed table sizes to the SpatialBench docs so users have clear expectations about data volume at each scale factor.

On the calculation side: the pandas memory_usage(deep=True) method reports the in-memory pandas size, which can differ significantly from the actual Parquet footprint. A more accurate way to compute the true uncompressed Parquet size is to read the column-chunk metadata via PyArrow:

import pyarrow.parquet as pq

def parquet_uncompressed_bytes(path):
    pf = pq.ParquetFile(path)
    uncomp = 0
    comp = 0

    for rg in range(pf.num_row_groups):
        rg_meta = pf.metadata.row_group(rg)
        for c in range(rg_meta.num_columns):
            col = rg_meta.column(c)
            uncomp += col.total_uncompressed_size
            comp += col.total_compressed_size

    return uncomp, comp

uncompressed, compressed = parquet_uncompressed_bytes("building.parquet")
print("Uncompressed bytes:", uncompressed)
print("Compressed bytes:", compressed)
print("Compression ratio:", uncompressed / compressed)

This gives the sizes recorded by the Parquet writer.

I’ll follow up with docs updates soon. Thanks again for raising this!

[prantogg]

File sizes for SF=1 using the above method -

Table	Size (sf1)
building	2.377 MB
customer	2.479 MB
driver	0.039 MB
trip	470.990 MB
vehicle	0.005 MB
zone	1345.672 MB

Total uncompressed size of SF1 = 1821.5625 MB = 1.7789 GB

[MrPowers]

@prantogg - didn't realize you can get the uncompressed size from the Parquet file. That is way better! Thank you!

[prantogg]

Yeah its pretty neat! A lot of engines rely on that metadata for estimating scan cost and planning! 🤓