ForgeCurves Apache Parquet Guide

Apache Parquet & Polars Compatibility

Standard columnar binary dataset storage, zero-copy selective projection, sub-millisecond footer metadata extraction, and bit-exact Polars interoperability in ForgeCurves.

1. First-Class In-Notebook Verbs

ForgeCurves integrates native Apache Parquet operations directly into the computational notebook kernel with standard syntax matching Octave and MATLAB® dataframe conventions:

parquetwrite

Serializes structs or OctaveTable instances into standard Parquet 2.0 binaries with optional Snappy or Zstandard compression and custom row group partitioning.

parquetread

Deserializes Parquet files with zero-copy column projection (SelectedVariableNames) and row group chunking (RowGroups).

parquetinfo

Instantly inspects the Compact Thrift metadata footer in under 250 μs without reading data pages, returning column schemas, row counts, and codecs.

% Write dense table with Snappy compression and 5,000-row row groups tbl = struct("id", (1:20000)', "val", randn(20000, 1)); parquetwrite("data.parquet", tbl, "Compression", "snappy", "RowGroupSize", 5000); % Inspect metadata instantaneously without reading data info = parquetinfo("data.parquet"); printf("Rows: %d, Columns: %d, FileSize: %d bytes\n", info.NumRows, info.NumColumns, info.FileSize); % Selectively project a single column from row group 2 subset = parquetread("data.parquet", "SelectedVariableNames", {"val"}, "RowGroups", 2);

2. Polars Interoperability & Schema Parity

ForgeCurves generates Parquet 2.0 binary files adhering strictly to the Apache Thrift Compact specification and the Apache Arrow format standard. Files exported from ForgeCurves load bit-for-bit in Python Polars (pl.read_parquet()), DuckDB, and PyArrow with zero schema warnings:

ForgeCurves Logical Kind Parquet Physical Type Parquet Logical Type Polars Data Type Arrow Type
Float64 (Numeric) DOUBLE None pl.Float64 float64
Int32 (Integer) INT32 INT(32, signed) pl.Int32 int32
Logical (Boolean) BOOLEAN None pl.Boolean bool
String (Cell Array) BYTE_ARRAY STRING (UTF8) pl.Utf8 utf8
Categorical BYTE_ARRAY ENUM / Dictionary pl.Categorical dictionary
# Direct bidirectional compatibility in Python Polars import polars as pl df = pl.read_parquet("forgecurves_export.parquet") print(df.schema) # Schema matches exact ForgeCurves types with zero conversion overhead

3. Empirical Benchmark Measurements (benchmark_parquet_library.m)

Performance benchmarks measured on the ForgeCurves browser runtime across all 6 standardized workloads from benchmark_parquet_library.m:

Workload Description Configuration Latency (ms) Throughput Compression Ratio Relative Speedup
Write (Uncompressed) 10k × 10 cols (0.76 MB) 117.95 ms 6.47 MB/s 100.1% 1.0× (Baseline)
Write (Snappy) 10k × 10 cols (0.76 MB) 18.83 ms 40.52 MB/s 100.1% 6.26× FASTER
Write (Zstandard) 10k × 10 cols (0.76 MB) 15.41 ms 49.51 MB/s 100.1% 7.65× FASTER
Read (Uncompressed) 10k × 10 cols (0.76 MB) 7.82 ms 97.51 MB/s — 1.0× (Baseline)
Read (Snappy) 10k × 10 cols (0.76 MB) 3.99 ms 191.38 MB/s — 1.96× FASTER
Read (Zstandard) 10k × 10 cols (0.76 MB) 2.90 ms 263.32 MB/s — 2.70× FASTER
Column Projection Wide schema (20 cols → 2 cols) 1.37 ms — — 3.51× SPEEDUP
Row Group Slicing Single row group (3,000 rows) 1.22 ms 2.47 M-rows/s — Selective Ingestion
Heterogeneous Mixed Types 10k rows (Int32, F64, Bool, Str) 5.00 ms 1,998,932 rows/s — 2.0 M-rows/sec
Metadata Inspection parquetinfo (50-col schema) 0.228 ms 4,392 calls/s — 228 μs per call

4. Tiered Parquet Persistence & Cloud Sync

Adhering to the Forgeboro Foundry Actor Model, ForgeCurves routes dataset persistence dynamically based on user authentication state:

Visitor (Anonymous)

Local IndexedDB storage (ForgeCurvesParquetDB) providing instant client-side offline dataset trials up to 50 MB.

Friend (Free Registered)

Authoritative cloud persistence (users/{userId}/parquet_stores) with account-scoped offline caching (curves_friend_parquet_cache_{userId}) up to 500 MB.

Member (Subscriber)

Extended 10 GB cloud storage with shared workspace permissions, concurrent revision history, and priority background synchronization.

Guest State Migration

When an anonymous Visitor registers or signs in, migrateGuestStoresToCloud() automatically promotes all local datasets to the user's cloud account seamlessly.