Standard columnar binary dataset storage, zero-copy selective projection, sub-millisecond footer metadata extraction, and bit-exact Polars interoperability in ForgeCurves.
ForgeCurves integrates native Apache Parquet operations directly into the computational notebook kernel with standard syntax matching Octave and MATLAB® dataframe conventions:
Serializes structs or OctaveTable instances into standard Parquet 2.0 binaries with optional Snappy or Zstandard compression and custom row group partitioning.
Deserializes Parquet files with zero-copy column projection (SelectedVariableNames) and row group chunking (RowGroups).
Instantly inspects the Compact Thrift metadata footer in under 250 μs without reading data pages, returning column schemas, row counts, and codecs.
ForgeCurves generates Parquet 2.0 binary files adhering strictly to the Apache Thrift Compact specification and the Apache Arrow format standard. Files exported from ForgeCurves load bit-for-bit in Python Polars (pl.read_parquet()), DuckDB, and PyArrow with zero schema warnings:
| ForgeCurves Logical Kind | Parquet Physical Type | Parquet Logical Type | Polars Data Type | Arrow Type |
|---|---|---|---|---|
| Float64 (Numeric) | DOUBLE |
None | pl.Float64 | float64 |
| Int32 (Integer) | INT32 |
INT(32, signed) |
pl.Int32 | int32 |
| Logical (Boolean) | BOOLEAN |
None | pl.Boolean | bool |
| String (Cell Array) | BYTE_ARRAY |
STRING (UTF8) |
pl.Utf8 | utf8 |
| Categorical | BYTE_ARRAY |
ENUM / Dictionary |
pl.Categorical | dictionary |
Performance benchmarks measured on the ForgeCurves browser runtime across all 6 standardized workloads from benchmark_parquet_library.m:
| Workload Description | Configuration | Latency (ms) | Throughput | Compression Ratio | Relative Speedup |
|---|---|---|---|---|---|
| Write (Uncompressed) | 10k × 10 cols (0.76 MB) | 117.95 ms | 6.47 MB/s | 100.1% | 1.0× (Baseline) |
| Write (Snappy) | 10k × 10 cols (0.76 MB) | 18.83 ms | 40.52 MB/s | 100.1% | 6.26× FASTER |
| Write (Zstandard) | 10k × 10 cols (0.76 MB) | 15.41 ms | 49.51 MB/s | 100.1% | 7.65× FASTER |
| Read (Uncompressed) | 10k × 10 cols (0.76 MB) | 7.82 ms | 97.51 MB/s | — | 1.0× (Baseline) |
| Read (Snappy) | 10k × 10 cols (0.76 MB) | 3.99 ms | 191.38 MB/s | — | 1.96× FASTER |
| Read (Zstandard) | 10k × 10 cols (0.76 MB) | 2.90 ms | 263.32 MB/s | — | 2.70× FASTER |
| Column Projection | Wide schema (20 cols → 2 cols) | 1.37 ms | — | — | 3.51× SPEEDUP |
| Row Group Slicing | Single row group (3,000 rows) | 1.22 ms | 2.47 M-rows/s | — | Selective Ingestion |
| Heterogeneous Mixed Types | 10k rows (Int32, F64, Bool, Str) | 5.00 ms | 1,998,932 rows/s | — | 2.0 M-rows/sec |
| Metadata Inspection | parquetinfo (50-col schema) | 0.228 ms | 4,392 calls/s | — | 228 μs per call |
Adhering to the Forgeboro Foundry Actor Model, ForgeCurves routes dataset persistence dynamically based on user authentication state:
Local IndexedDB storage (ForgeCurvesParquetDB) providing instant client-side offline dataset trials up to 50 MB.
Authoritative cloud persistence (users/{userId}/parquet_stores) with account-scoped offline caching (curves_friend_parquet_cache_{userId}) up to 500 MB.
Extended 10 GB cloud storage with shared workspace permissions, concurrent revision history, and priority background synchronization.
When an anonymous Visitor registers or signs in, migrateGuestStoresToCloud() automatically promotes all local datasets to the user's cloud account seamlessly.