The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks)
Why Stack Overflow recommends it
Stack Overflow users link it when explaining how Parquet stores data internally, for example bit-packing, row-group structure and how Snappy compression applies to pages.
Recommended by 2 people across 2 different months, 2020 to 2024.
What it is
A conference talk by Boudewijn Braams of Databricks on the Parquet columnar format, covering row-wise, columnar and hybrid storage, Parquet's on-disk layout (row-groups, column-chunks, pages) and encodings. It then goes through performance tips such as dictionary encoding, page compression, predicate pushdown, partitioning and handling many small files, with a note on Delta Lake.
Cited in 1 Wikipedia article, including Apache Parquet.
What people said
Recommends Niko Fohr · 2022-05 · Stack Overflow · on “Difference between Apache parquet and arrow”
See, for example The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks) [this video] [10:15-10:45]
Recommends boyangeor · 2024-01 · Stack Overflow · on “BigInt vs boolean size PySpark”
One optimization technique in parquet [LINK] , among many others, is bit-packing. If you have BIGINT column that has small numbers you only need to store the N significant bits that cover the range, for [0 - 1] range you only need 1 bit. See this presentation [this video] .
Cites MikA · 2020-09 · Stack Overflow · on “Is Snappy splittable or not splittable?”
Snappy is actually not splittable as bzip, but when used with file formats like parquet or Avro, instead of compressing the entire file, blocks inside the file format are compressed using snappy. To understand what is happening when you're compressing a parquet file with snappy compression, check the structure of a parquet file [source link] [this video] [LINK] Inside a parquet file, records are split into row-groups [basically a subset of rows from the original file], and each row-groups are composed of data-pages [Column chunks in image], each column chunk is composed of many pages where actual records are stored in encoded format[columnar] with metadata. when you enable snappy compression it c