What happens when lessons from the 1 Billion Row Challenge meet Apache Parquet? 🤔
Java Champion Gunnar Mörling is building Hardwood: a Java-based parser, reader and future writer for Parquet files.
🔸 TLDR
▪️ Lightweight Java access to Parquet files
▪️ No full Hadoop dependency stack
▪️ Multi-core processing at Parquet page level
▪️ AI-assisted code with strong human review
🔸 WHAT IS HARDWOOD?
Hardwood is an open-source Java library for parsing, querying and eventually writing Apache Parquet files.
The name is a joke: parquet and hardwood are both types of flooring. 🪵
Its goal is to provide a smaller, faster and more modern alternative to the established Parquet Java library.
🔸 WHAT IS A PARQUET ENGINE?
Apache Parquet is a column-oriented file format used for analytics.
Instead of storing complete rows one after another, it stores values column by column. This improves compression and makes analytical queries more efficient.
A Parquet engine reads, filters, selects and writes this data.
Hardwood is not a SQL database. It is a lower-level engine that tools such as Flink or query engines could use to access Parquet files.
🔸 HOW IS IT RELATED TO JAVA?
▪️ Written in Java
▪️ Uses the Java 11 HTTP client for remote files such as S3
▪️ Uses primitive arrays to avoid boxing overhead
▪️ Explores JVM multithreading, JIT vectorization and future SIMD APIs
▪️ Applies lessons from Gunnar’s 1 Billion Row Challenge
🔸 KEY POINTS FROM GUNNAR
💬 “Let’s default to not.”
Hardwood keeps its core lightweight and adds optional modules only when needed.
💬 “It’s built with AI, not built by AI.”
Around 95% of the code was AI-generated, but Gunnar reviews it and controls the architecture.
💬 “Without supervision, it just wouldn’t get those things right.”
AI can accelerate implementation, but clean APIs, consistency and maintainability still require engineering judgment.
A key optimization is page-level parallelism: instead of assigning one thread per column, Hardwood distributes smaller Parquet pages across CPU cores. ⚡
🔸 TAKEAWAYS
▪️ Fewer dependencies reduce security and maintenance risks
▪️ Fine-grained parallelism uses modern CPUs better
▪️ AI increases coding speed, not engineering responsibility
▪️ Hardwood is promising, but its maturity and write support still matter
#Java #ApacheParquet #Hardwood #JVM #OpenSource #DataEngineering #Performance #AIEngineering #GunnarMorling #1BRC
Go further with Java certification:
Java👇
Spring👇
SpringBook👇
JavaBook👇