Jay Cuthrell

Tip of the Apache Iceberg

A look at the past, present, and future of Apache Iceberg and related projects

Fudge Sunday readers will recall my use of songs as inspiration. While the newsletter is not going back to the series format, the lyrics of SchoolHouse Rock - Mother Necessity are appropriate.

https://www.youtube.com/watch?v=aBx-ilTzLec

The topic of ever more responsive, effective, and efficient data analytics is not new to Fudge Sunday newsletter. Today, there are rapidly updating cloud native projects and stories from the communities and companies involved in Hudi, Iceberg, Presto, Spark, Superset, Trino, and many more.

Shot

Just 132 days ago, Compute.AI was mentioned in Fudge Sunday #214 “Are You Gonna Go Parquet”.

A few readers were curious about how to “kick the tires” and get a demo of Compute.AI.

Chaser

Last week, the first public GitHub repository appeared for Compute.AI. Practitioners can now launch a containerized deployment of the ComputeAI SQL engine.

The Compute.AI team has posted updates on LinkedIn for the announcement itself as well as a video demo on the integration with Jupyter Notebook.

https://www.youtube.com/watch?v=Va-O-G0bNzg

Also, there is a video demo on the integration with Apache Superset.

https://www.youtube.com/watch?v=6XZMiQaHa24

As such, it’s time to add to the topics list. 🤓

Getting Informed

When reading about cloud native projects, it is common to see references to the Apache Software Foundation (ASF). Over almost a quarter of a century, ASF has grown to almost 300 projects.

One such project, Apache Iceberg, began in 2017 within Netflix and was donated to ASF in 2018 to promote the efficacy and longevity of a modern approach to low-level standards involving very large tables. This rapid promotion of the project within ASF is because the community is focused on ensuring Iceberg does certain things very well.

Then, in 2021, a commercial entity known as Tabular was formed and funded to simplify, secure, and streamline the adoption of Iceberg.

https://www.techmeme.com/230919/p43#a230919p43
https://www.youtube.com/watch?v=_GW3GYZK66U

Commercial options for Iceberg besides Tabular are growing. There are recognizable companies like Cloudera and Snowflake, but there are smaller companies like ClickHouse and Starburst that deserve attention as well — not to mention the Hudi play from companies like Onehouse.

https://www.techmeme.com/211028/p21#a211028p21
https://www.techmeme.com/220209/p15#a220209p15
https://www.techmeme.com/230203/p27#a230203p27

In 2024, expect to see more community activities including practitioner challenges that promote the art of the possible drawing upon various projects mentioned in this newsletter.

https://www.morling.dev/blog/one-billion-row-challenge/
https://ftisiot.net/posts/1brows/

So, what will be the next big thing in Apache Iceberg and related projects?

Until then… Place your bets!

p.s. As I’ve gotten older, I have come to appreciate getting snail mail. If you have time to drop me a postcard that would be amazing.

View this page on GitHub.