Description
In this video, we cover the maintenance side of Delta Lake that most tutorials skip —
VACUUM, OPTIMIZE, and Z-Ordering. You will learn how small files accumulate in Delta
tables, how OPTIMIZE and Z-Ordering fix this, how VACUUM manages storage and data
retention, and the critical mistakes that can break Time Travel or corrupt concurrent
operations.
This is video #5 in the Delta Lake Complete Guide 2026 series.
What we cover in this video:
▸ Why small files accumulate in Delta tables and why they hurt performance
▸ Using OPTIMIZE to compact small files into larger ones
▸ Z-Ordering — co-locating data for faster filtered reads
▸ How VACUUM removes unreferenced data files
▸ The default 7-day retention period and why it matters
▸ The mistake that breaks Time Travel — lowering retention incorrectly
▸ A safe maintenance schedule for production Delta tables
By the end of this video, you will be able to keep your Delta tables performant and
correctly sized in production — without breaking Time Travel or risking data loss.
━━━━━━━━━━━━━━━━━━━━━━━━━
🔗 LINKS
━━━━━━━━━━━━━━━━━━━━━━━━━
▸ Delta Lake Complete Guide 2026 playlist: [playlist link]
▸ Video #1 — What is Delta Lake: https://youtu.be/vRa1YFdfl9I
▸ Video #2 — Creating and managing Delta tables: https://youtu.be/_29tguUvnps
▸ Video #3 — Batch and streaming reads and writes: https://youtu.be/NgW--dV6X-g
▸ Video #4 — Time Travel: https://youtu.be/1-TyobuLKCY
▸ Full Databricks on AWS course (coming July 2026): Coming Soon
▸ Spark Performance Tuning course: www.scholarnest.com
▸ Generative AI for Data Engineers course: www.scholarnest.com
━━━━━━━━━━━━━━━━━━━━━━━━━
⏱ CHAPTERS
━━━━━━━━━━━━━━━━━━━━━━━━━
00:00 Introduction
[Add your chapter timestamps here before publishing]
━━━━━━━━━━━━━━━━━━━━━━━━━
📌 ABOUT THIS SERIES
━━━━━━━━━━━━━━━━━━━━━━━━━
The Delta Lake Complete Guide 2026 is a free, structured series for data engineers
who want to go beyond the basics. Every video is recorded as part of a full Databricks
on AWS course — real engineering depth, not simplified demos.
New videos every week. Subscribe and hit the bell so you don't miss any part of the series.
━━━━━━━━━━━━━━━━━━━━━━━━━
👤 ABOUT THE INSTRUCTOR
━━━━━━━━━━━━━━━━━━━━━━━━━
Data engineer and architect with 25 years of industry experience. I create self-paced
courses on Apache Spark, Databricks, Kafka, and Generative AI for data engineers.
📺 YouTube: Learning Journal (you're already here!)
🌐 Website: www.scholarnest.com
━━━━━━━━━━━━━━━━━━━━━━━━━
#DeltaLake #Databricks #ApacheSpark #PySpark #DataEngineering #LearningJournal