Delta Lake Tutorial L7 — MERGE INTO Explained: Upserts, CDC and SCD Patterns (PySpark 2026)

In this video, we cover MERGE INTO — the most powerful and most misunderstood Delta
Lake operation. You will learn how to implement upserts, build Change Data Capture
(CDC) pipelines, and handle Slowly Changing Dimensions (SCD Type 2) using MERGE INTO
in PySpark and Databricks.

This is video L7 in the Delta Lake Complete Guide 2026 series.

What we cover in this video:
▸ MERGE INTO syntax and how the matching condition works
▸ Implementing upserts — insert if new, update if existing
▸ Building a CDC pipeline using MERGE INTO
▸ SCD Type 2 — tracking historical changes with MERGE INTO
▸ Conditional updates and inserts within a single MERGE statement
▸ Common MERGE INTO performance pitfalls and how to avoid them
▸ When NOT to use MERGE INTO

By the end of this video, you will be able to implement production-grade upsert and
CDC patterns confidently using Delta Lake.

━━━━━━━━━━━━━━━━━━━━━━━━━
🔗 LINKS
━━━━━━━━━━━━━━━━━━━━━━━━━
▸ Delta Lake Complete Guide 2026 playlist: https://www.youtube.com/playlist?list=PLkz1SCf5iB4cA6iYeXXvqporVpq2t8jvs
▸ L1 — What is Delta Lake and why it matters on AWS: https://youtu.be/_29tguUvnps
▸ L2 — Creating and managing Delta tables: https://youtu.be/vRa1YFdfl9I
▸ L3 — Batch and streaming reads and writes: https://youtu.be/NgW--dV6X-g
▸ L4 — Time Travel: https://youtu.be/1-TyobuLKCY
▸ L5 — VACUUM, OPTIMIZE and Z-Ordering: [video link]
▸ L6 — RESTORE and rollback: https://youtu.be/8e2a1MYgdqU
▸ Spark Performance Tuning course: www.scholarnest.com
▸ Generative AI for Data Engineers course: www.scholarnest.com

━━━━━━━━━━━━━━━━━━━━━━━━━
📌 ABOUT THIS SERIES
━━━━━━━━━━━━━━━━━━━━━━━━━
The Delta Lake Complete Guide 2026 is a free, structured 11-part series for data
engineers who want to go beyond the basics. Every video is recorded as part of a full
Databricks on AWS course — real engineering depth, not simplified demos.

New videos every week. Subscribe and hit the bell so you don't miss any part of the series.

━━━━━━━━━━━━━━━━━━━━━━━━━
👤 ABOUT THE INSTRUCTOR
━━━━━━━━━━━━━━━━━━━━━━━━━
Data engineer and architect with 25 years of industry experience. I create self-paced
courses on Apache Spark, Databricks, Kafka, and Generative AI for data engineers.

📺 YouTube: Learning Journal (you're already here!)
🌐 Website: www.scholarnest.com

━━━━━━━━━━━━━━━━━━━━━━━━━
#DeltaLake #Databricks #ApacheSpark #PySpark #DataEngineering #CDC #LearningJournal