Description
For more information about Stanford’s graduate programs, visit:
https://stanford.io/grad-ed-cme295-A2026
To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/
Chapters:
00:00:00 Introduction
00:09:12 Flash attention
00:50:48 Distributed training
00:54:25 Data parallelism (DP)
00:55:30 Zero Redundancy Optimizer (ZeRO)
01:00:29 Tensor parallelism (TP)
01:02:05 Pipeline parallelism (PP)
01:04:12 Context parallelism (CP)
01:06:10 Expert parallelism (EP)
01:09:52 Inference optimizations
01:22:49 KV caching
01:24:30 Paged attention
01:28:01 Grouped-query attention (GQA)
01:30:43 Multi-head latent attention (MLA)
01:32:40 Speculative decoding
01:45:27 Multi-token prediction (MTP)
Afshine Amidi is an Adjunct Professor at Stanford University.
Shervine Amidi is an Adjunct Professor at Stanford University.
Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X