Weekly Schedule & Timeline | M.Sc. Advanced NLP (Semester 4051)

Week 1 S1 Lecture

History of NLP, Modern LLM Foundations & Paradigm Shifts

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 1 – Introduction & Fundamentals (01/13)Stanford Week 1, Tue – History of NLP

📌 Module 1: Introduction to Large Language Models

💡 Course overview, logistics, and foundational language modeling paradigms.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Language Models are Few-Shot Learners (GPT-3) — Brown et al. (NeurIPS 2020)
📄 Sparks of Artificial General Intelligence: Early experiments with GPT-4 — Bubeck et al. (Microsoft Research 2023)

Week 1 S2 Lecture

Word Representations & Learned Vector Semantics (Word2Vec, GloVe)

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 2 – Fundamentals: Learned Representations (01/15)Stanford Week 1, Thu – Word Vectors

📌 Foundations: Information Representation & Distributed Semantics

💡 Distributional hypothesis, Skip-gram with negative sampling, and word similarity.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Efficient Estimation of Word Representations in Vector Space (Word2Vec) — Mikolov et al. (ICLR 2013)
📄 GloVe: Global Vectors for Word Representation — Pennington, Socher, Manning (EMNLP 2014)

Week 2 S3 Lecture

Neural Network Fundamentals & Backpropagation for NLP

Alireza Nazari

Stanford Week 2, Tue – Backpropagation and Neural Network Basics

📌 Foundations: Neural Computation & Optimization

💡 Computational graphs, gradient descent variants, and PyTorch autograd engine.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Deep Learning (Review) — LeCun, Bengio, Hinton (Nature 2015)
📄 Automatic Differentiation in PyTorch — Paszke et al. (NeurIPS 2017 Autodiff Workshop)

Week 2 S4 Lecture

Autoregressive Language Modeling & Cross-Entropy Objectives

Alireza Nazari

CMU Lecture 3 – Fundamentals: Autoregressive Language Modeling (01/20)Stanford Week 2, Thu – Language Models and RNNs (Part 1)

📌 Module 1: Language Modeling Objectives & Perplexity

💡 N-gram baselines vs neural probabilistic LMs, cross-entropy loss, and evaluation metrics.

Slides PDF

Suggested Readings & Landmark Papers:

📄 A Neural Probabilistic Language Model — Bengio, Ducharme, Vincent, Jauvin (JMLR 2003)
📄 Generating Sequences With Recurrent Neural Networks — Alex Graves (2013)

Week 3 S5 Lecture

Recurrent Architectures: RNNs, LSTMs & Gated Recurrent Units

Alireza Nazari

CMU Lecture 4 – Architectures I: Recurrent Neural Networks (01/22)Stanford Week 2, Thu – Language Models and RNNs (Part 2)

📌 Module 3: Sequence Encoders and Recurrent Structures

💡 Vanishing/exploding gradients, gating mechanisms, and sequence-to-sequence formulations.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Long Short-Term Memory — Hochreiter & Schmidhuber (Neural Computation 1997)
📄 Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Chung, Gulcehre, Cho, Bengio (NeurIPS 2014)

Week 3 S6 Lecture

Attention Mechanisms & The Transformer Architecture

Alireza Nazari

CMU Lecture 5 – Architectures II: Attention and Transformers (01/27)Stanford Week 3, Tue – Transformers

📌 Module 3: Transformers (Encoder, Decoder, Positional Encodings)

💡 Scaled dot-product attention, multi-head attention, RoPE, and causal masking.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Attention Is All You Need — Vaswani et al. (NeurIPS 2017)
📄 RoFormer: Enhanced Transformer with Rotary Position Embedding — Su et al. (Neurocomputing 2024)

Week 4 S7 Lecture

Subword Tokenization & Multilingual Language Modeling

Prof. Behrouz Minaei-Bidgoli

Stanford Week 7, Thu – Tokenization and Multilinguality (Guest: Julie Kallini)

📌 Module 3: Subword Tokenization (BPE, WordPiece, Unigram)

💡 Byte Pair Encoding, tokenizer fertility, out-of-vocabulary handling, and multilingual transfer.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Neural Machine Translation of Rare Words with Subword Units (BPE) — Sennrich, Haddow, Birch (ACL 2016)
📄 Cross-lingual Language Model Pretraining (XLM) — Lample & Conneau (NeurIPS 2019)

Week 4 S8 Lecture

LLM Pretraining: Architectures, Data Pipelines & Systems

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 6 – Learning I: Pretraining (01/29)Stanford Week 4, Tue – Pretraining (Scaling, Systems, Data)

📌 Module 2: Pretraining Strategies & Data Scaling

💡 Web scraping pipelines, deduplication, quality filtering, and pretraining stability.

Slides PDF

Suggested Readings & Landmark Papers:

📄 The Llama 3 Herd of Models — Dubey et al. (Meta AI 2024)
📄 The RefinedWeb Dataset for Falcon LLM — Penedo et al. (NeurIPS 2023)

Week 5 Q1 Quiz

Quiz 1 (Foundations & Architecture Review)

Teaching Staff

📌 Assessment covering Sessions S1 through S8

💡 Short theoretical quiz followed by interactive Q&A and assignment review.

Week 5 S9 Lecture

Scaling Laws, In-Context Learning & Efficient Adaptation (PEFT / Prompting)

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 7 – Scaling Laws and In-Context Learning (02/03)Stanford Week 5, Tue – Efficient Adaptation (Prompting + PEFT)

📌 Module 7 & 2: Prompting, ICL & Parameter-Efficient Transfer

💡 Chinchilla compute-optimal scaling, emergent abilities, prompt engineering, and PEFT.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Training Compute-Optimal Large Language Models (Chinchilla) — Hoffmann et al. (NeurIPS 2022)
📄 Scaling Laws for Neural Language Models — Kaplan et al. (OpenAI 2020)

Week 6 S10 Lecture

Parameter Fine-Tuning, Knowledge Distillation & LoRA Practical Workshop

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 8 – Learning III: Fine-tuning and Distillation (02/05)Stanford Week 9, Thu – Tinker and LoRA Without Regret (Guest: John Schulman)

📌 Module 2 & 7: Fine-Tuning, Distillation & Low-Rank Adaptation

💡 Full fine-tuning vs LoRA/QLoRA rank decomposition, teacher-student distillation.

Slides PDF

Suggested Readings & Landmark Papers:

📄 LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. (ICLR 2022)
📄 QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers, Pagnoni, Holtzman, Zettlemoyer (NeurIPS 2023)

Week 6 S11 Lecture

Reinforcement Learning Fundamentals in NLP

Mohaddeseh Emadi

CMU Lecture 16 – Reinforcement Learning I: Fundamentals (03/12)

📌 Module 6: Reinforcement Learning in NLP – Theoretical Basics

💡 Markov decision processes, policy gradients (PPO), value functions, and reward modeling.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Deep Reinforcement Learning from Human Preferences — Christiano et al. (NeurIPS 2017)
📄 Proximal Policy Optimization Algorithms — Schulman, Wolski, Dhariwal, Radford, Klimov (2017)

Week 7 S12 Lecture

Post-Training with Human Preferences & Feedback (RLHF & DPO)

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 17 – Reinforcement Learning II: Applications (03/17)Stanford Week 4, Thu – Post-training (RLHF, SFT, DPO)

📌 Module 6: Model Training with Human Feedback (RLHF, DPO, KTO)

💡 Bradley-Terry preference model, InstructGPT pipeline, Direct Preference Optimization, and refusal tuning.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Training language models to follow instructions with human feedback (InstructGPT) — Ouyang et al. (NeurIPS 2022)
📄 Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafailov, Sharma, Mitchell, Ermon, Manning, Finn (NeurIPS 2023)

Week 7 S13 Lecture

Model Quantization & Parallel / Distributed Training Systems

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 19 – Quantization (03/26) + Lecture 20 – Parallelism & Distributed (03/31)

📌 Module 7: Compute-Optimal Training, Parallelism & Quantization

💡 FP8/INT4 quantization (AWQ, GPTQ, bitsandbytes), tensor parallelism (Megatron), ZeRO-3.

Slides PDF

Suggested Readings & Landmark Papers:

📄 AWQ: Activation-aware Weight Quantization for LLM Compression — Lin et al. (MLSys 2024)
📄 ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Rajbhandari et al. (SC 2020)

Week 8 Q2 Quiz

Quiz 2 (Adaptation, Alignment & Systems Review)

Teaching Staff

📌 Assessment covering Sessions S9 through S13

💡 In-class evaluation of alignment, PEFT, and distributed systems concepts.

Week 8 S14 Lecture

Mixture of Experts (MoE) & Long-Sequence Length Scaling (Mamba, FlashAttention)

Mohaddeseh Emadi

CMU Lecture 21 – Mixture of Experts (04/02) + Lecture 22 – Scaling Sequence Length (04/07)

📌 Module 7: Large-Scale LLM Architectures & Sparse Routing

💡 Sparse gating, expert load balancing, FlashAttention IO-awareness, and state-space models (Mamba).

Slides PDF

Suggested Readings & Landmark Papers:

📄 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Shazeer et al. (ICLR 2017)
📄 FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao (ICLR 2024)
📄 Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Gu & Dao (2023)

Week 9 S15 Lecture

Decoding Algorithms, Chain-of-Thought (CoT) & Test-Time Scaling

Mohammad Khaki

CMU Lecture 9 – Decoding Algorithms (02/10) + Lecture 23 – Test-Time Scaling (04/14)Stanford Week 6, Thu – Reasoning 1 + Week 7, Tue – Reasoning 2

📌 Module 9: Language Inference, Reasoning & Search Scaling

💡 Top-k, nucleus (top-p), beam search, chain-of-thought prompting, process supervision, and test-time search.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al. (NeurIPS 2022)
📄 Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Parameters — Snell et al. (Google DeepMind 2024)

Week 9 S16 Lecture

Retrieval-Augmented Generation (RAG) & Dense Passage Retrieval

Mohaddeseh Emadi

CMU Lecture 10 – Modeling I: Retrieval and RAG (Guest: Akari Asai) (02/12)Stanford Week 5, Thu – Agents, Tool Use, and RAG (Part 1)

📌 Module 8: Retrieval-Based Modeling & Knowledge Integration

💡 Dense retrieval (DPR, ColBERT), vector indexing (HNSW), chunking, re-ranking, and Self-RAG.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. (NeurIPS 2020)
📄 Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Asai et al. (ICLR 2024)

Week 10 S17 Lecture

Autonomous Agents, Function Calling & Tool Use

Mohammad Khaki

CMU Lecture 18 – Language Model-Based Agents (03/19)Stanford Week 5, Thu – Agents, Tool Use, and RAG (Part 2)

📌 Module 9: Chatbots, Tool-Augmented LLMs & Understanding

💡 ReAct prompting, function calling schema, environment feedback loops, and LangGraph agents.

Slides PDF

Suggested Readings & Landmark Papers:

📄 ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al. (ICLR 2023)
📄 Toolformer: Language Models Can Teach Themselves to Use Tools — Schick et al. (NeurIPS 2023)

Week 10 MID Midterm Exam

Midterm Examination

Course Instructor & Staff

📌 Comprehensive assessment covering Sessions S1 through S17

💡 Formal mid-term examination covering theoretical concepts and practical algorithms.

Week 11 S18 Lecture

Evaluation Methodologies, Benchmarks & Leaderboards in LLMs

Mohaddeseh Emadi

CMU Lecture 13 – Evaluation Techniques (02/24)Stanford Week 6, Tue – Benchmarking and Evaluation

📌 Module 5: Large Language Model Evaluation & Benchmarks

💡 MMLU, GSM8K, HumanEval, LLM-as-a-Judge, contamination detection, and Chatbot Arena Elo ratings.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Measuring Massive Multitask Language Understanding (MMLU) — Hendrycks et al. (ICLR 2021)
📄 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. (NeurIPS 2023)

Week 11 S19 Lecture

Interpretability, Social Impacts, Bias, Safety & Privacy in NLP

Prof. Behrouz Minaei-Bidgoli

Stanford Week 8, Tue – Interpretability (Guest: Been Kim) + Thu – Social Impacts (Risks)

📌 Module 4 & 5: Security, Privacy, Bias, Hallucination & Robustness

💡 Mechanistic interpretability, sparse autoencoders, prompt injection, red teaming, and watermarking.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Bricken et al. (Anthropic 2023)
📄 Universal and Transferable Adversarial Attacks on Aligned Language Models — Zou, Phan, Chen, Kantchelian, Bartolo, Hendrycks (2023)

Week 12 Q3 Quiz

Quiz 3 (Advanced Inference, RAG & Safety Review)

Teaching Staff

📌 Assessment covering Sessions S14 through S19

💡 Evaluation covering MoE, RAG, agents, benchmarking, and interpretability.

Week 12 S20 Lecture

Multimodal Models: Vision-Language Representations & Generation (VLM)

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 11 – Modeling II: Multimodal I (02/17) + Lecture 12 – Multimodal II (02/19)Stanford Week 9, Tue – Multimodality (Guest: Luke Zettlemoyer)

📌 Module 7 & 10: Multimodal Training & Cross-Modal Generation

💡 CLIP contrastive pretraining, visual token projections (LLaVA), cross-modal attention, and generation.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Radford et al. (OpenAI, ICML 2021)
📄 Visual Instruction Tuning (LLaVA) — Liu, Li, Wu, Lee (NeurIPS 2023)

Week 13 S21 Lecture

Diffusion Models & Flow Matching for Natural Language Generation

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 15 – Modeling IV: Diffusion and Flows (03/10)

📌 Module 10: Natural Language Generation with Continuous & Diffusion Models

💡 Continuous-time diffusion, discrete diffusion (Plaid, Bit-Diffusion), flow matching, and non-autoregressive generation.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Diffusion-LM Improves Controllable Text Generation — Li et al. (NeurIPS 2022)
📄 Flow Matching for Generative Modeling — Lipman, Chen, Ben-Hamu, Nicklas, Le (ICLR 2023)

Week 13 S22 Lecture

Graph Neural Networks for Natural Language Processing (GNN for NLP)

Prof. Behrouz Minaei-Bidgoli

📌 Module 11: Graph Neural Networks for Language Processing

💡 Knowledge graph embeddings, GCN/GAT for text classification, dependency graph parsing, and KG-LLM fusion.

Slides PDF

Suggested Readings & Landmark Papers:

📄 Graph Neural Networks for Natural Language Processing: A Survey — Wu, Wu, Cao, Xie, Yuan, Huang (Foundations and Trends in ML 2021)
📄 Graph Convolutional Networks for Text Classification — Yao, Mao, Luo (AAAI 2019)

Week 14 S23 Lecture

Research Skills, Experimental Design & Future Frontiers in NLP

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 14 – Research Skills and Experimental Design (02/26)Stanford Week 10, Tue – Open Questions in NLP 2026

📌 Synthesis & Academic Research Preparation

💡 Formulating hypotheses, ablation engineering, writing ACL papers, and emergent research frontiers.

Slides PDF

Suggested Readings & Landmark Papers:

📄 A Survey on Large Language Models — Zhao et al. (AI Open 2023)
📄 On the Opportunities and Risks of Foundation Models — Bommasani et al. (Stanford CRFM 2021)

Week 14 REV Review

Comprehensive Course Review & Project Synthesis

Prof. Behrouz Minaei-Bidgoli

📌 Full Curriculum Modules 1 through 11 Review

💡 Comprehensive synthesis of the entire semester in preparation for project presentations.

Week 15 POST1 Presentation

Project Presentations – Session 1

Student Research Teams & Staff

📌 Practical Application of Curriculum Modules

💡 Student team presentations and system demonstrations – Session 1.

Week 15 POST2 Presentation

Project Presentations – Session 2

Student Research Teams & Staff

📌 Practical Application of Curriculum Modules

💡 Student team presentations and system demonstrations – Session 2.

Course Schedule & Timeline