Weekly Schedule & Timeline | M.Sc. Advanced NLP (Semester 4051)

Week 1 S1 Lecture

Saturday 1405-06-28

History of NLP, Modern LLM Foundations & Paradigm Shifts

Prof. Behrouz Minaei-Bidgoli

CMU Lecture 1 – Introduction & Fundamentals (01/13)Stanford Week 1, Tue – History of NLP

📌 Module 1: Introduction to Large Language Models

Slides (PDF)

Suggested Readings & Landmark Papers:

📄 Language Models are Few-Shot Learners (GPT-3) — Tom B. Brown et al. (NeurIPS 2020)
📄 Sparks of Artificial General Intelligence: Early experiments with GPT-4 — Sébastien Bubeck et al. (Microsoft Research 2023)
📄 Natural Language Understanding (Eisenstein Book, Chapter 1-2) — Jacob Eisenstein (MIT Press)
📄 On the Opportunities and Risks of Foundation Models — Rishi Bommasani et al. (Stanford CRFM 2021)

Week 1 S2 Lecture

Monday 1405-06-30

Word Representations, Distributed Semantics & Vector Embeddings (Word2Vec, GloVe)

Fateme Marefat

CMU Lecture 2 – Fundamentals: Learned Representations (01/15)Stanford Week 1, Thu – Word Vectors

📌 Module 1: Information Representation & Vector Embeddings

Slides (PDF)

Suggested Readings & Landmark Papers:

📄 Efficient Estimation of Word Representations in Vector Space (Word2Vec) — Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean (ICLR 2013)
📄 GloVe: Global Vectors for Word Representation — Jeffrey Pennington, Richard Socher, Christopher D. Manning (EMNLP 2014)
📄 Distributed Representations of Words and Phrases and their Compositionality — Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean (NeurIPS 2013)
📄 Enriching Word Vectors with Subword Information (fastText) — Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov (TACL 2017)

Week 2 S3 Lecture

Saturday 1405-07-04

Neural Network Fundamentals & Backpropagation in NLP

Alireza Nazari

Stanford Week 2, Tue – Backpropagation and Neural Network Basics

📌 Foundations: Neural Computation & Optimization

Slides (PDF)

Suggested Readings & Landmark Papers:

📄 Deep Learning (Review Paper) — Yann LeCun, Yoshua Bengio, Geoffrey Hinton (Nature 2015)
📄 Yes you should understand backprop — Andrej Karpathy (2016)
📄 Automatic Differentiation in PyTorch — Adam Paszke et al. (NeurIPS Autodiff Workshop 2017)
🎬 The spelled-out intro to neural networks and backpropagation: building micrograd — Andrej Karpathy (YouTube Lecture)
🎬 Neural Networks Pt. 2: Backpropagation Main Ideas — StatQuest with Josh Starmer
🎬 Backpropagation, intuitively | Deep Learning Chapter 3 — 3Blue1Brown (Grant Sanderson)

Week 2 S4 Lecture

Monday 1405-07-06

Autoregressive Language Modeling & Cross-Entropy Objectives

Alireza Nazari

CMU Lecture 3 – Fundamentals: Autoregressive Language Modeling (01/20)Stanford Week 2, Thu – Language Models and RNNs (Part 1)

📌 Module 1: Statistical & Probabilistic Language Modeling

Slides (PDF)

Suggested Readings & Landmark Papers:

📄 A Neural Probabilistic Language Model — Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin (JMLR 2003)
📄 Generating Sequences With Recurrent Neural Networks — Alex Graves (2013)
📄 Speech and Language Processing (3rd ed. draft, Chapter 3: N-gram Language Models) — Dan Jurafsky, James H. Martin

Week 3 S5 Lecture

Saturday 1405-07-11

Recurrent Sequence Architectures: RNNs, LSTMs & Gated Recurrent Units (GRUs)

Alireza Nazari

CMU Lecture 4 – Architectures I: Recurrent Neural Networks (01/22)Stanford Week 2, Thu – Language Models and RNNs (Part 2)

📌 Module 3: Sequential Encoders & Recurrent Architectures

Slides (PDF)

Suggested Readings & Landmark Papers:

📄 Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber (Neural Computation 1997)
📄 Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio (NeurIPS 2014)
📄 Understanding LSTM Networks — Christopher Olah (2015)
📄 The Unreasonable Effectiveness of Recurrent Neural Networks — Andrej Karpathy (2015)
🎬 Recurrent Neural Networks (RNNs), Clearly Explained!!! — StatQuest with Josh Starmer
🎬 Long Short-Term Memory (LSTM), Clearly Explained — StatQuest with Josh Starmer

Week 3 S6 Lecture

Monday 1405-07-13

Attention Mechanisms & Transformer Architecture (RoPE, Multi-Head Attention)

Setare Kahnamuee

CMU Lecture 5 – Architectures II: Attention and Transformers (01/27)Stanford Week 3, Tue – Transformers

📌 Module 3: Transformer Architecture & Multi-Head Attention

Slides (PDF) Extra Slides (PDF)

Suggested Readings & Landmark Papers:

📄 Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit et al. (NeurIPS 2017)
📄 The Illustrated Transformer — Jay Alammar (2018)
📄 RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE) — Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu (Neurocomputing 2024)
📄 The Annotated Transformer — Austin Rush et al. (Harvard NLP)

Week 4 Q1 Quiz

Saturday 1405-07-18

Quiz 1 (Covers Sessions S1–S6)

Teaching Staff

💡 Covers Sessions S1–S6 (Word Vectors, Neural Basics, LM, RNNs, LSTMs, Transformers)

Quiz Paper (PDF)

Week 4 S7 Lecture

Monday 1405-07-20

Subword Tokenization (BPE, SentencePiece) & Multilingual Language Models

Fateme Marefat

Stanford Week 7, Thu – Tokenization and Multilinguality (Guest: Julie Kallini)

📌 Module 3: Subword Tokenization (BPE, SentencePiece)

Slides (PDF)

Suggested Readings & Landmark Papers:

📄 Neural Machine Translation of Rare Words with Subword Units (BPE) — Rico Sennrich, Barry Haddow, Alexandra Birch (ACL 2016)
📄 Subword Regularization: Improving Neural Network Translation Models (SentencePiece / Unigram) — Taku Kudo (ACL 2018)
📄 Cross-lingual Language Model Pretraining (XLM) — Guillaume Lample, Alexis Conneau (NeurIPS 2019)

Week 5 S8 Lecture

Saturday 1405-07-25

LLM Pretraining: Web Data Pipelines, Quality Filtering & Scaling

Setare Kahnamuee

CMU Lecture 6 – Learning I: Pretraining (01/29)Stanford Week 4, Tue – Pretraining (Scaling, Systems, Data)

📌 Module 2: Pretraining Strategies & Data Curation Pipelines

Suggested Readings & Landmark Papers:

📄 Language Models are Unsupervised Multitask Learners (GPT-2) — Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever (OpenAI 2019)
📄 The Llama 3 Herd of Models — Abhimanyu Dubey et al. (Meta AI 2024)
📄 The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora — Guilherme Penedo et al. (NeurIPS 2023)

Week 5 S9 Lecture

Monday 1405-07-27

Scaling Laws (Chinchilla) & In-Context Learning (ICL)

Setare Kahnamuee

CMU Lecture 7 – Scaling Laws and In-Context Learning (02/03)Stanford Week 4, Tue – Pretraining (Scaling)

📌 Module 7: Prompting & ICL / Module 2: Transfer Learning

Suggested Readings & Landmark Papers:

📄 Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann et al. (DeepMind, NeurIPS 2022)
📄 Scaling Laws for Neural Language Models — Jared Kaplan et al. (OpenAI 2020)

Week 6 S10 Lecture

Saturday 1405-08-02

Parameter-Efficient Fine-Tuning (PEFT: LoRA, QLoRA) & Knowledge Distillation

Saba Sattari

CMU Lecture 8 – Learning III: Fine-tuning and Distillation (02/05)Stanford Week 5, Tue – Efficient Adaptation + Week 9, Thu – Tinker and LoRA (John Schulman)

📌 Module 2: Fine-Tuning & Distillation / Module 7: Adaptation

Suggested Readings & Landmark Papers:

📄 LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu et al. (ICLR 2022)
📄 QLoRA: Efficient Finetuning of Quantized LLMs — Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (NeurIPS 2023)
📄 Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean (NeurIPS Deep Learning Workshop 2014)
📄 Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang (ACL 2021)

Week 6 S11 Lecture

Monday 1405-08-04

Decoding Algorithms, Sampling Strategies & Chain-of-Thought (CoT) Prompting

Mohammad Khaki

CMU Lecture 9 – Decoding Algorithms (02/10)Stanford Week 6, Thu – Reasoning 1

📌 Module 9: Inference & Language Reasoning Systems

Suggested Readings & Landmark Papers:

📄 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou (NeurIPS 2022)
📄 The Curious Case of Neural Text Degeneration (Nucleus / Top-p Sampling) — Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi (ICLR 2020)

Week 7 S12 Lecture

Saturday 1405-08-09

Advanced Reasoning: System 2 Reasoning & Search

Mohammad Khaki

Stanford Week 7, Tue – Reasoning 2

📌 Module 9: Inference & Language Reasoning Systems

Suggested Readings & Landmark Papers:

📄 Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao et al. (Princeton & DeepMind, NeurIPS 2023)
📄 STaR: Bootstrapping Reasoning With Reasoning — Eric Zelikman et al. (Stanford, NeurIPS 2022)
📄 Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown et al. (2024)

Week 7 Q2 Quiz

Monday 1405-08-11

Quiz 2 (Covers Sessions S7–S12)

Teaching Staff

Week 8 S13 Lecture

Saturday 1405-08-16

Retrieval-Augmented Generation (RAG) & Dense Passage Retrieval (DPR, Self-RAG)

Mohadeseh Emadi

CMU Lecture 10 – Modeling I: Retrieval and RAG (Guest: Akari Asai) (02/12)Stanford Week 5, Thu – Agents, Tool Use, and RAG (Part 1)

📌 Module 8: Retrieval-Augmented Modeling & Knowledge Integration

Suggested Readings & Landmark Papers:

📄 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG) — Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al. (NeurIPS 2020)
📄 Dense Passage Retrieval for Open-Domain Question Answering (DPR) — Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih (EMNLP 2020)
📄 Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi (ICLR 2024)

Week 8 S14 Lecture

Monday 1405-08-18

Vision-Language Models (CLIP, LLaVA) & Multimodal Alignment

Poorya Saneei

CMU Lecture 11 – Modeling II: Multimodal I (02/17) + Lecture 12 – Modeling III: Multimodal II (02/19)Stanford Week 9, Tue – Multimodality (Guest: Luke Zettlemoyer)

📌 Module 7: Multimodal Training / Module 10: Multimodal Generation

Suggested Readings & Landmark Papers:

📄 Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal et al. (OpenAI, ICML 2021)
📄 Visual Instruction Tuning (LLaVA) — Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee (NeurIPS 2023)
📄 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT) — Alexey Dosovitskiy et al. (ICLR 2021)

Week 9 S15 Lecture

Saturday 1405-08-23

LLM Evaluation Methodologies, Benchmarks & Leaderboards (MMLU, LLM-as-a-Judge)

Mohadeseh Emadi

CMU Lecture 13 – Evaluation Techniques (02/24)Stanford Week 6, Tue – Benchmarking and Evaluation

📌 Module 5: Evaluation Methodologies for Large Language Models

Suggested Readings & Landmark Papers:

📄 Measuring Massive Multitask Language Understanding (MMLU) — Dan Hendrycks et al. (ICLR 2021)
📄 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng, Wei-Lin Chiang, Hao Zhang, Siyuan Zhuang, Yonghao Zhuang, Lingxiao Zheng et al. (NeurIPS 2023)
📄 Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Noah A. Smith et al. (2024)

Week 9 S16 Lecture

Monday 1405-08-25

Graph Neural Networks for Natural Language Processing (GNN for NLP)

Poorya Saneei, Setare Kahnamuee

📌 Module 11: Graph Neural Networks for NLP

Suggested Readings & Landmark Papers:

📄 Graph Neural Networks for Natural Language Processing: A Survey — Lingfei Wu, Yu Chen, Kai Shen, Xiaojie Guo et al. (Foundations and Trends in ML 2021)
📄 Graph Convolutional Networks for Text Classification — Liang Yao, Chengsheng Mao, Yuan Luo (AAAI 2019)
📄 Semi-Supervised Classification with Graph Convolutional Networks (GCN) — Thomas N. Kipf, Max Welling (ICLR 2017)

Week 10 S17 Lecture

Saturday 1405-08-30

Diffusion Models & Flow Matching for Natural Language Generation

Poorya Saneei

CMU Lecture 15 – Modeling IV: Diffusion and Flows (03/10)

📌 Module 10: Natural Language Generation Tasks with LLMs

Suggested Readings & Landmark Papers:

📄 Diffusion-LM Improves Controllable Text Generation — Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto (NeurIPS 2022)
📄 Flow Matching for Generative Modeling — Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nicklas, Matt Le (ICLR 2023)
📄 Denoising Diffusion Probabilistic Models (DDPM) — Jonathan Ho, Ajay Jain, Pieter Abbeel (NeurIPS 2020)

Week 10 S18 Lecture

Monday 1405-09-02

Reinforcement Learning Fundamentals in Natural Language Processing

Saba Sattari

CMU Lecture 16 – Reinforcement Learning I: Fundamentals (03/12)

📌 Module 6: Reinforcement Learning in NLP – Foundations

Suggested Readings & Landmark Papers:

📄 Deep Reinforcement Learning from Human Preferences — Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei (NeurIPS 2017)
📄 Proximal Policy Optimization Algorithms (PPO) — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (OpenAI 2017)

Week 11 Q3 Quiz

Saturday 1405-09-07

Quiz 3 (Covers Sessions S13–S18)

Teaching Staff

Week 11 S19 Lecture

Monday 1405-09-09

Post-Training with Human Feedback & Preference Alignment (RLHF & DPO)

Poorya Saneei

CMU Lecture 17 – Reinforcement Learning II: Applications (03/17)Stanford Week 4, Thu – Post-training (RLHF, SFT, DPO)

📌 Module 6: Post-Training with Human Feedback (RLHF & DPO)

Suggested Readings & Landmark Papers:

📄 Training language models to follow instructions with human feedback (InstructGPT) — Long Ouyang et al. (OpenAI, NeurIPS 2022)
📄 Direct Preference Optimization: Your Language Model is Secretly a Reward Model (DPO) — Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (NeurIPS 2023)

Week 12 S20 Lecture

Saturday 1405-09-14

Language Model-Based Autonomous Agents, Function Calling & Tool Use (ReAct)

Mohammad Khaki

CMU Lecture 18 – Language Model-Based Agents (03/19)Stanford Week 5, Thu – Agents, Tool Use, and RAG (Part 2)

📌 Module 9: Conversational Agents & Language Understanding

Suggested Readings & Landmark Papers:

📄 ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao (ICLR 2023)
📄 Toolformer: Language Models Can Teach Themselves to Use Tools — Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom (NeurIPS 2023)
📄 World of Bits: An Open-Domain Platform for Web-Based Agents — Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan O. Hernandez, Percy Liang (ICML 2017)

Week 12 S21 Lecture

Monday 1405-09-16

Model Quantization (AWQ, GPTQ, FP8/INT4) & Efficient Inference

Mohammad Khaki

CMU Lecture 19 – Quantization (03/26) - Lecture 20 – Parallelism and Distributed Training (03/31)

📌 Module 7: Compute-Optimal Training & Weight Quantization

Suggested Readings & Landmark Papers:

📄 LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer (NeurIPS 2022)
📄 AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Song Han (MLSys 2024)
📄 ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He (SC 2020)

Week 13 S22 Lecture

Saturday 1405-09-21

Mixture of Experts (MoE) & Efficient Attention (FlashAttention-2)

Mohadeseh Emadi

CMU Lecture 21 – Mixture of Experts (04/02) + Lecture 22 – Scaling Sequence Length

📌 Module 7: Scalable LLM Architectures (MoE & FlashAttention)

Suggested Readings & Landmark Papers:

📄 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean (ICLR 2017)
📄 FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning — Tri Dao (ICLR 2024)
📄 Mixtral of Experts — Albert Q. Jiang et al. (Mistral AI 2024)

Week 13 S23 Lecture

Monday 1405-09-23

Process Supervision & Test-Time Compute Scaling

Saba Sattari

CMU Lecture 23 – Test-Time Scaling (04/14)

📌 Module 9: Process Supervision & Test-Time Compute Scaling

Suggested Readings & Landmark Papers:

📄 Let's Verify Step by Step (PRM800K & Process Supervision) — Hunter Lightman et al. (OpenAI 2023)
📄 Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell et al. (Google DeepMind / UC Berkeley 2024)
📄 Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking — Eric Zelikman et al. (Stanford 2024)

Week 14 S24 Lecture

Saturday 1405-09-28

Mechanistic Interpretability, Safety, Bias & Adversarial Attacks in LLMs

Fateme Marefat

Stanford Week 8, Tue – Interpretability (Guest: Been Kim) + Week 8, Thu – Social Impacts (Risks)

📌 Module 4: Safety, Privacy & Bias / Module 5: Qualitative Evaluation

Suggested Readings & Landmark Papers:

📄 Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Trenton Bricken et al. (Anthropic 2023)
📄 Universal and Transferable Adversarial Attacks on Aligned Language Models — Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson (2023)
📄 Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) — Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres (ICML 2018)

Week 14 S25 Lecture

Monday 1405-09-30

Research Skills, ACL Experimental Design & Open Research Frontiers in NLP

Setare Kahnamuee

CMU Lecture 14 – Research Skills and Experimental Design (02/26)Stanford Week 10, Tue – Open Questions in NLP 2026

📌 Synthesis, ACL Experimental Design & Project Guidance

Suggested Readings & Landmark Papers:

📄 A Survey of Large Language Models — Wayne Xin Zhao et al. (AI Open 2023)
📄 Holistic Evaluation of Language Models (HELM) — Percy Liang et al. (Stanford CRFM, TMLR 2023)

Week 15 BUF1 Buffer Session

Saturday 1405-10-05

Reserve & Buffer Session (Overflow / Make-Up Class – Part 1)

Teaching Staff

💡 Reserved buffer session for syllabus catch-up, make-up lectures, or student project consultations in case of delays or unforeseen cancellations.

Week 15 BUF2 Buffer Session

Monday 1405-10-07

Reserve & Buffer Session (Overflow / Make-Up Class – Part 2)

Teaching Staff

💡 Reserved buffer session for syllabus catch-up, make-up lectures, or student project consultations in case of delays or unforeseen cancellations.

Finals Week FINAL Final Exam

Wednesday 1405-10-30

Comprehensive Final Examination (Covers Sessions S13–S25)

Teaching Staff

📌 Comprehensive Examination: Theory, Mathematical Formulations & Applied Architectures (Sessions S13–S25)

💡 Examination Time: 11:00 – 13:00 (Official IUST Final Examination Period)

Course Schedule & Timeline