Course Project: Multi-Hop Question Answering with RAG & GraphRAG
An end-to-end graduate research engineering project building state-of-the-art Multi-Hop Retrieval-Augmented Generation (RAG) and Knowledge Graph (GraphRAG) systems with exact IR evaluation, hybrid retrieval, graph traversal, and Dual-GPU acceleration on Kaggle.
Project Mentorship Team (Teaching Assistants)
The project teaching assistants guide students through implementation, host office hours, and conduct individual oral defense evaluations:
Milestones & Oral Defense Schedule
The capstone project is organized into 5 progressive milestones. Following code and report submissions for each milestone, students participate in an individual oral defense check-in with the assigned teaching assistants:
| Phase | Topic & Scope | Submission Due Date | Oral Defense Window | Lead TA / Status |
|---|---|---|---|---|
| Phase 1 |
Dense Retrieval & Evaluation
Handcrafted IR metrics (Recall@k, HitAll@k, MRR, nDCG@10), document masking, and 3 baselines (Closed-Book, Oracle-Context, Dense RAG)
|
Thursday, 1405-07-30 (23:59) | Sunday, 1405-08-03 to 1405-08-07 |
Mohammad Amin Haeri (@rstlss0)
● Active (Detailed in Handout)
|
| Phase 2 |
Hybrid Retrieval & Re-ranking
Sparse lexical retrieval (BM25), Reciprocal Rank Fusion (RRF), Cross-Encoder re-ranking, and token context budgeting
|
Thursday, 1405-08-21 (23:59) | Sunday, 1405-08-24 to 1405-08-28 | Will be released in Handout update |
| Phase 3 |
Knowledge Graph & GraphRAG
Triple extraction on passage subsets, entity graph construction, 2-hop traversal, and graph-augmented prompting
|
Thursday, 1405-09-12 (23:59) | Sunday, 1405-09-15 to 1405-09-19 | Will be released in Handout update |
| Phase 4 |
Multi-Hop Routing & Final Evaluation
Adaptive query routing, iterative step-by-step reasoning, query reformulation, and predictions on hidden benchmark
|
Thursday, 1405-10-03 (23:59) | Sunday, 1405-10-06 to 1405-10-10 | Will be released in Handout update |
| Phase 5 |
Supplementary Phase (TBD)
Supplementary details and specifications will be announced in future updates.
|
Thursday, 1405-11-01 (23:59) | Sunday, 1405-11-04 to 1405-11-07 | To be announced |
Dense Retrieval & Evaluation
Handcrafted IR metrics (Recall@k, HitAll@k, MRR, nDCG@10), document masking, and 3 baselines (Closed-Book, Oracle-Context, Dense RAG).
Hybrid Retrieval & Re-ranking
Sparse lexical retrieval (BM25), Reciprocal Rank Fusion (RRF), Cross-Encoder re-ranking, and token context budgeting.
Knowledge Graph & GraphRAG
Triple extraction on passage subsets, entity graph construction, 2-hop traversal, and graph-augmented prompting.
Multi-Hop Routing & Final Evaluation
Adaptive query routing, iterative step-by-step reasoning, query reformulation, and predictions on hidden benchmark.
Supplementary Phase (TBD)
Supplementary details and specifications will be announced in future updates.
Submission Requirements & Delivery Format
For each milestone, students submit their work via the course portal. Please refer to the official Project Handout (PDF) and files in the Starter Kit for exact mathematical formulas, baseline requirements, and output schemas:
A clean, modular private repository containing pinned dependencies in requirements.txt, and a comprehensive README.md detailing environment setup and exact reproduction commands.
Prediction files formatted strictly according to predictions_format.md (provided in starter.zip), ready for automated scoring via the official score.py evaluation script.
A concise engineering report covering model selection, comparative metric tables with confidence intervals, baseline evaluations, and qualitative error analysis.
The official score.py script in starter.zip computes Exact Match (EM), Token F1, and IR retrieval metrics along with 95% bootstrap confidence intervals ($B=1000$):
Evaluation Protocol & Oral Defense Structure
Grade Weighting
Approximately 40% of each phase grade is assigned to code correctness, prediction quality, and the written report; 60% is determined by the live oral defense and technical comprehension.
Oral Defense Check-In (10–15 Mins)
A 10–15 minute check-in with the phase lead TA: live pipeline execution on 3–5 random queries, inspecting retrieved passages, prompt templates, and testing live parameter adjustments.
Conceptual & Analytical Q&A
In-depth discussion on metric trade-offs, empirical effects of hyperparameters (e.g. $k$, document masking, prompt design), and failure mode diagnosis.
Deadlines, Grace Policy & Software Engineering Bonus
48-Hour Grace Budget
Each student receives 48 free grace hours across the entire semester to accommodate unexpected delays without penalty.
Late Deduction Rate
Submissions beyond the 48 grace hours incur a 0.5-point deduction per hour (out of 100 for the phase).
Extended LMS Window
The submission portal remains open for 96 hours (4 days) after the deadline to handle unforeseen technical issues.
Early Submission Bonus
Submitting early with strong performance (>80%) awards bonus hours (up to 24 hours) back into your semester grace quota.
🏆 Software Engineering & MLOps Extra Credit
Adopting production-grade software engineering practices earns extra credit, evaluated in 1–2 minutes during the oral defense:
run_config.yaml).
pytest verifying IR metrics and document masking.