Learning-Resources

A compilation of resources for keeping up with the latest trends in NLP.

Note: This resource list is a work in progress. More papers and topics will be added regularly. Contributions and suggestions are welcome!

Some Fundamental Transformers

1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2. GPT1
3. GPT2
4. T5
5. XLNet: Generalized Autoregressive Pretraining for Language Understanding
6. RoBERTa: A Robustly Optimized BERT Pretraining Approach
7. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
8. Longformer: The Long-Document Transformer
9. ROFORMER
10. Language Models are Few-Shot Learners - GPT3 paper

Fundamental LLM & Transformer Papers/Blogs

1. Attention is all you need
2. Memory Is All You Need
3. Byte-pair Encoding
4. The Illustrated Transformer Blog
5. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer - MoE paper for LMs
6. Fast Transformer Decoding: One Write-Head is All You Need - Multi-Query Attention (MQA) Paper
7. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints - Grouped Query Attention Paper

Reinforcement Learning for LLMs

1. Basics of RL - OpenAI
2. Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism
3. InstructGPT
4. Training language models to follow instructions with human feedback
5. Deep Reinforcement Learning from Human Preferences

DPO:

1. DPO paper
2. Blog - Math behind DPO

PPO:

1. Proximal Policy Optimization Algorithms
2. PPO Docs OpenAI
3. Understanding PPO from First Principles Blog

GRPO:

1. DeepSeekMath
2. Blog - GRPO Explained
3. DeepSeek-R1

Mechanistic Interpretability

1. Basic Mech Interp Essay
2. Toy Neural Nets with low dimensional inputs
3. Mechanistic Interpretability for AI Safety Review
4. A Mathematical Framework for Transformer Circuits
5. Circuit Tracing: Revealing Computational Graphs in Language Models

Scaling Laws

1. Scaling Laws for Neural Language Models
2. Scaling Laws for Autoregressive Generative Modeling
3. Sacling Laws of Synthetic Data for Lnguage Models
4. Scaling Laws for Transfer
5. Unified Scaling Laws for Routed Language Models - Scaling laws for MoEs

MLSys

1. Mixed Precision Training
2. Matrix multiplication - Nvidia Blog
3. Understanding GPU Performance - Nvidia Blog
4. How to Train Really Large Models on Many GPUs? - Blog
5. Efficiently Scaling Transformer Inference
6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
7. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills