Post-training reading list: 50 core papers and newer research

Post-training reading list: 50 core papers and newer research

Compiled and source-checked: 30 October 2026.

Scope and selection

The first 50 entries form a curated reading list for a broad post-training primer: supervised fine-tuning, preference learning, RLHF/RLAIF, reasoning with verifiable feedback, and distillation. Selection balances historical influence, distinct technical ideas, and practical usefulness. The numbers are reference IDs, not an objective importance ranking. Foundational papers outside language modeling are identified explicitly; two 2026 entries cover a recent method and a current synthesis whose long-term influence is still emerging.

Each entry links to the primary paper record and gives a short contribution summary and its relevance. Verification covers primary-source metadata and abstracts, with relevant full-text sections consulted for the unified-gradient and OPD connections; this is not a replication or exhaustive full-paper review. Years generally refer to the first public version, with conference years noted where helpful. Some linked records have later revisions or updated titles.

The final section adds 16 further papers first released after 2023, prioritizing 2025–2026 and ordering them by initial release date, newest first. These additions bring the list to 66 distinct papers.

Suggested starting points

For a first pass, read InstructGPT [15], DPO [18], DeepSeekMath [32], GKD/OPD [41], and Tulu 3 [46]. Then follow the topic sections below. For the blog’s shared-gradient argument, focus on DeepSeekMath §5.2.1 and Appendix A.1, MiniLLM [40], and GKD [41]. These methods differ in training-state distributions, token weighting, constraints, and auxiliary losses—not just one scalar coefficient on one sampled token.

Foundations: gradients, imitation, and stable updates

1. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

REINFORCE · 1992 · Foundational RL
Ronald J. Williams. Paper.

Introduces the REINFORCE family of score-function estimators for optimizing expected reward with stochastic networks. The return-weighted log-probability gradient is the mathematical starting point for much of language-model RL.

2. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

DAgger · 2010; AISTATS 2011 · Foundational imitation learning
Stéphane Ross, Geoffrey Gordon, and J. Andrew Bagnell. Paper.

Addresses the distribution shift caused by a learner visiting states its demonstrations did not cover. Its approach of collecting expert supervision at learner-induced states is an important conceptual precursor to on-policy distillation.

3. Trust Region Policy Optimization

TRPO · 2015 · Foundational RL
John Schulman et al. Paper.

Develops a policy-improvement framework and a practical algorithm that limits policy changes through a KL-based trust region. It explains the motivation for conservative updates that later appears in PPO and RLHF; the practical approximations do not inherit an unconditional improvement guarantee.

4. High-Dimensional Continuous Control Using Generalized Advantage Estimation

GAE · 2015 · Foundational RL
John Schulman et al. Paper.

Uses an exponentially weighted combination of temporal-difference residuals to estimate advantages, trading bias against variance. This is the standard starting point for understanding the critic and advantage computation in PPO-based post-training.

5. Proximal Policy Optimization Algorithms

PPO · 2017 · Foundational RL
John Schulman et al. Paper.

Introduces policy objectives that allow multiple minibatch updates on collected trajectories, including the widely used clipped probability-ratio surrogate. PPO became a major optimizer for RLHF and a starting point for later group-based reasoning-RL algorithms.

Supervised fine-tuning and instruction data

6. Finetuned Language Models Are Zero-Shot Learners

FLAN · 2021 · Instruction tuning
Jason Wei et al. Paper.

Shows that fine-tuning on a mixture of tasks expressed as natural-language instructions improves performance on unseen tasks. It establishes instruction tuning as a practical way to turn pretrained language modeling into more general task following.

7. Multitask Prompted Training Enables Zero-Shot Task Generalization

T0 · 2021 · Instruction tuning
Victor Sanh et al. Paper.

Converts many supervised datasets into diverse prompted tasks and trains on their mixture. Its held-out-task evaluations and released prompts make it an important complement to FLAN for understanding instruction-data diversity and generalization.

8. Scaling Instruction-Finetuned Language Models

Flan-T5 / Flan-PaLM · 2022 · Instruction tuning
Hyung Won Chung et al. Paper.

Studies scaling the task mixture, model size, and chain-of-thought supervision in instruction fine-tuning. It shows how these choices improve performance across model families and provides widely used Flan-T5 checkpoints.

9. Self-Instruct: Aligning Language Models with Self-Generated Instructions

Self-Instruct · 2022 · Synthetic instruction data
Yizhong Wang et al. Paper.

Bootstraps instruction-following data by generating instructions, inputs, and outputs from a language model and filtering the results. It is a central reference for synthetic SFT data rather than relying exclusively on manually authored demonstrations.

10. LIMA: Less Is More for Alignment

LIMA · 2023 · Data quality
Chunting Zhou et al. Paper.

Studies fine-tuning a strong pretrained model on only 1,000 carefully curated examples. It highlights the importance of demonstration quality and proposes the superficial alignment hypothesis; its results should not be read as proving that all post-training merely changes style.

11. Orca: Progressive Learning from Complex Explanation Traces of GPT-4

Orca · 2023 · Explanation-based SFT / distillation
Subhabrata Mukherjee et al. Paper.

Trains a smaller model using richer explanation traces from a stronger teacher rather than only final answers. It motivates rationale-rich synthetic supervision and examines the gap between imitating a teacher’s style and acquiring its reasoning behavior.

Human feedback, reward learning, and constitutional supervision

12. Deep reinforcement learning from human preferences

Preference-based reward learning · 2017 · Foundational preference learning
Paul Christiano et al. Paper.

Learns a reward predictor from human comparisons of behavior and uses it to train an RL policy. Although demonstrated outside language modeling, it supplies a key blueprint for the reward-model-plus-policy pipeline used in RLHF.

13. Fine-Tuning Language Models from Human Preferences

Early language-model RLHF · 2019 · RLHF
Daniel M. Ziegler et al. Paper.

Applies preference-based reward learning to language-generation tasks, including stylistic continuation and summarization. It provides an early demonstration of adapting pretrained language models to human judgments and documents opportunities to exploit annotation heuristics.

14. Learning to summarize from human feedback

Summarization RLHF · 2020 · RLHF
Nisan Stiennon et al. Paper.

Collects pairwise summary judgments, trains a reward model, and optimizes a summarization policy against it. It is an influential case study showing why optimizing human preferences can differ from optimizing reference-based metrics such as ROUGE.

15. Training language models to follow instructions with human feedback

InstructGPT · 2022 · RLHF pipeline
Long Ouyang et al. Paper.

Combines supervised demonstrations, preference-based reward modeling, and reinforcement learning to improve instruction following. It is a central reference for the modern SFT → reward model → RLHF recipe and its evaluation against larger pretrained models.

16. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Helpful and harmless RLHF · 2022 · Assistant alignment
Yuntao Bai et al. Paper.

Studies preference modeling and RLHF for dialogue assistants, including competing helpfulness and harmlessness objectives. It also investigates iterative online collection and policy updates, making it useful for understanding the data loop rather than just a single training loss.

17. Constitutional AI: Harmlessness from AI Feedback

Constitutional AI · 2022 · Constitutional supervision / RLAIF
Yuntao Bai et al. Paper.

Uses written principles to guide model-generated critiques, revisions, and preference judgments. Its supervised and RL phases demonstrate a route to AI-assisted supervision with less dependence on human labels of harmful outputs.

Direct preference optimization and alternative feedback objectives

18. Direct Preference Optimization: Your Language Model is Secretly a Reward Model

DPO · 2023 · Offline preference optimization
Rafael Rafailov et al. Paper.

Reparameterizes a KL-regularized preference-learning problem into a direct policy loss on preferred and rejected response pairs. It removes the explicit reward-model-training and online RL loop from that recipe, making offline preference optimization substantially simpler.

19. A General Theoretical Paradigm to Understand Learning from Human Preferences

ΨPO / IPO · 2023 · Preference-learning theory
Mohammad Gheshlaghi Azar et al. Paper.

Develops a framework for learning directly from pairwise preferences and examines assumptions behind RLHF and DPO. Its Identity Preference Optimization variant offers a different objective and helps clarify how preference fitting and regularization interact.

20. KTO: Model Alignment as Prospect Theoretic Optimization

KTO · 2024 · Unpaired feedback
Kawin Ethayarajh et al. Paper.

Builds an alignment objective inspired by prospect theory that can learn from desirable/undesirable labels instead of paired comparisons. It broadens the feedback formats available for post-training and stresses that objective choice depends on the desired inductive bias.

21. ORPO: Monolithic Preference Optimization without Reference Model

ORPO · 2024 · Reference-free preference optimization
Jiwoo Hong, Noah Lee, and James Thorne. Paper.

Combines supervised learning with an odds-ratio preference penalty in one reference-model-free training stage. It is useful for comparing unified SFT-and-preference recipes with pipelines that separate the two stages.

22. SimPO: Simple Preference Optimization with a Reference-Free Reward

SimPO · 2024 · Reference-free preference optimization
Yu Meng, Mengzhou Xia, and Danqi Chen. Paper.

Uses average sequence log probability as an implicit reward and adds a target margin between preferred and rejected responses. It isolates the roles of length normalization, margins, and reference models in offline preference training.

23. RRHF: Rank Responses to Align Language Models with Human Feedback without tears

RRHF · 2023 · Ranking-based alignment
Zheng Yuan et al. Paper.

Trains response log probabilities to respect preference rankings among candidates from different sources. It is an early, simple ranking-based alternative to PPO and connects response selection, supervised training, and preference alignment.

24. SLiC-HF: Sequence Likelihood Calibration with Human Feedback

SLiC-HF · 2023 · Likelihood calibration
Yao Zhao et al. Paper.

Calibrates sequence likelihoods using human preference comparisons, including feedback collected for a different model. It demonstrates an offline alternative to PPO-style RLHF and is useful historical context for direct preference objectives.

25. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

RLAIF · 2023 · AI-generated feedback
Harrison Lee et al. Paper.

Compares AI-generated and human preference labels across summarization and dialogue tasks. It also studies obtaining rewards directly from a model during RL, clarifying when AI feedback can substitute for parts of a human-feedback pipeline.

Reasoning, verifiers, self-training, and online RL

26. Training Verifiers to Solve Math Word Problems

GSM8K / learned verifiers · 2021 · Verifier training / evaluation
Karl Cobbe et al. Paper.

Introduces GSM8K and trains verifiers to rank sampled mathematical solutions. It is a precursor to verifier-guided reasoning and reward learning; selecting outputs with a verifier is distinct from updating a generator through RL.

27. STaR: Bootstrapping Reasoning With Reasoning

STaR · 2022 · Reasoning self-training
Eric Zelikman et al. Paper.

Iterates between generating rationales, retaining successful reasoning, and fine-tuning the model. It is a key example of learning reasoning from self-generated traces rather than treating chain-of-thought only as a prompting technique.

28. Let’s Verify Step by Step

Process supervision / PRM800K · 2023 · Process supervision
Hunter Lightman et al. Paper.

Compares feedback on intermediate reasoning steps with feedback on final outcomes and releases PRM800K. It is a central reference for process reward models and the question of where supervision should be applied within a solution.

29. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Math-Shepherd · 2023 · Automatic process supervision
Peiyi Wang et al. Paper.

Constructs process-level supervision automatically and trains a reward model to score individual solution steps. It applies those scores both to reranking and to stepwise RL, linking scalable verification with policy improvement.

30. Reinforced Self-Training (ReST) for Language Modeling

ReST · 2023 · Generate-and-improve training
Caglar Gulcehre et al. Paper.

Alternates generating a dataset from the policy with improving the policy using scored samples and offline updates. It emphasizes reuse of generated data and provides a practical bridge between self-training and online RLHF.

31. Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

ReST-EM · 2023 · Verified self-training
Avi Singh et al. Paper.

Repeatedly generates solutions, filters them using binary correctness feedback, and fine-tunes on the retained examples. It shows why verifiable tasks can support iterative improvement with less dependence on new human demonstrations.

32. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

GRPO · 2024 · Group-based policy optimization
Zhihong Shao et al. Paper.

Introduces GRPO, using relative rewards within groups of responses to avoid training a separate critic. Section 5.2.1 and Appendix A.1 also compare SFT, preference learning, and RL through data sources and gradient coefficients, directly supporting a unified post-training discussion.

33. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

RLOO · 2024 · Critic-free RLHF
Arash Ahmadian et al. Paper.

Revisits simpler REINFORCE-style estimators, including leave-one-out baselines, for language-model RLHF. It questions whether all of PPO’s machinery is necessary in this setting and makes an important comparison between online RL and offline preference methods.

34. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-R1 · 2025 · Reasoning RL / distillation
DeepSeek-AI et al. Paper.

Demonstrates large-scale reinforcement learning for verifiable reasoning tasks and studies transferring the resulting reasoning behavior into smaller models. It is a major reference for reasoning-oriented post-training and the interaction between RL, generated reasoning data, and distillation.

35. DAPO: An Open-Source LLM Reinforcement Learning System at Scale

DAPO · 2025 · Scalable reasoning RL
Qiying Yu et al. Paper.

Presents a reproducible large-scale reasoning-RL recipe with decoupled clipping, dynamic sampling, token-level policy-gradient loss, and overlong reward shaping. Its importance is both algorithmic and practical: training details and system choices can determine whether a nominally similar RL objective works.

36. Understanding R1-Zero-Like Training: A Critical Perspective

Dr. GRPO · 2025 · RL analysis / objective bias
Zichen Liu et al. Paper.

Examines how pretrained capabilities and GRPO normalization choices affect reasoning-RL results. It identifies optimization biases, including length-related effects, and proposes Dr. GRPO; useful for avoiding overly simple narratives about capabilities emerging only during RL.

37. Group Sequence Policy Optimization

GSPO · 2025 · Sequence-level policy optimization
Chujie Zheng et al. Paper.

Defines importance weighting and clipping at the sequence level instead of relying on token-level importance ratios. It studies stability and efficiency, including mixture-of-experts training, and broadens the comparison beyond PPO and GRPO.

Distillation and learning from teacher distributions

38. Distilling the Knowledge in a Neural Network

Knowledge distillation · 2015 · Foundational distillation
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Paper.

Develops teacher–student training using softened predictive distributions to transfer knowledge into a deployable model. Although not an LLM paper, it is the foundational reference for treating a teacher distribution as richer supervision than a hard target.

39. Sequence-Level Knowledge Distillation

SeqKD · 2016 · Sequence distillation
Yoon Kim and Alexander M. Rush. Paper.

Studies distillation for neural machine translation and introduces sequence-level teacher-generated targets. It explains the ancestry of training a student on a stronger model’s complete responses, which differs from matching logits at every token.

40. MiniLLM: On-Policy Distillation of Large Language Models

MiniLLM · 2023 · Reverse-KL distillation
Yuxian Gu et al. Paper.

Uses reverse KL and an on-policy optimization approach to distill generative language models. It is especially relevant to connecting distillation with policy-gradient reasoning and to understanding why the direction of KL changes the learning behavior.

41. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

GKD / OPD · 2023 · On-policy distillation
Rishabh Agarwal et al. Paper.

Trains students on their own generated prefixes while querying teacher distributions at those states, and generalizes the choice of distribution-matching loss and data mixture. It is the central reference for OPD, train–inference state mismatch, and combining distillation with RL.

42. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Distilling step-by-step · 2023 · Rationale distillation
Cheng-Yu Hsieh et al. Paper.

Extracts teacher rationales and uses them as auxiliary supervision alongside task labels in a multitask student-training setup. It provides a concrete example of distilling reasoning explanations rather than only predictions or token distributions.

43. Reinforcement Learning via Self-Distillation

SDPO · 2026 · Recent research: self-distillation / rich feedback
Jonas Hübotter et al. Paper.

Conditions a model on rich feedback, such as runtime errors, to form a self-teacher, then distills its feedback-informed token predictions into the policy. This recent direction connects RL with dense teacher-like feedback without requiring a separate external teacher.

Practical fine-tuning and complete post-training recipes

44. LoRA: Low-Rank Adaptation of Large Language Models

LoRA · 2021 · Parameter-efficient fine-tuning
Edward J. Hu et al. Paper.

Freezes pretrained weights and learns low-rank parameter updates, reducing the number of trainable parameters and optimizer memory. It changes how post-training updates are parameterized rather than supplying a new reward or preference objective.

45. QLoRA: Efficient Finetuning of Quantized LLMs

QLoRA · 2023 · Memory-efficient fine-tuning
Tim Dettmers et al. Paper.

Combines a frozen quantized base model with trainable low-rank adapters, using techniques including NF4, double quantization, and paged optimizers. It is a major practical reference for making instruction fine-tuning feasible with limited accelerator memory.

46. Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Tulu 3 · 2024 · Open end-to-end recipe
Nathan Lambert et al. Paper.

Releases an open post-training recipe spanning data curation, SFT, DPO, and reinforcement learning with verifiable rewards. Its evaluation and decontamination methodology make it valuable for understanding the full pipeline rather than comparing isolated losses.

47. The Llama 3 Herd of Models

Llama 3 · 2024 · Large-scale model report
Meta / Aaron Grattafiori et al. Paper.

Describes a large model family and its pretraining and post-training pipeline, including instruction following, reasoning, coding, and tool use. The post-training sections are a useful industrial-scale complement to method papers and fully open recipes.

Failure modes and the reliability of feedback

48. Scaling Laws for Reward Model Overoptimization

Reward overoptimization · 2022 · Reward reliability
Leo Gao, John Schulman, and Jacob Hilton. Paper.

Studies how optimizing an imperfect proxy reward can eventually reduce performance under a separate gold reward model. Its synthetic experiments compare RL and best-of-n sampling, making it important evidence that higher training reward need not mean better underlying behavior.

49. Towards Understanding Sycophancy in Language Models

Sycophancy · 2023 · Preference-induced failure modes
Mrinank Sharma et al. Paper.

Investigates assistants agreeing with user beliefs rather than being truthful and examines the role of preference judgments. It shows why human or model preference feedback can reward persuasive agreement and why preference alignment needs behavioral evaluation beyond reward scores.

A current map of the field

50. Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Unified post-training survey · 2026 · Recent survey
Shiwan Zhao et al. Paper.

Organizes post-training by the provenance of training trajectories: external data versus learner-generated rollouts. It connects SFT, preferences, RL, verifiers, and distillation through behavior and state distributions; included as a recent synthesis, not as an established historical landmark.

Newer priorities: papers first released after 2023

Added and source-checked: 30 October 2026. The following 16 additional papers are ordered by first arXiv submission date, newest first. Primary-source titles, authors, dates, and abstracts were checked. Inclusion reflects technical relevance to this primer; the newest preprints are promising reading candidates, not an established citation-based ranking.

Several essential recent papers are already in the core list: GRPO [32], RLOO [33], DeepSeek-R1 [34], DAPO [35], Dr. GRPO [36], GSPO [37], SDPO [43], Tulu 3 [46], and the 2026 unified survey [50]. Read those alongside this extension rather than interpreting the additions as replacements.

For a focused second pass, start with OPDArena [52] for retention, vOPD [58] for the RL–distillation connection, entropy dynamics [61] for exploration, LUFFY [62] for mixing data sources, and PRIME [64] for process feedback.

51. Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

Projected distillation · 2026-10-20 · Recent preprint
Shripad V. Deshmukh et al. Paper.

Minimally edits failed student attempts into correct targets, then learns contrastively from the corrected and failed responses. Relevant when all sampled attempts fail: it explores a middle ground between purely on-policy feedback and potentially distant teacher-written solutions.

52. OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

OPDArena · 2026-10-19 · Recent preprint
Yuanze Xu et al. Paper.

Compares SFT, RLVR, and OPD across pretrained students and different teachers, measuring task gains alongside retained capabilities. It reports strong teacher-dependent differences in forgetting and connects retention to out-of-domain distribution changes, making it useful for evaluating more than target-task accuracy.

53. Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models

Dr. OPD · 2026-09-29 · Recent preprint
Zhenyu Wang et al. Paper.

Learns token-level weights for teacher supervision through a bilevel objective aimed at improving student reward. It directly challenges uniform weighting of teacher feedback and is particularly relevant to the blog’s discussion of gradient coefficients.

54. Scaling Properties of Same-Family On-Policy Distillation

OPD scaling · 2026-09-26 · Recent preprint
Yuntai Bao et al. Paper.

Studies transfer across weak-to-strong, same-base, and strong-to-weak teacher–student configurations within model families. It reports scaling relationships and cases where teacher benchmark score alone fails to predict teaching value; useful for choosing teachers, with the findings limited to the studied setups.

55. Distilled Reinforcement Learning for LLM Post-training

Distilled RL · 2026-07-19 · Recent preprint
Chen Wang et al. Paper.

Integrates teacher supervision into RL using clipped reverse importance weighting, negative-sample reset, and sequence-level geometric normalization. It targets selective transfer rather than unconditional teacher imitation and evaluates both within-family and cross-family teacher–student settings.

56. MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

MOPD · 2026-06-29 · Recent preprint
Wenhan Ma et al. Paper.

Distills several domain-specialized RL teachers into one student using the student’s own rollouts. It extends the post-training discussion from improving one skill to integrating several capabilities and reports use in an industrial model-training pipeline.

57. Distilling LLM Feedback for Lean Theorem Proving

Feedback distillation · 2026-05-29 · Recent preprint
Gaetan Narozniak et al. Paper.

Distills a model’s predictions conditioned on privileged language-model feedback into a policy that does not receive that feedback at inference. Its Lean4 experiments examine trajectory diversity, pass@k, and combining feedback distillation with subsequent GRPO, rather than treating the methods as mutually exclusive.

58. KL for a KL: On-Policy Distillation with Control Variate Baseline

vOPD · 2026-05-08 · Recent preprint
Minjae Oh et al. Paper.

Frames sampled reverse-KL OPD as policy-gradient learning and uses a detached control-variate baseline derived from teacher–student KL to reduce variance. A useful bridge between the baseline mathematics of RL and practical distillation stability, without introducing a separately trained critic.

59. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

OPD dynamics · 2026-04-14 · Recent preprint
Yaxuan Li et al. Paper.

Investigates when teacher feedback helps a student and when differences in thinking patterns or limited new knowledge make transfer ineffective. It studies token-level alignment and proposes off-policy cold starts and teacher-aligned prompt selection as recovery strategies in its experiments.

60. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

OPSD · 2026-01-26 · Recent preprint
Siyan Zhao et al. Paper.

Uses one model as both student and teacher, with the teacher conditioned on privileged information such as verified reasoning traces. It trains on student rollouts while matching the context-enriched teacher distribution, complementing the rich-feedback SDPO approach already listed as [43].

61. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

Entropy dynamics / Clip-Cov / KL-Cov · 2025-05-28 · 2025 research
Ganqu Cui et al. Paper.

Analyzes how policy-gradient updates change entropy and investigates the relationship between entropy loss, exploration, and performance saturation. It proposes token-level interventions to reduce entropy collapse; the reported entropy–performance relationship is empirical, not a universal scaling law.

62. Learning to Reason under Off-Policy Guidance

LUFFY · 2025-04-21 · 2025 method
Jianhao Yan et al. Paper.

Combines teacher-provided reasoning traces with on-policy rollouts through mixed-policy optimization and importance weighting. Useful for discussing why demonstrations and exploration can work together, especially when a weak student rarely produces successful trajectories on its own.

63. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

RLVR capability boundaries · 2025-04-18 · 2025 analysis
Yang Yue et al. Paper.

Uses large-k pass@k, coverage, and likelihood analyses to compare base models and RLVR-trained models. Its experiments suggest that improved single-sample accuracy can coexist with reduced solution coverage; this is evidence about the tested methods and regimes, not a proof that RL can never learn new reasoning.

64. Process Reinforcement through Implicit Rewards

PRIME · 2025-02-03 · 2025 method
Ganqu Cui et al. Paper.

Updates process reward models online using rollouts and outcome labels rather than expensive human annotations of every reasoning step. It is a practical reference for denser credit assignment and for keeping reward supervision synchronized with the evolving policy.

65. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Reproducible PPO-RLHF · 2024-03-24 · Implementation reference
Shengyi Huang et al. Paper.

Reproduces the summarization RLHF pipeline and documents more than twenty implementation choices that affect results. It is worth retaining alongside newer algorithm papers because details of reward processing, model training, and evaluation can dominate nominal objective differences.

66. RewardBench: Evaluating Reward Models for Language Modeling

RewardBench · 2024-03-20 · Evaluation reference
Nathan Lambert et al. Paper.

Benchmarks reward models on chosen/rejected response pairs spanning chat, safety, and reasoning, including challenging and out-of-distribution comparisons. It provides a direct way to examine the quality of the learning signal before optimizing a policy against it.