Reasoning in Language Models

Reasoning in language models refers to solving tasks that require intermediate deductions, calculations, or planning. Researchers study prompting, search, training, and inference computation as ways to improve performance. Generated explanations are not direct evidence of the model's internal reasoning.[1][4]

Reference entry · last updated 20260909 · Previous version

First principles and definitions

An autoregressive model predicts the next token from its context. A generated calculation can become part of the context used to produce the answer. Prompting for intermediate steps is one method; sampling and evaluating candidate solutions are methods.[1][3]

Answer correctness and explanation faithfulness are distinct. A model can produce plausible steps that fail to explain what caused its answer. Lanham et al. found that faithfulness varied across the models and tasks they tested.[4]

Prompting and sampled answers

Chain-of-thought (CoT) prompting elicits intermediate steps before an answer. Wei et al. used worked examples in the prompt. Kojima et al. studied zero-shot prompting with “Let's think step by step,” followed by answer extraction. Both reported gains on selected reasoning benchmarks.[1][2]

Self-consistency samples several reasoning paths and selects the most common answer. Wang et al. aggregate over sampled paths, not over an identified set of errors. Agreement can improve benchmark accuracy but cannot establish correctness when paths share mistakes.[3]

Test-time compute

Test-time compute is computation spent while answering a request. It can fund longer generation, additional candidates, or evaluation and search.

Snell et al. studied verifier-guided search and iterative answer revision on mathematical problems. The effective allocation depended on problem difficulty and the available model and method. Their results support task-dependent allocation, not a universal power law between inference compute and accuracy.[5]

Step scoring and search

Process supervision

Outcome supervision scores a completed answer; process supervision provides feedback on intermediate steps. Lightman et al. trained models from human step-level feedback and evaluated them by selecting among sampled mathematical solutions. Process supervision outperformed outcome supervision in that experiment. A learned score remains an estimate, not a proof that a step is valid.[6]

Search and reinforcement learning

Tree of Thoughts, studied by Yao et al., explores alternative intermediate steps, evaluates them, and can backtrack. The paper reports results on Game of 24, creative writing, and mini crosswords. This is an explicit search procedure, not a property established by asking for one chain of thought.[7]

DeepSeek-R1-Zero is a training example. It starts from a pretrained base model and applies reinforcement learning without a preliminary supervised fine-tuning stage. The report uses accuracy and format rewards and describes emerging behaviors such as checking and revising answers. This does not imply that all reasoning models use the same training or search architecture.[8]

Evaluation and limits

The cited studies measure different outcomes: final-answer accuracy, selection of mathematical solutions, task success under search, or changes caused by interventions on intermediate text. Their results should be read with the tested model, task, and compute budget. Higher answer accuracy alone does not demonstrate faithful explanations.[4][5][6]

References

Jason Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
Takeshi Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners.
Xuezhi Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models.
Tamera Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning.
Charlie Snell et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.
Hunter Lightman et al. (2023). Let's Verify Step by Step.
Shunyu Yao et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models.
DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.