System 1 vs. System 2 Thinking in LLM Architecture

2026-07-06

To understand why test-time compute is so effective, we must borrow from cognitive psychology—specifically, Daniel Kahneman's framework of human cognition. Kahneman divides thought into two distinct modes: System 1 (fast, intuitive, automatic, and heuristic-based) and System 2 (slow, deliberate, analytical, and logical) [cite: 14].

Foundational LLMs are, by their architectural nature, System 1 thinkers [cite: 14]. They excel at rapidly recalling facts, mimicking linguistic patterns, and generating immediate, intuitive responses. However, because standard autoregressive generation operates strictly forward—predicting one token at a time without the ability to backtrack or verify—these models stumble on tasks requiring deep logical deduction, long-horizon planning, or multi-step mathematical proofs [cite: 14, 15].

Achieving artificial System 2 thinking requires breaking the model out of its linear generation loop. It requires the introduction of "thoughts"—sequences of tokens representing intermediate steps in a reasoning process that are hidden from the final user output [cite: 15]. By allowing the AI to simulate future states, evaluate different reasoning paths, and iteratively refine its logic, developers are bridging the gap between intuitive pattern matching and deliberate problem-solving [cite: 14, 15]. This explicit representation of reasoning paths allows for systematic exploration, enabling the model to navigate complex search spaces and converge on optimal solutions rather than just statistically probable ones [cite: 15].

The Mechanics of Test-Time Compute

Scaling inference compute is not as simple as asking a model to "think harder." It requires robust algorithmic frameworks to structure the model's internal exploration. While various methodologies exist, their effectiveness critically depends on the difficulty of the prompt and the available compute budget [cite: 7, 16].

Best-of-N and Beam Search

At the foundational level of test-time scaling are parallel and sequential sampling techniques designed to surface the best possible response from a multitude of attempts.

Best-of-N (Majority Voting): The system generates N independent, complete solutions to a problem in parallel. An evaluator (often the model itself, acting as a judge) scores these solutions, or the system simply selects the most frequently occurring final answer through self-consistency [cite: 6, 16, 17]. This is highly effective for tasks with deterministic answers (like arithmetic) but becomes computationally wasteful on open-ended reasoning tasks.
Beam Search: Rather than generating entirely independent solutions, the model generates N possible options for the very first step of the problem. It scores these initial steps, discards the lowest-performing paths, and only retains the top candidates (the "beam"). From those top candidates, it generates the next step, scores again, and repeats [cite: 17, 18].

While Beam Search is more efficient than Best-of-N, it traditionally relies on log-likelihood scoring—meaning it favors the most predictable next step rather than the most logically sound one [cite: 18]. In complex reasoning, the correct mathematical derivation is often of low statistical likelihood compared to a common misconception, leading Beam Search to discard valid, innovative reasoning paths prematurely.

Monte Carlo Tree Search (MCTS)

To overcome the limitations of simple likelihood scoring, advanced reasoning models implement algorithms derived from reinforcement learning, specifically Monte Carlo Tree Search (MCTS). MCTS provides a principled framework for balancing the exploration of new, untested reasoning paths against the exploitation of paths that currently seem promising [cite: 19].

Think of MCTS as navigating a massive, complex physical maze. Instead of running blindly down the first path you see (System 1), you leave breadcrumbs at every intersection. You walk down one path, and if it looks like a dead end, you backtrack to the last breadcrumb and try a different route, dedicating your time to the paths that show the most promise of an exit. This structured exploration prevents the model from getting stuck in logical dead ends.

When applied to LLMs, MCTS constructs a decision tree where each node represents a partial reasoning trace (a state), and each branch represents the generation of the next logical step (an action) [cite: 20]. The algorithm operates in four continuous phases [cite: 19]:

Selection: The system traverses the current tree of thoughts from the root (the prompt) to a leaf node, selecting paths that currently hold the highest estimated value.
Expansion: The LLM generates one or more possible next steps (actions) from that leaf node, expanding the tree deeper into the solution space.
Simulation: The model rapidly "rolls out" a potential continuation from the new step to see if it eventually leads to a viable conclusion.
Backpropagation: The success or failure of that simulation is passed back up the tree, updating the value scores of all preceding nodes. This ensures that early reasoning steps are judged based on their long-term utility [cite: 19].

MCTS introduces strategic adaptability. By structuring its reasoning like a maze, the AI actively explores paths, scores potential outcomes, and backtracks to promising nodes when an error is detected. If a reasoning path begins to look unpromising, the tree search naturally abandons it, reallocating compute to more fruitful branches [cite: 14]. This mirrors human planning processes, enabling the LLM to identify high-reward paths and exhibit generalized reasoning across highly diverse, complex domains [cite: 14].

The Reward Mechanism: ORMs vs. PRMs

The efficacy of algorithms like MCTS hinges entirely on how accurately the system can score a specific reasoning step. If the AI cannot reliably tell the difference between a brilliant deduction and a logical fallacy, tree search simply scales the generation of garbage. This introduces the critical distinction between Outcome Reward Models (ORMs) and Process Reward Models (PRMs).

The Limits of Outcome Reward Models (ORMs)

Traditional reinforcement learning for LLMs relies on Outcome Reward Models. An ORM provides a single, sparse reward at the absolute end of the generation process—evaluating only the final answer [cite: 21]. If a model outputs the correct final code snippet or mathematical proof, the ORM issues a positive reward [cite: 21].

While ORMs are substantially easier and cheaper to train because they do not require human annotators to grade intermediate work, they suffer from critical alignment failures. Most notably, they are susceptible to false positive logic—also known as reward hacking [cite: 22]. An LLM might make two completely flawed logical leaps that happen to cancel each other out, miraculously arriving at the correct final answer. An ORM will reward this behavior, inadvertently teaching the model that hallucinated logic is acceptable as long as the destination is correct [cite: 21].

Enter Process Reward Models (PRMs)

Process Reward Models resolve the sparse reward problem by applying stepwise supervision. A PRM evaluates the entire reasoning trajectory, assigning a mathematical correctness score to every single intermediate thought or action [cite: 21].

Feature	Outcome Reward Models (ORMs)	Process Reward Models (PRMs)
Evaluation Scope	Evaluates only the final, terminal output [cite: 21].	Evaluates every intermediate step in the reasoning chain [cite: 22, 23].
Reward Signal	Sparse (only given at the end) [cite: 21].	Dense (continuous feedback loop) [cite: 21, 22].
Training Cost	Low (requires only pairs of prompts and final answers) [cite: 22].	High (requires step-by-step human annotation or advanced synthetic data) [cite: 22].
Primary Vulnerability	Prone to rewarding flawed logic that accidentally yields the right answer [cite: 21].	Prone to overcrediting incorrect individual steps (false positives) [cite: 22].

By providing granular, dense supervision, PRMs ensure that the model follows logical, safe, and verifiable paths [cite: 21]. In high-stakes environments like code generation or mathematical theorem proving, PRMs are what allow an MCTS algorithm to prune dead-end branches immediately, drastically improving the efficiency of the search process [cite: 21]. Instead of waiting for a 100-step proof to finish before realizing step 3 was flawed, the PRM flags the error instantly, forcing the model to backtrack.

Implementing MCTS with Process Reward Models

To ground these concepts, we can observe how a Process Reward Model interacts with a tree-search algorithm in practice. In the architecture of an inference-scaled model, the generative policy (the base LLM) acts as the action-generator, while the PRM acts as the critic.

The following Python implementation demonstrates a simplified interaction where an LLM generates candidate reasoning steps, and a PRM evaluates them dynamically to select the optimal path to a final answer.

def mcts_with_prm(state_node, prm_model, max_steps=10):
 for step in range(max_steps):
 # Generate possible next reasoning steps (actions) from the current state
 candidate_steps = state_node.generate_next_steps()
 
 best_score = -float('inf')
 best_step = None
 
 for step_candidate in candidate_steps:
 # PRM evaluates the intermediate reasoning step (dense reward)
 # Evaluates the full history plus the new candidate
 step_score = prm_model.evaluate_step(state_node.history + [step_candidate])
 
 if step_score > best_score:
 best_score = step_score
 best_step = step_candidate
 
 # The tree advances down the path with the highest PRM score
 state_node.apply_step(best_step)
 
 # Check if the reasoning chain has reached a natural conclusion
 if state_node.is_terminal():
 break
 
 return state_node.get_final_answer()

In this snippet, candidate_steps represents the expansion phase of the search. The LLM might generate three different ways to approach the next line of a physics equation. The prm_model.evaluate_step() function provides the critical dense reward signal [cite: 24]. Rather than blindly trusting the LLM's autoregressive likelihood, the PRM actively assesses the logical validity of each branch, updating the best_score. Once the optimal step is chosen, it is applied to the state node, and the cycle repeats until the model determines it has reached a terminal, solved state [cite: 23, 24].

To ground this abstract algorithm in reality, consider an illustrative case study where an AI is tasked with solving a complex algebra problem: extracting the roots of a high-degree polynomial.

The Flawed Step (False Positive/Negative): During the 'Expansion' phase, the base model generates a step attempting to factor the equation but incorrectly drops a negative sign.
PRM Evaluation: Under a traditional ORM, if the model miraculously canceled out this error in a later step and arrived at the correct answer, the flawed logic would be rewarded. However, the PRM actively evaluates this specific algebraic manipulation as soon as it is generated. It calculates a distinctly low mathematical correctness probability for that exact flawed step.
Backtracking: The PRM flags the step as invalid—a dead end. The MCTS algorithm immediately halts generation on this branch, backtracks to the previous verified step (the last known good 'breadcrumb'), and prompts the model to generate a different, mathematically sound operation. This surgical correction prevents the model from wasting extensive compute on a doomed reasoning chain.

Vulnerabilities and Failure Modes: Reward Hacking in PRMs

Despite their power in unlocking System 2 reasoning, Process Reward Models are not infallible. The primary bottleneck in current test-time scaling research is the phenomenon of reward hacking, driven specifically by PRM false positives [cite: 22].

Because PRMs are neural networks themselves, they can be deceived. Research indicates that PRMs often overcredit incorrect intermediate steps—they assign a high correctness probability to a logical step that looks structurally sound but contains a subtle mathematical or factual error [cite: 22]. When a PRM issues a false positive, it locally misguides the decoding process. The tree search algorithm, trusting the PRM, commits heavy computational resources to a branch that is actually a dead end [cite: 22].

When these false positives aggregate across long reasoning chains, they systematically bias the entire search [cite: 22]. The model ends up trapped in a highly-rated hallucination. Conversely, false negatives (where the PRM punishes a correct but highly unorthodox reasoning step) mainly slow down convergence, but false positives impose a hard asymptotic ceiling on the alignment and capability of the model [cite: 22]. Mitigating this requires label-efficient recipes, such as overcredit contrastive losses and negative augmentation during the training of the PRM, to teach the reward model to be deeply skeptical of seemingly plausible logic [cite: 22].

Furthermore, structure search-based reasoning incurs substantial computational overhead. The sheer volume of simulations required to map out a Monte Carlo tree makes these models less suitable for real-time, low-latency applications [cite: 14]. Generating 10 to 100 times more tokens per query translates directly to higher API (Application Programming Interface) costs; running a maximum-compute reasoning model can cost thousands of dollars for a single, highly complex query [cite: 1, 5]. For context, OpenAI's o1-preview API costs $15 per 1 million input tokens and $60 per 1 million output tokens (which includes hidden reasoning tokens), compared to the standard GPT-4o's much lower cost of $2.50 for input and $10 for output per million tokens [cite: 25, 26].

This massive token generation deeply impacts user-facing latency. Instead of streaming responses instantly, reasoning models can introduce wait times ranging from 5 to 60+ seconds, and sometimes multiple minutes for a single complex query [cite: 9]. To manage this friction, the industry is rapidly adapting User Experience (UX) paradigms—such as displaying streaming "thinking tokens," collapsible reasoning trace dropdowns, or progressive status indicators—to maintain user engagement and provide transparency while the model deliberates in the background [cite: 9].

Hardware, Economic, and Regulatory Implications

The transition toward inference-time compute scaling is triggering secondary shockwaves across the AI industry, primarily in infrastructure provisioning and governmental regulation.

Economically, the ratio of training compute to inference compute is shifting dramatically [cite: 5]. As models are designed to think longer, inference clusters are becoming the primary bottleneck. Analysts project that global demand for inference compute will exceed training compute demand by an astonishing 118x by 2026 [cite: 5]. By 2030, inference operations could claim up to 75% of total AI compute, fundamentally driving a multi-trillion dollar shift in hardware investments [cite: 5]. Silicon manufacturers are adapting by optimizing their next-generation architectures (like the NVIDIA Blackwell series) explicitly for multi-agent and inference-heavy workloads [cite: 5].

More pressingly, test-time scaling poses an immediate crisis for AI safety legislation. Existing and proposed regulatory frameworks, such as the European Union's AI Act, heavily rely on training compute thresholds (e.g., triggering strict oversight for models trained with more than 10^25 FLOPs (Floating Point Operations Per Second)) to identify potentially dangerous frontier models [cite: 5].

Inference-time scaling renders this calculus dangerously obsolete. A smaller model—one that falls well below regulatory scrutiny during its pre-training phase—can be imbued with massive inference-time compute. By generating tens of thousands of internal thought tokens and utilizing aggressive MCTS, this unregulated model can achieve capabilities, and potential risks, that far exceed those of heavily regulated baseline models [cite: 5, 6]. Policymakers risk severely underestimating a model's real-world impact if safety evaluations do not account for dynamic, query-time compute allocation [cite: 5, 6]. A model that tests as perfectly safe under standard evaluation constraints might suddenly exhibit dangerous planning capabilities when granted prolonged "thinking time" after deployment [cite: 6].

Conclusion: The Future of Cognitive Architecture

The AI industry has moved beyond the brute-force era of merely stuffing more data into larger parameter spaces. We are witnessing the maturation of cognitive architectures. By scaling test-time compute, models are no longer reacting; they are deliberating.

The integration of Process Reward Models with search algorithms like MCTS represents the true dawn of System 2 artificial reasoning. While challenges remain—most notably the computational latency and the vulnerability of PRMs to reward hacking—the trajectory is clear. The future of artificial intelligence will not be defined solely by what a model already knows when it finishes training, but by how rigorously, adaptively, and deeply it can think when presented with a problem.

Sources & References