Every time ChatGPT gives you a useful answer, it got there the same way you learned not to touch a hot stove. Try. Get burned. Try differently. There is no manual, no labeled dataset, just consequences. Kevin Murphy just published 252 pages mapping exactly how this works—from Atari games to the LLM on your screen—and the through-line is simpler than the hype suggests.
// WHAT_IS_THIS
Reinforcement Learning is learning from punishment and reward, not from examples. A supervised model is a student with an answer key. An RL agent is a dog with a clicker. Or a Casablanca delivery driver who discovers that taking the Corniche at 8am is suicide, not because someone told him, but because he lost three hours and missed his bonus.
Murphy's paper isn't breaking new ground with a single algorithm. It's doing something more useful. It connects the dots between the Q-learning that DeepMind used to beat Atari in 2013 and the RLHF (Reinforcement Learning from Human Feedback) that makes modern LLMs useful instead of toxic. The same mathematical skeleton powers both. If you understand one loop—the agent, the environment, the state, the reward—you understand the machinery behind both game-playing bots and conversational AI.
Why this matters for you: RL is moving out of research labs and into business operations. Inventory management. Energy grids. Delivery routing. The paper shows you exactly which flavor of RL fits which business pain.
// THE_CORE_IDEA
Layer 1 — What's actually happening?
Murphy frames RL as three different answers to the same question: "How do I get better at this sequence of decisions?"
Value-based methods learn to score situations. They build an internal map of "good" and "bad" states—like a driver learning which intersections are always jammed. Policy-based methods skip the map and learn muscle memory directly—"when I see this traffic pattern, I turn left here." Model-based methods are the planners: they simulate tomorrow before acting today, predicting "if I send the truck now, I'll hit the port closure."
Layer 2 — The mechanism
The Policy Gradient Theorem — the mathematical proof that you can improve a policy by following the gradient of expected reward. The key insight is the likelihood ratio estimator — a method that lets you update your strategy using only the outcomes you observe, without needing to know how the environment works internally. The update looks like this: you increase the probability of actions that led to good outcomes, scaled by how good they were relative to your baseline — this scaling factor is the advantage function (A(s,a) = Q(s,a) - V(s), measuring whether an action was better than your usual choice at that state).
For LLMs, Murphy highlights that we now use variants like GRPO (Group Relative Policy Optimization). Instead of training a separate critic network—which is computationally brutal at LLM scale—you generate a group of answers, normalize their rewards against each other, and update directly. This is how DeepSeek-R1 was trained. It's the same policy gradient logic, stripped down for industrial-scale deployment.
// WHY_THIS_MATTERS_HERE
Let's get specific. You run a logistics company in Tangier. Your trucks deliver to the médina, the port, and the industrial zone. Traffic is chaos, parking is nonexistent, and your drivers learn local tricks—like which guard at Gate 3 takes tea money versus which one enforces the rules.
A model-free RL system (Q-learning, PPO) would need to ruin hundreds of deliveries before it learns these unwritten rules. Expensive. A model-based system—covered in Chapter 4 of Murphy's survey—could simulate Casablanca traffic first, learn the dynamics, then deploy. The catch? You need a decent simulator. For most African SMEs, that's 2–3 years out. But the value-based methods? You can deploy those now if you have historical route data.
Or consider agritech. Drip irrigation in the Sahel. An RL agent controls water valves based on soil moisture. But sensors drift, seasons change—this is a non-stationary environment. Murphy covers this in his section on regret minimization—algorithms that adapt when the world changes under them. Critical here, where climate volatility is the baseline, not the edge case.
The honest takeaway: The math is ready. The compute is cheapening. The gap is data cleanliness and simulation infrastructure. Most local businesses should start with off-policy value methods using historical data—what Murphy calls offline RL—before trying anything that needs real-time exploration on live customers.
// THE_TECHNICAL_BREAKDOWN
Murphy structures the field into clean buckets that map to implementation choices.
Value-based RL (Chapter 2) covers Temporal Difference learning—updating your guess about a state based on the next state's value. The Bellman equation is the heart here: V(s) = R(s) + γE[V(s')]. The deadly triad—function approximation plus bootstrapping plus off-policy learning—can make your neural network diverge instead of converge. This is why Stable-Baselines3 exists; most people shouldn't implement DQN from scratch.
[CONCEPT] The Deadly Triad
"When you combine three things—using a neural network to approximate values (function approximation), learning from your own past experiences that don't match your current strategy (off-policy), and updating your guess based on your next guess (bootstrapping)—your system can spiral into instability. Like trying to balance on a ball while the ball is rolling downhill."
Policy-based methods (Chapter 3) moved us to continuous control. PPO—Proximal Policy Optimization—clips the policy update to prevent wild swings. Murphy notes the Wasserstein Policy Optimization work from 2025, which treats policy updates as gradient flows in probability space, offering better stability than vanilla policy gradients.
The LLM section (Chapter 6) is where research is hottest right now. RLHF uses a reward model—trained on human preference pairs—to guide the LLM. But the paper also covers RLVR (RL with Verifiable Rewards) for math and coding, where you don't need human preferences; the compiler or equation solver gives you a binary 0/1 reward. This is how AlphaProof works, and it's immediately applicable to local fintech validation pipelines.
[CONCEPT] Model-Based RL
"Instead of learning from real failures, you learn from imagined ones. You build a world model—a neural network that predicts the next state and reward—and you plan inside your head before acting. Like a chess player thinking three moves ahead, or a dispatch manager simulating tomorrow's routes before the trucks leave the depot."
Limitations? Murphy is clear: Sample inefficiency is brutal. AlphaGo trained on decades of game simulation. Real-world robots need millions of dollars of compute. And partial observability—when you can't see the full state of the market or the traffic—remains largely unsolved outside specialized applications like Cicero (the Diplomacy-playing agent).
// THE_REAL_QUESTION
The algorithms in Murphy's survey are public. The PyTorch implementations are on GitHub. The data—your delivery logs, your sensor readings, your transaction histories—is sitting on local servers right now. So why are African AI deployments still mostly using imported models trained on Californian datasets?
The real question isn't whether we can build these systems—we clearly can. It's whether we're going to train agents that understand Ramadan logistics, seasonal agricultural patterns, and local bargaining behavior, or whether we'll just pay API fees to foreign platforms that don't know the difference between Casablanca and California.