Reinforcement Learning

Learning from consequences

No labels, no answer key — just actions and their consequences. An agent observes the environment's state, takes an action, receives a reward and a new state, and must learn a policy (state → action strategy) maximizing cumulative reward. It's how animals learn, how you learned to ride a bicycle, and the third paradigm promised back in the ML path.

$\text{state } s_t \xrightarrow{\text{action } a_t} \text{reward } r_t,\; \text{state } s_{t+1}$

Two devilish features distinguish RL from supervised learning:

• Delayed credit. The winning chess move gets its reward 30 moves later. Which of your 60 actions deserve credit? (The credit assignment problem.)
• You generate your own data. The agent only learns about actions it tries — leading to the field's signature dilemma.

Exploration vs. exploitation

Your favorite restaurant is good (exploit), but the unknown one across the street might be better (explore). Explore too little → stuck on mediocre habits; too much → never cash in. The simplest recipe, $\epsilon$-greedy — act greedily, but with probability $\epsilon$ try something random, decaying $\epsilon$ over time — already captures the essential trade-off, one with no free lunch and deep mathematical roots (bandit theory, used daily in ad serving and A/B testing).

Value learning: Q-learning

Attach a number $Q(s, a)$ to each state-action pair: expected total future reward if I take action $a$ here, then act well afterwards. Discount future rewards by $\gamma \in (0,1)$ per step (a bird in the hand). Values must be self-consistent — today's value equals today's reward plus the discounted value of the best tomorrow (the Bellman equation) — and Q-learning turns that consistency into an update rule:

$Q(s,a) \leftarrow Q(s,a) + \alpha \Big[ r + \gamma \max_{a'} Q(s', a') - Q(s,a) \Big]$

The bracket is the temporal-difference error — surprise, in the same sense dopamine neurons encode it (one of AI's loveliest neuroscience parallels). Iterate enough and Q converges; the policy is then "pick the highest-Q action".

Deep RL: values become networks

Tables of Q-values die with large state spaces (there are more Go positions than atoms in the universe). Replace the table with a neural network. DQN (2013) learned 50 Atari games from raw pixels, sparking the deep-RL era; AlphaGo/AlphaZero combined policy/value networks with tree search and self-play to surpass all human game knowledge from scratch. Robotics learns locomotion and manipulation in simulation (then crosses the sim-to-real gap); RL tunes datacenter cooling, chip layouts, and recommendation systems.

And a twist you already use: RLHF — reinforcement learning from human feedback. Human preferences train a reward model, which RL then optimizes the LLM against; it's a key step that turned raw next-token predictors into helpful assistants. Beware the field's classic failure, reward hacking: agents maximize the metric, not your intent (a boat-racing agent famously learned to spin in circles collecting bonus targets instead of finishing). Specifying rewards that capture what we actually want is a microcosm of AI alignment — next module's subject.

Key takeaways

• RL = policies from rewards; delayed credit and self-generated data make it uniquely tricky.
• Bellman consistency → Q-learning; deep networks scale it to pixels and Go.
• RLHF aligns LLMs with RL machinery; reward hacking previews the alignment problem at large.