Reinforcement Learning

Reinforcement Learning - Learning from Interaction

למידת חיזוק: הסוכן לומד מניסיון ושגיאה.

Quick Start - Q-Learning (No Libraries)

import numpy as np

# Simple GridWorld
# Agent starts at (0,0), goal at (3,3), in 4x4 grid
grid_size = 4
n_states = grid_size * grid_size
n_actions = 4  # up, down, left, right

# Q-table: state × action → expected reward
Q = np.zeros((n_states, n_actions))

# Hyperparameters
learning_rate = 0.1
discount_factor = 0.99  # gamma
epsilon = 1.0           # exploration rate
epsilon_decay = 0.995
epsilon_min = 0.01
episodes = 1000

def state_to_pos(state):
    return state // grid_size, state % grid_size

def pos_to_state(row, col):
    return row * grid_size + col

def step(state, action):
    """Take action, return (next_state, reward, done)."""
    row, col = state_to_pos(state)
    if action == 0: row = max(0, row - 1)          # up
    elif action == 1: row = min(grid_size-1, row + 1)  # down
    elif action == 2: col = max(0, col - 1)          # left
    elif action == 3: col = min(grid_size-1, col + 1)  # right

    next_state = pos_to_state(row, col)
    done = (next_state == n_states - 1)  # Goal at last cell
    reward = 1.0 if done else -0.01      # Small penalty per step
    return next_state, reward, done

# Training loop
for episode in range(episodes):
    state = 0  # Start at (0,0)
    total_reward = 0

    for t in range(100):  # Max steps per episode
        # Epsilon-greedy action selection
        if np.random.random()  0 else 0
        return np.array([
            self.balance, self.shares, price, change
        ], dtype=np.float32)

    def step(self, action):
        price = self.prices[self.current_step]

        # Execute action
        if action == 1 and self.balance >= price:    # Buy
            self.shares += 1
            self.balance -= price
        elif action == 2 and self.shares > 0:         # Sell
            self.shares -= 1
            self.balance += price

        self.current_step += 1
        done = self.current_step >= len(self.prices) - 1

        # Reward: portfolio value change
        portfolio = self.balance + self.shares * price
        reward = (portfolio - self.initial_balance) / self.initial_balance

        return self._get_obs(), reward, done, False, {}

# Use with Stable-Baselines3
env = TradingEnv(prices=stock_prices)
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=50_000)

Pattern 6: Policy Gradient (REINFORCE)

import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions import Categorical
import gymnasium as gym

class PolicyNetwork(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, action_dim),
            nn.Softmax(dim=-1)
        )

    def forward(self, x):
        return self.network(x)

def reinforce(env_name="CartPole-v1", episodes=1000, gamma=0.99, lr=1e-3):
    env = gym.make(env_name)
    state_dim = env.observation_space.shape[0]
    action_dim = env.action_space.n

    policy = PolicyNetwork(state_dim, action_dim)
    optimizer = optim.Adam(policy.parameters(), lr=lr)

    for episode in range(episodes):
        state, _ = env.reset()
        log_probs = []
        rewards = []

        # Collect trajectory
        done = False
        while not done:
            state_t = torch.FloatTensor(state)
            probs = policy(state_t)
            dist = Categorical(probs)
            action = dist.sample()

            log_probs.append(dist.log_prob(action))
            state, reward, terminated, truncated, _ = env.step(action.item())
            rewards.append(reward)
            done = terminated or truncated

        # Compute discounted returns
        returns = []
        G = 0
        for r in reversed(rewards):
            G = r + gamma * G
            returns.insert(0, G)

        returns = torch.FloatTensor(returns)
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)

        # Policy gradient update
        # ∇J(θ) = E[∑ ∇log π(a|s) · G_t]
        loss = -torch.stack(log_probs) @ returns
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if episode % 50 == 0:
            print(f"Episode {episode}, Total Reward: {sum(rewards):.0f}")

reinforce()

Reference Navigation

For detailed content, see:

• RL Fundamentals: reference/rl_fundamentals.md - MDP, Bellman equation, value/policy
• Q-Learning & DQN: reference/q_learning_dqn.md - Tabular Q, DQN tricks, Double DQN
• Policy Gradient: reference/policy_gradient.md - REINFORCE, Actor-Critic, PPO math
• Gym Environments: reference/gym_environments.md - Custom envs, wrappers, benchmarks

Common Mistakes to Avoid

1. No Experience Replay in DQN

# WRONG: Train on single transitions (correlated, unstable)
loss = compute_loss(state, action, reward, next_state)

# CORRECT: Sample random batch from replay buffer
batch = buffer.sample(batch_size=64)  # Breaks correlation!
loss = compute_batch_loss(batch)

2. No Target Network

# WRONG: Q-network chases its own tail
target = reward + gamma * q_network(next_state).max()  # Moving target!

# CORRECT: Stable target from frozen network
target = reward + gamma * target_network(next_state).max()
# Update target_network periodically (every N episodes)

3. Reward Shaping Gone Wrong

# WRONG: Dense reward that creates shortcuts
reward = -distance_to_goal  # Agent might find exploit

# BETTER: Sparse + well-defined
reward = 1.0 if reached_goal else 0.0
# Or: shaped reward that preserves optimal policy

Teaching Mode

RL vs Supervised Learning

Supervised Learning:          Reinforcement Learning:
┌──────────┐                 ┌──────────┐
│ Dataset  │ = Fixed         │ Agent    │ = Learns by doing
│ (X, y)   │   answers       │          │
└────┬─────┘                 └────┬─────┘
     │                            │
     ▼                            ▼
  Learn:                      Explore:
  "input → output"            "try action → see result"
                              "good result → do more"
                              "bad result → do less"

Key difference:
- Supervised: Teacher gives correct answers
- RL: Agent discovers good actions through trial and error
- RL: Reward may be delayed (not immediate feedback)

Exploration vs Exploitation

Restaurant Dilemma:
├── Exploitation: Go to your favorite restaurant (known good)
├── Exploration: Try a new restaurant (might be better... or worse)
└── Balance: Mostly exploit, sometimes explore

Epsilon-Greedy:
  With probability ε:   Random action  (explore)
  With probability 1-ε: Best known action (exploit)

  ε starts high (1.0) → lots of exploration
  ε decays over time   → more exploitation
  ε minimum (0.01)     → always a little exploration

Cross-References

• Neural networks: ../deep-learning-core/SKILL.md - NN architecture for policy/value nets
• PyTorch: ../pytorch-mastery/SKILL.md - Training loops, GPU optimization
• MLOps: ../mlops-experiment/SKILL.md - Tracking RL experiments
• ML fundamentals: ../ml-fundamentals/SKILL.md - Evaluation metrics

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: levy-n
• Source: levy-n/claude-useful-skills
• License: MIT

Install and usage instructions live in the source repository linked above.