Rl Innovator

RL Innovator

Purpose

Meta-skill that orchestrates rl-methodology to explore novel algorithm designs, identify limitations of existing approaches, and propose mathematically grounded improvements.

When to Use

Invoke this skill when you need to:

• Identify limitations of an existing RL algorithm and propose improvements
• Design a novel algorithm variant for a specific problem
• Conduct a systematic exploration of the algorithm design space
• Validate a proposed algorithm through theory and implementation
• Compare algorithm variants along multiple dimensions

Innovation Pipeline

Phase 1: Problem Analysis

Goal: Understand the limitation or opportunity

1. Identify the baseline algorithm and its properties using the Analysis Procedure in rl-methodology
2. Characterize the problem setting:

• What makes the problem hard? (large state space, continuous actions, sparse rewards, partial observability, non-stationarity)
• What are the baseline's failure modes in this setting?

1. Literature positioning: What has been tried before? What gaps remain?

Phase 2: Design Exploration

Goal: Generate candidate modifications

Use the Algorithm Design Procedure in rl-methodology to explore modifications along these dimensions:

Dimension 1: Objective Function

• Standard: maximize expected discounted return
• Alternatives: risk-sensitive (CVaR), constrained (Lagrangian), multi-objective (Pareto), entropy-regularized (MaxEnt)
• Innovation: Define a new objective that addresses the identified limitation

Dimension 2: Update Rule

• Standard: single-step TD, MC return, n-step return
• Alternatives: Retrace(lambda), V-trace, generalized advantage estimation (GAE)
• Innovation: New target construction, multi-timescale updates, auxiliary tasks

Dimension 3: Approximation Architecture

• Standard: MLP, linear
• Alternatives: attention, graph neural networks, recurrent, ensemble
• Innovation: Architecture that encodes problem structure (symmetry, locality, hierarchy)

Dimension 4: Exploration Strategy

• Standard: epsilon-greedy, Boltzmann
• Alternatives: UCB, posterior sampling, curiosity-driven (ICM, RND), count-based
• Innovation: Exploration that leverages problem structure

Dimension 5: Data Usage

• Standard: on-policy (discard after use), off-policy replay buffer
• Alternatives: prioritized replay, hindsight replay (HER), model-based data augmentation
• Innovation: New replay strategies, data weighting schemes

Phase 3: Theoretical Validation

Goal: Verify mathematical soundness of the proposed algorithm

Use the Convergence Proof Procedure in rl-methodology to:

1. Formulate as SA: Write the proposed update in Robbins-Monro or Dvoretzky form
2. Check convergence conditions:

• Learning rate conditions satisfied?
• Noise conditions met (unbiased or bounded bias)?
• Contraction property holds?

1. Identify failure modes:

• Deadly triad analysis
• Stability under function approximation
• Sensitivity to hyperparameters

1. Establish guarantees:

• What fixed point does it converge to?
• Error bounds relative to optimal?
• Sample complexity?

Phase 4: Implementation & Testing

Goal: Validate empirically

Use the tabular templates in rl-methodology for simple baselines, and standard PyTorch implementations of the classic deep-RL algorithms (DQN, A2C, PPO, DDPG, TD3, SAC) as starting scaffolds:

1. Implement the algorithm — start from a standard scaffold of the closest algorithm (e.g., modify the SAC update rule, extend the PPO loop) or build from scratch using shared components (MLP, ReplayBuffer, RolloutBuffer, GAE, polyak update)
2. Set up test environments:

• Book's 3×3 grid world (sanity check, use rl-methodology tabular templates)
• Classic control tasks (CartPole, MountainCar, Pendulum) via Gymnasium
• Continuous control (HalfCheetah, Hopper) for deep RL variants

1. Run experiments:

• Compare against baseline on same environment
• Measure: cumulative reward, convergence speed, final policy quality
• Ablation: test each modification independently

1. Analyze results:

• Does the theoretical advantage manifest empirically?
• Are there unexpected failure modes?
• How sensitive to hyperparameters?

Phase 5: Synthesis

Goal: Consolidate findings

1. Summarize the novel algorithm:

• Name, update rules, pseudocode
• Theoretical properties (convergence, error bounds)
• Empirical results

1. Compare to baselines:

• Table of algorithms vs. metrics
• Qualitative analysis of when the new algorithm helps

1. Identify next steps:

• Remaining limitations
• Follow-up experiments
• Potential extensions

Example Innovation Workflows

Workflow 1: "Improve Q-learning for Large State Spaces"

1. Analysis: Q-learning is tabular; doesn't scale. With FA, the deadly triad emerges.
2. Design: Combine Q-learning + target networks + experience replay + linear FA

→ This is essentially DQN (rediscovery validates the framework)

1. Theory: No convergence guarantee exists for DQN (deadly triad: FA + bootstrapping + off-policy). Target networks and experience replay are empirical stabilization heuristics. Extended Dvoretzky applies only to the tabular case.
2. Implement: standard DQN implementation
3. Test: Compare tabular Q-learning vs. DQN on grid worlds of increasing size

Workflow 2: "Design Variance-Reduced Policy Gradient"

1. Analysis: REINFORCE has high variance due to full-return Monte Carlo estimates
2. Design: Add baseline (variance reduction) + critic (bootstrap) + advantage normalization

→ Actor-Critic with advantage function

1. Theory: Baseline invariance (E[grad ln pi * b(S)] = 0) ensures unbiasedness. TD error delta_t is an unbiased estimate of advantage A(s,a).
2. Implement: A2C template
3. Test: Compare REINFORCE vs. A2C on convergence speed

Workflow 3: "Novel n-step Actor-Critic with Adaptive n"

1. Analysis: n-step returns trade off bias and variance. Fixed n is suboptimal.
2. Design: Adaptive n based on TD error magnitude. High |delta| → use smaller n (more bootstrap, less variance). Low |delta| → use larger n (less bias).
3. Theory: Show this is a valid stochastic approximation with bounded bias. The adaptive mechanism doesn't violate SA conditions if n is bounded.
4. Implement: Modified A2C with adaptive n-step returns
5. Test: Compare fixed-n vs. adaptive-n on environments with varying noise

Evaluation Criteria for Novel Algorithms

| Criterion | Weight | How to Assess | |-----------|--------|---------------| | Mathematical soundness | High | Convergence proof or clear conditions | | Empirical improvement | High | Better reward/convergence on benchmarks | | Simplicity | Medium | Minimal additional hyperparameters | | Generality | Medium | Works across problem settings | | Novelty | Medium | Not a trivial variant of existing work | | Interpretability | Low | Can explain why it works |

Anti-Patterns to Avoid

1. Complexity without justification: Don't add mechanisms that don't address a clear limitation
2. Theory-practice gap: Don't claim convergence guarantees that only hold under unrealistic conditions
3. Overfitting to benchmarks: Test on diverse environments, not just one
4. Ignoring baselines: Always compare against well-tuned standard algorithms
5. Hyperparameter sensitivity: An algorithm that only works for specific hyperparameters is fragile

Output Format

When proposing a novel algorithm:

1. Motivation: What limitation are we addressing?
2. Proposed Algorithm: Name, update rules, pseudocode
3. Theoretical Analysis: Convergence, error bounds, complexity
4. Experimental Design: Environments, baselines, metrics
5. Results: Empirical comparison (if implemented)
6. Discussion: When does this help? When does it fail?
7. Next Steps: Further improvements, open questions

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: chuongdlb
• Source: chuongdlb/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.