# Simpo Loss Function

> Implement SimPO loss with length-normalized rewards and target margin.

- **Type:** Skill
- **Install:** `agentstack add skill-cxcscmu-skilllearnbench-simpo-loss-function`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [cxcscmu](https://agentstack.voostack.com/s/cxcscmu)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [cxcscmu](https://github.com/cxcscmu)
- **Source:** https://github.com/cxcscmu/SkillLearnBench/tree/main/skills/b1-one-shot-claude-haiku-4-5/nlp-paper-reproduction/simpo-loss-function
- **Website:** https://cxcscmu.github.io/SkillLearnBench

## Install

```sh
agentstack add skill-cxcscmu-skilllearnbench-simpo-loss-function
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# SimPO Loss Function Implementation

## Overview
SimPO (Simple Preference Optimization) implements a preference optimization objective that uses length-normalized average log probability as an implicit reward, with a target reward margin component.

## Key Formula

```
L_SimPO(πθ) = -E_(x,yw,yl)~D log σ(β/|yw| log πθ(yw|x) - β/|yl| log πθ(yl|x) - γ)
```

## Components

### 1. Length-Normalized Reward
- **Formula**: `r_SimPO(x, y) = β/|y| * log πθ(y|x)`
- **Purpose**: Average log probability per token, prevents length bias
- **Why**: Aligns training with generation metric (which uses average log likelihood for beam search)

### 2. Bradley-Terry Objective
- **Formula**: `p(yw ≻ yl | x) = σ(r(x, yw) - r(x, yl) - γ)`
- **Purpose**: Probabilistic ranking between winning and losing responses
- **Function**: σ is sigmoid function

### 3. Target Reward Margin (γ)
- **Purpose**: Ensure reward difference exceeds a target threshold
- **Effect**: Improves generalization by enforcing margin between classes
- **Typical range**: 0.5 to 1.5

## Implementation Details

### Computing Log Probabilities
```python
# log_probs shape: (batch_size, seq_len)
# Sum across sequence dimension to get total log probability
log_prob_sum = log_probs.sum(dim=1)  # (batch_size,)

# Divide by sequence length for normalization
seq_lengths = (input_ids != pad_token_id).sum(dim=1)  # (batch_size,)
avg_log_prob = log_prob_sum / seq_lengths.float()  # (batch_size,)
```

### Computing Reward Differences
```python
# Batch structure: pairs of (winning, losing) responses
batch_size = avg_log_probs.shape[0]
winning_rewards = avg_log_probs[:batch_size//2]
losing_rewards = avg_log_probs[batch_size//2:]

# Reward difference with margin
reward_diff = beta * winning_rewards - beta * losing_rewards - gamma
```

### Computing Loss
```python
# Bradley-Terry with sigmoid
import torch.nn.functional as F

sigmoid_term = torch.sigmoid(reward_diff)
loss = -torch.log(sigmoid_term).mean()
```

## Common Pitfalls

1. **Not using length normalization**: Creates bias toward longer sequences
2. **Wrong batch structure**: Ensure paired winning/losing responses
3. **Missing average in log probabilities**: Use sum/length, not just sum
4. **Gradient flow**: Ensure no detach() breaks gradients to model

## Hyperparameters
- **β**: Temperature parameter, typically 2.0-2.5
- **γ**: Target margin, typically 0.3-1.6, depends on setting
- **learning_rate**: Usually small, 1e-6 to 5e-7

## References
- SimPO Paper: Section 2.3 "The SimPO Objective"
- Length normalization: Equation (3) in paper
- Gradient analysis: Appendix F

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [cxcscmu](https://github.com/cxcscmu)
- **Source:** [cxcscmu/SkillLearnBench](https://github.com/cxcscmu/SkillLearnBench)
- **License:** MIT
- **Homepage:** https://cxcscmu.github.io/SkillLearnBench

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-cxcscmu-skilllearnbench-simpo-loss-function
- Seller: https://agentstack.voostack.com/s/cxcscmu
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.