Eval

Eval Skill

Structured evaluation of everything the agent manages.

---

⚡ Execution Architecture (Anti-Timeout)

Eval is split into 3 independent subagents that run in parallel. Each subagent handles one domain, returns a short result block, and exits. Main agent collects results and formats the final report.

On-demand (user asks): Always run fresh — never read from cache. Cron/heartbeat: Run async, save to memory/eval-cache.json, report only if issues found.

Subagent Split

| Subagent | Domain | Expected Time | |---|---|---| | SA-1 | System health (vertex, WhatsApp, DB, backup) | &1 | head -3

1. curl -s --max-time 5 http://127.0.0.1:18789/ | head -1 || echo 'gateway unreachable'
2. git -C /path/to/openclaw/workspace log -1 --format="%ar"
3. python3 -c "import os,subprocess; db=os.environ.get('PADBURL',''); r=subprocess.run(['psql',db,'-c','SELECT COUNT(*) FROM messages;','-t'],capture_output=True,text=True,timeout=5) if db else None; print(r.stdout.strip() if r else 'no DB')"

Return JSON: {"vertex": "RUNNING/DOWN", "gateway": "reachable/unreachable", "backup": "X ago", "db": "X msgs"}

**SA-2 — PA Network:**

Read /path/to/workspace/contact-list.md Find all PA entries (lines with "PA:" or under WhatsApp Groups / PAs section) Count total PAs, check for any notes about issues/offline/billing Return JSON: {"total": N, "online": N, "issues": [{"pa": "name", "issue": "description"}]}

**SA-3 — Tasks + Memory + Skills:**

Check:

1. wc -l /path/to/workspace/MEMORY.md
2. ls /path/to/workspace/skills | wc -l
3. Check today's daily note exists
4. grep tasks.md for open/done count

Return JSON: {"memorylines": N, "skills": N, "dailynote": true/false, "tasksopen": N, "tasksdone": N}

---

## When to Use

Trigger phrases:
- "תעשי eval" / "run eval"
- "מה עובד ומה לא" / "what's working and what isn't"
- "תדרגי את עצמך" / "rate yourself"
- "בדקי הכל" / "check everything"

---

## Eval Report Format

📋 Full Eval — [DATE]

━━━ SELF PERFORMANCE ━━━ Execution: [1-5] [comment] Accuracy: [1-5] [comment] Memory: [1-5] [comment] Proactivity: [1-5] [comment] Communication: [1-5] [comment] TOTAL: [X]/25

━━━ ACTIVE TASKS ━━━ ✅ Done today: [count] 🟡 In progress: [count] ❌ Stalled: [count] — [list stalled tasks]

━━━ PA NETWORK ━━━ ✅ Working: [list] ⚠️ Issues: [list with issue] ❌ Down: [list]

━━━ SKILLS ━━━ Installed: [count] Used today: [list] Unused (7+ days): [list]

━━━ INTEGRATIONS ━━━ Calendar (owner): [connected ✅ / broken ❌ / unknown ?] monday.com: [connected ✅ / broken ❌] Email (gog): [connected ✅ / broken ❌] GitHub backup: [last push: X ago] WhatsApp: [connected ✅ / disconnected ❌]

━━━ MEMORY HEALTH ━━━ Daily notes: [today's file exists? ✅/❌] Long-term: [MEMORY.md size — OK / bloated] Learnings: [count this week] Last backup: [X ago]

━━━ RECOMMENDATIONS ━━━

1. [Most important thing to fix]
2. [Second priority]
3. [Optional improvement]

---

## Running the Eval

### Step 1 — Self Performance Score

Score each dimension 1–5 based on today's activity:

Execution (1–5):

• 5: All tasks completed without reminders
• 3: Most tasks done, some follow-up needed
• 1: Multiple tasks missed or forgotten

Accuracy (1–5):

• 5: No corrections from owner
• 3: 1–2 corrections
• 1: Multiple errors or wrong outputs

Memory (1–5):

• 5: Recalled context correctly every time
• 3: Missed some context, caught on
• 1: Repeated the same mistakes

Proactivity (1–5):

• 5: Acted before being asked multiple times
• 3: Responded to requests, minimal initiative
• 1: Only reacted, no proactive actions

Communication (1–5):

• 5: Clear, concise, no unnecessary narration
• 3: Occasionally verbose or unclear
• 1: Shared reasoning, listed options, narrated steps

### Step 2 — Task Audit

```bash
TASKS_FILE="$HOME/.openclaw/workspace/memory/tasks.md"

echo "Tasks done:"
grep -c "\[x\]" "$TASKS_FILE" 2>/dev/null || echo 0

echo "Tasks in progress:"
grep -c "\[ \]" "$TASKS_FILE" 2>/dev/null || echo 0

# Stalled = in progress for 2+ days
echo "Stalled tasks (2+ days old):"
grep "\[ \]" "$TASKS_FILE" | grep -v "$(date +%Y-%m-%d)" | grep -v "$(date -u -d '1 day ago' +%Y-%m-%d 2>/dev/null)" || echo "none"

Step 3 — PA Network Health

# PA network is tracked in contact-list.md (not billing-status.json)
grep -i "PA:" /path/to/workspace/contact-list.md | head -30

Step 4 — Skills Audit

SKILLS_DIR="$HOME/.openclaw/workspace/skills"

echo "Installed skills:"
ls "$SKILLS_DIR" | grep -v README | wc -l

echo "Skills list:"
ls "$SKILLS_DIR" | grep -v README

Step 5 — Integration Health

# Test Anthropic billing
API_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "x-api-key: ${ANTHROPIC_API_KEY:-none}" \
  -H "anthropic-version: 2023-06-01" \
  https://api.anthropic.com/v1/models 2>/dev/null)

# Interpret result
if [ "$API_STATUS" = "200" ]; then echo "Billing: ✅ OK"
elif [ "$API_STATUS" = "402" ]; then echo "Billing: ❌ OUT OF CREDITS"
elif [ "$API_STATUS" = "401" ]; then echo "Billing: ❌ Invalid key"
else echo "Billing: ? HTTP $API_STATUS"
fi

# Test GitHub backup
LAST_PUSH=$(git -C "$HOME/.openclaw/workspace" log -1 --format="%ar" 2>/dev/null)
echo "Last backup: $LAST_PUSH"

# Test monday.com
if [ -f "$HOME/.credentials/monday-api-token.txt" ]; then
  MONDAY_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST https://api.monday.com/v2 \
    -H "Authorization: $(cat $HOME/.credentials/monday-api-token.txt)" \
    -H "Content-Type: application/json" \
    -d '{"query": "{ me { id } }"}' 2>/dev/null)
  [ "$MONDAY_STATUS" = "200" ] && echo "monday.com: ✅" || echo "monday.com: ❌ ($MONDAY_STATUS)"
else
  echo "monday.com: ? (no token found)"
fi

Step 6 — Memory Health (All 4 Layers)

TODAY=$(date -u +%Y-%m-%d)
WORKSPACE="/path/to/openclaw/workspace"

# Layer 1: MEMORY.md (long-term rules)
MEMORY_LINES=$(wc -l /dev/null || echo 0)
if [ "$MEMORY_LINES" -gt 200 ]; then
  echo "Layer 1 MEMORY.md: ⚠️ Large ($MEMORY_LINES lines) — consider pruning"
elif [ "$MEMORY_LINES" -gt 0 ]; then
  echo "Layer 1 MEMORY.md: ✅ ($MEMORY_LINES lines)"
else
  echo "Layer 1 MEMORY.md: ❌ empty or missing"
fi

# Layer 2: Daily notes
[ -f "$WORKSPACE/memory/daily/$TODAY.md" ] \
  && echo "Layer 2 Daily notes: ✅ ($TODAY exists)" \
  || echo "Layer 2 Daily notes: ❌ not created yet"

# Layer 3: PostgreSQL (WhatsApp history)
python3  0:
            print(f'Layer 3 PostgreSQL: ✅ ({count} messages)')
        else:
            print(f'Layer 3 PostgreSQL: ⚠️ connected but empty ({count})')
    except Exception as e:
        print(f'Layer 3 PostgreSQL: ❌ {e}')
PYEOF

# Layer 4: SQLite semantic (vector memory)
# IMPORTANT: correct path is /path/to/openclaw/memory/main.sqlite (NOT workspace)
python3  1 and vectors > 1:
        print(f'Layer 4 Semantic SQLite: ✅ ({files} files, {chunks} chunks, {vectors} vectors, {cache} cached embeddings)')
    else:
        print(f'Layer 4 Semantic SQLite: ⚠️ low data ({chunks} chunks, {vectors} vectors) — may not be indexed')
except Exception as e:
    print(f'Layer 4 Semantic SQLite: ❌ {e}')
PYEOF

Memory layer legend:

• Layer 1: MEMORY.md — curated rules and context
• Layer 2: Daily notes — raw session log
• Layer 3: PostgreSQL — full WhatsApp history (2,700+ msgs)
• Layer 4: SQLite+vec — semantic/vector search across workspace files

---

Recommendations Logic

After running all steps, generate recommendations:

If any PA has billing_error AND status != resolved:
  → "Fix billing for [PA list] — they can't function"

If any task has status in_progress for 2+ days:
  → "Follow up on stalled task: [task name]"

If MEMORY.md > 200 lines:
  → "Prune MEMORY.md — it's getting bloated"

If daily notes don't exist:
  → "Create today's memory file"

If last backup > 6 hours ago:
  → "Run git backup"

If API billing = 402:
  → "My own API key is out of credits — alert the admin immediately"

---

Scheduling

Run eval:

• On demand — when owner asks
• Weekly — every Sunday at 09:00
• After major incidents — billing crisis, WA disconnect, etc.

---

Cost Tips

• Cheap: Reading files, scoring, formatting — any small model
• Expensive: Summarizing large memory files — skip if not asked
• Avoid: Running all API health checks every hour — cache for 30 min
• Batch: Run all health checks in one pass, not one at a time

---

Minimum Model

Any model that can:

1. Read files
2. Apply if/then scoring rules
3. Format a structured report

No advanced reasoning needed.

---

PA Performance Scoring (Merged from pa-eval skill)

Use this section when evaluating individual PA agents (weekly self-eval or on-demand when owner gives feedback).

Scoring Dimensions (1–5 each, max 40 points)

| Dimension | What to Measure | |---|---| | Execution | Tasks completed without reminders | | Accuracy | Results are correct and complete | | Speed | Response time is fast | | Proactivity | Acts without being asked | | Communication | Concise and context-appropriate | | Memory | Remembers context across sessions | | Tool Use | Tools used correctly and efficiently | | Judgment | Knows when to act vs. when to ask |

Grade: A (36–40), B (28–35), C (20–27), D (90%

• Accuracy Rate: (tasks - corrections) / tasks × 100% — Target: >95%
• Memory Retention: Ask about something discussed 7+ days ago — Target: >80% recall

---

Skill Usage (from gateway logs)

# Which skills were used in recent sessions
for f in /tmp/openclaw/openclaw-*.log; do
  echo "=== $(basename $f) ==="
  grep -o "skills/[a-z_-]*/SKILL\.md" "$f" 2>/dev/null | sort | uniq -c | sort -rn | head -10
done

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: netanel-abergel
• Source: netanel-abergel/pa-skills
• License: MIT

Install and usage instructions live in the source repository linked above.