Qa

QA — Feature Regression Testing

Run automated and interactive regression tests for a San feature based on its doc in docs/features/.

Arguments

• ` — Feature number (e.g. 01, 6) or filename stem (e.g. 6-hooks, cli-startup`). Partial matches are fine.
• --interactive — Also run the interactive tmux tests (default: automated only).
• --pane — For interactive tests, split a pane in the current tmux window instead of creating a separate session. This lets the user watch the test live side-by-side.

If only a number is given, zero-pad and match against filenames (e.g. 1 → 1-cli-startup.md).

Workflow

1. Resolve the feature doc

docs/features/-.md

Read the file. Extract:

• Automated Tests section — the go test commands and known test cases.
• Interactive Tests (tmux) section — the step-by-step tmux script.

If the feature cannot be found, list all available features and ask the user to pick one.

2. Build the binary

Always build first so tests run against the latest code:

make build

If the build fails, stop and report the error — no point running tests against stale code.

3. Run automated tests

Execute every go test command listed in the feature doc's Automated Tests section. Capture stdout/stderr and exit codes. Each command is a separate test group.

Record results as:

• PASS — exit code 0
• FAIL — exit code non-zero; include the failure output

4. Run interactive tests (only with --interactive or --pane)

Skip this phase unless the user explicitly opts in.

Pane mode (--pane)

Split a pane in the current tmux window for the test:

# Detect current session and window
TMUX_TARGET=$(tmux display-message -p '#{session_name}:#{window_index}')
# Split horizontally, 50% width
tmux split-window -h -t "$TMUX_TARGET" -l '50%'
# The new pane is at $TMUX_TARGET.{next_pane_index}

Run all interactive test steps in that pane via tmux send-keys. Use tmux capture-pane -p to read the pane output after each step and verify the Expected comments from the doc.

Session mode (default when --interactive without --pane)

Create a dedicated tmux session:

tmux new-session -d -s qa_ -x 220 -y 60

Run the interactive test steps there. Kill the session when done.

Interpreting interactive results

For each test step:

1. Send the command via tmux send-keys.
2. Wait the prescribed sleep duration (or a reasonable default).
3. Capture the pane with tmux capture-pane -t -p.
4. Check the captured output against the Expected comment.
5. Record PASS/FAIL with the captured output as evidence.

After all steps, clean up (kill session/pane, remove temp files) as specified in the doc's cleanup section.

5. Generate the report

Print a summary table:

## QA Report: Feature  — 

### Automated Tests
| Test Group               | Result |
|--------------------------|--------|
| go test ./internal/x/... | PASS   |
| go test ./tests/int/...  | FAIL   |

### Interactive Tests
| Step                         | Result | Notes                    |
|------------------------------|--------|--------------------------|
| Test 1: Basic TUI startup    | PASS   | TUI with input box shown |
| Test 2: Print mode           | FAIL   | No output after 15s      |

### Summary
Automated: 2/3 passed
Interactive: 3/4 passed
Overall: PARTIAL PASS

Use PASS, FAIL, or SKIP for each item. Include failure details inline so the user can act on them immediately.

Important notes

• The binary path is ./bin/san (built by make build), not the installed san.
• Interactive tests that require an LLM response need a configured provider. If no provider is set, mark those steps as SKIP with a note.
• For interactive tests, use Ctrl+C (C-c) to exit san, not q (which gets interpreted as user input).
• Always clean up temp files and tmux sessions/panes after testing, even on failure.
• If a test step times out (no expected output after the sleep), mark it FAIL and move on — don't hang.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: genai-io
• Source: genai-io/san
• License: Apache-2.0
• Homepage: https://genai-io.github.io/san/

Install and usage instructions live in the source repository linked above.