Agentsop Output Format By Model

Output-Format-by-Model — 让格式服务于任务，而不是反过来

> One-liner: Different output formats carry different cognitive load for the model. Code-in-JSON is the canonical proof: the same model writes worse code when wrapped in a JSON tool-call than when emitted as plain text + diff. The reverse failure (asking for prose when you need a typed object) is just as common. Pick format per (task × consumer), not by reflex.

---

1. 何时激活 (When to activate)

Activate this skill before committing to an output schema in any of these situations:

| Trigger | Signal | |---|---| | Designing a coder-agent | "should the model return a apply_patch tool call or plain-text diff?" | | Adding a tool to an existing agent | "tool input has a code / query / sql / regex field — should I nest it in JSON or leave it as a string?" | | Building extraction / classification | "should I use dspy.Predict typed fields, Pydantic + response_format=json_schema, or just markdown?" | | Wiring an evaluator | "the metric needs a number — but the model also has to reason to produce it" | | Migrating a working prompt to "structured outputs" | someone said "let's make it safer with JSON schema" | | Tool-call harness adds latency / errors | repeated json.JSONDecodeError, escaping bugs, truncated outputs |

Anti-triggers (skip this skill):

• The format is fixed by an external API (e.g. you must return OpenAI function-call JSON — no choice).
• One-shot exploratory prompting where no consumer parses the output yet.
• The task is itself about JSON (e.g. "fix this malformed JSON") — see §6.

---

2. 核心心智模型 (Core mental model)

2.1 Three-layer claim

            ┌──────────────────────────────────────────────────┐
            │ FORMAT FOLLOWS FUNCTION                          │
            │                                                  │
            │  Some formats add cognitive load to the model    │
            │  and measurably degrade quality on the           │
            │  *content* the format is supposed to wrap.       │
            └──────────────────────────────────────────────────┘
                         ▲                          ▲
                         │                          │
         What's being consumed?       Who consumes it?
         (code? prose? entities?      (human reader? parser?
          number? action selection?)   downstream LM? compiler?)
                         │                          │
                         └──────────┬───────────────┘
                                    ▼
                       FORMAT SELECTION
                  (text+diff | markdown | JSON | tool_use
                   | grammar-constrained | typed field)

Aider's code-in-json benchmark is the load-bearing empirical anchor:

• GPT-4 Turbo refactor benchmark with unified diff: 61% pass.
• Same model with JSON-wrapped code: 20% pass.
• That is a 3× swing from format alone, not from model choice or prompt rewording. [aider.chat/2023/12/21/unified-diffs.html, aider.chat/2024/08/14/code-in-json.html]
• The pattern holds across every major model tested in 2024 (Claude Sonnet, DeepSeek Coder, GPT-4o). Even with OpenAI strict-mode JSON validity enforced, **the code inside the JSON degraded** — more SyntaxError / IndentationError. Sonnet kept syntax clean but still scored lower overall. [aider.chat/2024/08/14/code-in-json.html]

Academic generalization, same year:

• "Let Me Speak Freely?" (arXiv 2408.02442) — format-restricted reasoning drops 10–15% on math and complex analysis vs free-form-then-convert. [arxiv.org/abs/2408.02442]
• StructEval (arXiv 2505.20139) — even o1-mini only scores 75.58 average on structural-output generation. [arxiv.org/html/2505.20139v1]

2.2 The two reflexes to unlearn

| Reflex | When it's wrong | |---|---| | "Structured output is always safer." | False for code, multi-step reasoning, free-form prose. Strictness ≠ quality of contents. | | "Markdown is only for humans." | False — markdown is also the highest-fidelity wire format for many LM-to-LM hand-offs (Aider uses it; DSPy's default chat adaptor uses field-marked markdown over JSON for many signatures). |

2.3 Mental model: format as a tax on the generation surface

Every formatting requirement consumes some of the model's attention budget. The tax is:

• Low for formats the model has seen millions of times in pretraining (markdown, unified diff, Python).
• Medium for JSON containing simple scalar fields (entity extraction, dates, booleans).
• High for JSON containing code, multi-line strings with escapes, or deeply nested reasoning. Each \n becomes \\n; each quote becomes \"; the model has to track this while also solving the actual problem.

⇒ Heuristic: the more semantically dense the content, the cheaper the format must be.

---

3. SOP 工作流 (Decision workflow)

Three questions, in order:

Step 1 — What's being consumed?

| Content type | Default format | Why | |---|---|---| | Source code edits | plain text + diff format (SEARCH/REPLACE, unified diff, or str_replace tool with code as a single string field) | Aider 20%→61% on GPT-4 Turbo; same direction across all models. [aider.chat/2024/08/14/code-in-json.html] | | Source code (full file rewrite) | plain text or markdown fenced block | Same reason. JSON-wrap adds escaping tax. | | Structured data extraction (entities, dates, IDs, classifications) | JSON / Pydantic / typed OutputField | Schema helps here — fields are the task. | | Action selection (which tool to call) | JSON tooluse | Tool name + scalar args. The decision is structured by definition. | | Action body (SQL, regex, code, file contents) | single string field inside tooluse — do not sub-structure | Same code-in-JSON penalty applies to any code-shaped payload. | | Reasoning / intermediate steps | markdown or Python-style scratchpad | CoT in JSON measurably degrades; see "Let Me Speak Freely?" [arxiv.org/abs/2408.02442] | | Numeric answer with reasoning | markdown reasoning + final answer in fenced block or final-line convention | Don't force the reasoning into a JSON reasoning field — it shortens and stiffens. | | Free-form prose (summary, explanation, customer reply) | markdown | Native to instruction-tuned models. | | Mixed (e.g., extract entities AND rewrite the document) | split into two calls or two passes — see §5 Case B | One format can't serve two contents well. |

Step 2 — Who consumes the output?

| Consumer | Constraint | Implication | |---|---|---| | Human in a chat UI | Render-friendly | Markdown wins. JSON is hostile. | | json.loads / Pydantic parser | Must be valid | JSON with schema, or a known-safe envelope (...) with markdown body. | | Downstream LM (LM-to-LM pipeline) | Reads what was written | Markdown is more robust than JSON when content includes code/math; the next LM ingests it natively. | | Compiler / interpreter (PoT, code-exec sandbox) | Must be a valid program | Output as code in a fenced block, not as a JSON program field. | | Tool dispatcher (function calling) | Needs tool name + args | JSON tooluse with scalar args; multi-line bodies go in a single string field. | | Diff applier (Aider, git apply, strreplace) | Must apply cleanly | Format dictated by the applier: SEARCH/REPLACE for Aider diff, unified diff for git, str_replace for Anthropic editor tool. |

Step 3 — Which model am I targeting?

Format support is model-specific. Aider maintains a per-model edit-format default precisely because of this. [aider.chat/docs/more/edit-formats.html]

| Model family | Best code-edit format | Notes | |---|---|---| | GPT-4 Turbo / GPT-4o | udiff or SEARCH/REPLACE diff | 20%→61% with udiff on refactor. [aider.chat/2023/12/21/unified-diffs.html] | | Claude 3.5/3.7 Sonnet | SEARCH/REPLACE diff | Tendency to write too much — instruct minimal blocks. [aider.chat/2024/07/01/sonnet-not-lazy.html] | | Gemini family | diff-fenced | Path inside the fence. [aider.chat/docs/more/edit-formats.html] | | GPT-4.1 / OpenAI patch tool | patch protocol | OpenAI-specific, multi-action robust. | | Weak / local models (Llama-3-8B class, GPT-3.5) | whole file rewrite | Diff parsing failures dominate; whole-file is dumb-but-stable. | | Reasoning models (o1, o3) as architect | architect mode: reasoner emits prose plan → editor model emits diff | o1-preview alone: 79.7%; o1-preview + Sonnet editor: 82.7%. [aider.chat/2024/09/26/architect.html] |

Combined decision tree:

                    What's being emitted?
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
      code              structured           free-form
      edits              data fields          prose/reasoning
        │                   │                   │
        ▼                   ▼                   ▼
   diff/SEARCH-       JSON / Pydantic /    markdown
   REPLACE/           tool_use scalar
   str_replace,       args
   code as string
        │                   │                   │
        ▼                   ▼                   ▼
   pick per-model   wrap any code-          do NOT
   edit format      shaped payload in        force into
   (Aider table)    a single string field    JSON schema

---

4. 操作模型 (Task → format pairs)

Ten concrete operations. Format choice is not arbitrary — each citation is the empirical anchor.

| # | Task | Recommended format | Rationale / Evidence | |---|---|---|---| | 1 | "Edit auth.py to use JWT" (coder-agent core loop) | Plain-text SEARCH/REPLACE diff in markdown fence | Aider's measured 3× on GPT-4 Turbo, generalizes across models. [aider.chat/2024/08/14/code-in-json.html] | | 2 | "Apply this patch to a file via Claude's text-editor tool" | Anthropic str_replace tool — oldstr / newstr as string fields, no JSON sub-structure inside the code | Matches Anthropic's published tool-design guidance: fields should be high-signal scalars, not low-level identifiers. [docs.anthropic.com text-editor-tool] | | 3 | "Extract invoice fields (vendor, amount, date, line items)" | JSON / Pydantic / DSPy typed OutputField | Schema is the task. Structured-output benchmarks show high accuracy here. [arxiv.org/html/2505.20139v1] | | 4 | "Classify ticket priority (P0/P1/P2/P3)" | Single-token output or JSON {"priority": "..."} | Trivial; structure helps determinism. | | 5 | "Answer a math word problem" | Markdown reasoning + final answer in \boxed{} or fenced final line | "Let Me Speak Freely?" — 10–15% degradation when locked into JSON reasoning. [arxiv.org/abs/2408.02442] | | 6 | "Decide which tool to call next" | JSON tool_use with tool name + scalar args | Action selection is intrinsically structured. | | 7 | "Generate the SQL query for that tool" | Tool args contain {"sql": "SELECT ..."} — SQL as a raw single string, no further JSON sub-structure | Same family as code-in-JSON; SQL is code-shaped. | | 8 | "Write a customer-support reply" | Markdown | Native to instruction tuning; JSON-wrap costs nothing useful. | | 9 | "Summarize a paper and return key claims as a list" | Markdown prose + a fenced claims: YAML or JSON block at the end (two-zone output) | Best of both: prose flows freely, downstream parser reads the trailing block. | | 10 | "Rewrite a long document AND extract entities" | Two passes: pass 1 rewrite (markdown), pass 2 extract (JSON, fed the pass-1 output) | One call cannot serve both contents at peak quality. |

---

5. 困境决策案例 (Dilemma cases / worked examples)

Case A — "I'm writing an extractor for invoice fields. Plain text or JSON?"

Trigger: New extraction pipeline. Fields = vendor_name, total_amount, invoice_date, line_items[].

Constraints:

• Output is consumed by a downstream Python parser.
• Some line items contain free-text descriptions with newlines and quotes.
• Pydantic validation is desired.

Decision steps:

1. The content is structured data, not code. JSON tax is low here — fields are short scalars or short strings.
2. Use Pydantic + OpenAI response_format=json_schema or DSPy typed OutputFields. Field names carry the semantic load (DSPy's "Signatures carry semantic load" principle [dspy.ai/learn/programming/signatures]).
3. For line_items[].description containing free-text: still inside JSON — descriptions are prose data, not code. Escaping cost is real but bounded.
4. Watch for: extraction-then-rewrite mission creep. If a future spec says "also rewrite the invoice text in cleaner language" → switch to Case B's two-pass approach.

Outcome: JSON wins. Schema validation catches missing fields cheaply; no measurable degradation expected on this content shape.

Case B — "Same task but the extractor also rewrites the invoice text"

Trigger: PM adds requirement — "and produce a cleaned-up version of the invoice document body."

Constraints:

• "Cleaned text" is multi-paragraph prose with embedded numbers, possibly tables.
• Putting cleaned_text: "..." inside the same JSON forces multi-paragraph escaping and competes with the extraction reasoning for attention.

Decision steps:

1. Recognize this as the mixed-content trap. One LM call cannot serve both contents at peak.
2. Two solutions, choose by latency budget:

• Two-pass (preferred): Call 1 returns {vendor, amount, date, line_items} as JSON. Call 2 receives the original doc + extracted JSON, returns markdown rewrite.
• Two-zone single call: Prompt requests markdown rewrite first, then ===EXTRACTION=== separator, then a fenced JSON block. Parser splits on the separator. Cheaper but more fragile.

1. Do not nest the markdown rewrite as a JSON string field. The model will compress it and lose structure.

Outcome: Plain text + JSON split, not unified JSON. Format follows content, even within one logical task.

---

Case C — "Tool call decision, but the tool's input is SQL"

Trigger: Building a data-analyst agent. One tool is run_query(sql: str).

Constraints:

• The agent must decide which tool to call (structured choice).
• The agent must write SQL (code-shaped content).
• Provider is OpenAI / Anthropic tool_use.

Decision steps:

1. The tool-selection layer is intrinsically structured — use JSON tool_use. The model picks run_query vs read_file vs list_tables.
2. The tool body is code. Per Aider's findings, code in JSON degrades. Mitigations:

• Make sql a single top-level string field. Don't sub-structure it ({from: ..., where: ...}). Let the model write idiomatic SQL.
• Keep other args as scalars (e.g. dry_run: bool).
• Expect some quality hit vs returning SQL as plain text; budget for retry/repair.

1. If quality is unacceptable: switch the architecture. Generate SQL in a separate plain-text LM call (markdown fenced ```sql block), then a thin downstream agent wraps it into the tool call. This is the Aider "architect" pattern transplanted to data tools. [aider.chat/2024/09/26/architect.html]

Outcome: JSON for the decision; single string for the SQL body; consider a two-step generate-then-dispatch if quality matters.

---

Case D — "Reviewer says 'use structured output everywhere for safety'"

Trigger: Eng-org-wide push to add response_format=json_schema to every LM call.

Constraints:

• The org has been bitten by json.loads failures.
• Some calls produce code; some produce prose; some produce typed objects.
• The reviewer is conflating "schema-valid" with "quality."

Decision steps:

1. Surface the empirical finding: JSON validity ≠ content quality. Cite Aider's 20%→61% (unified diff vs JSON) [aider.chat/2024/08/14/code-in-json.html] and "Let Me Speak Freely?" [arxiv.org/abs/2408.02442].
2. Audit each call by content type using §3 Step 1 table.
3. For code-emitting calls: keep markdown; a

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: agentsope
• Source: agentsope/SkillAlchemy
• License: MIT

Install and usage instructions live in the source repository linked above.