Research Paper Writing

Research Paper Writing Pipeline

End-to-end pipeline for producing publication-ready ML/AI research papers targeting NeurIPS, ICML, ICLR, ACL, AAAI, and COLM. This skill covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission.

This is not a linear pipeline — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops.

┌─────────────────────────────────────────────────────────────┐
│                    RESEARCH PAPER PIPELINE                  │
│                                                             │
│  Phase 0: Project Setup ──► Phase 1: Literature Review      │
│       │                          │                          │
│       ▼                          ▼                          │
│  Phase 2: Experiment     Phase 5: Paper Drafting ◄──┐      │
│       Design                     │                   │      │
│       │                          ▼                   │      │
│       ▼                    Phase 6: Self-Review      │      │
│  Phase 3: Execution &           & Revision ──────────┘      │
│       Monitoring                 │                          │
│       │                          ▼                          │
│       ▼                    Phase 7: Submission               │
│  Phase 4: Analysis ─────► (feeds back to Phase 2 or 5)     │
│                                                             │
└─────────────────────────────────────────────────────────────┘

---

When To Use This Skill

Use this skill when:

• Starting a new research paper from an existing codebase or idea
• Designing and running experiments to support paper claims
• Writing or revising any section of a research paper
• Preparing for submission to a specific conference or workshop
• Responding to reviews with additional experiments or revisions
• Converting a paper between conference formats
• Writing non-empirical papers — theory, survey, benchmark, or position papers (see [Paper Types Beyond Empirical ML](#paper-types-beyond-empirical-ml))
• Designing human evaluations for NLP, HCI, or alignment research
• Preparing post-acceptance deliverables — posters, talks, code releases

Core Philosophy

1. Be proactive. Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate.
2. Never hallucinate citations. AI-generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as [CITATION NEEDED].
3. Paper is a story, not a collection of experiments. Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready.
4. Experiments serve claims. Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative.
5. Commit early, commit often. Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history.

Proactivity and Collaboration

Default: Be proactive. Draft first, ask with the draft.

| Confidence Level | Action | |-----------------|--------| | High (clear repo, obvious contribution) | Write full draft, deliver, iterate on feedback | | Medium (some ambiguity) | Write draft with flagged uncertainties, continue | | Low (major unknowns) | Ask 1-2 targeted questions via clarify, then draft |

| Section | Draft Autonomously? | Flag With Draft | |---------|-------------------|-----------------| | Abstract | Yes | "Framed contribution as X — adjust if needed" | | Introduction | Yes | "Emphasized problem Y — correct if wrong" | | Methods | Yes | "Included details A, B, C — add missing pieces" | | Experiments | Yes | "Highlighted results 1, 2, 3 — reorder if needed" | | Related Work | Yes | "Cited papers X, Y, Z — add any I missed" |

Block for input only when: target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first.

---

Phase 0: Project Setup

Goal: Establish the workspace, understand existing work, identify the contribution.

Step 0.1: Explore the Repository

# Understand project structure
ls -la
find . -name "*.py" | head -30
find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding"

Look for:

• README.md — project overview and claims
• results/, outputs/, experiments/ — existing findings
• configs/ — experimental settings
• .bib files — existing citations
• Draft documents or notes

Step 0.2: Organize the Workspace

Establish a consistent workspace structure:

workspace/
  paper/               # LaTeX source, figures, compiled PDFs
  experiments/         # Experiment runner scripts
  code/                # Core method implementation
  results/             # Raw experiment results (auto-generated)
  tasks/               # Task/benchmark definitions
  human_eval/          # Human evaluation materials (if needed)

Step 0.3: Set Up Version Control

git init  # if not already
git remote add origin 
git checkout -b paper-draft  # or main

Git discipline: Every completed experiment batch gets committed with a descriptive message. Example:

Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task)
Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier

Step 0.4: Identify the Contribution

Before writing anything, articulate:

• The What: What is the single thing this paper contributes?
• The Why: What evidence supports it?
• The So What: Why should readers care?

> Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?"

Step 0.5: Create a TODO List

Use the todo tool to create a structured project plan:

Research Paper TODO:
- [ ] Define one-sentence contribution
- [ ] Literature review (related work + baselines)
- [ ] Design core experiments
- [ ] Run experiments
- [ ] Analyze results
- [ ] Write first draft
- [ ] Self-review (simulate reviewers)
- [ ] Revise based on review
- [ ] Submission prep

Update this throughout the project. It serves as the persistent state across sessions.

Step 0.6: Estimate Compute Budget

Before running experiments, estimate total cost and time:

Compute Budget Checklist:
- [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs)
- [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds)
- [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate)
- [ ] Total budget ceiling and contingency (add 30-50% for reruns)

Track actual spend as experiments run:

# Simple cost tracker pattern
import json, os
from datetime import datetime

COST_LOG = "results/cost_log.jsonl"

def log_cost(experiment: str, model: str, input_tokens: int, output_tokens: int, cost_usd: float):
    entry = {
        "timestamp": datetime.now().isoformat(),
        "experiment": experiment,
        "model": model,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "cost_usd": cost_usd,
    }
    with open(COST_LOG, "a") as f:
        f.write(json.dumps(entry) + "\n")

When budget is tight: Run pilot experiments (1-2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs.

Step 0.7: Multi-Author Coordination

Most papers have 3-10 authors. Establish workflows early:

| Workflow | Tool | When to Use | |----------|------|-------------| | Overleaf | Browser-based | Multiple authors editing simultaneously, no git experience | | Git + LaTeX | git with .gitignore for aux files | Technical teams, need branch-based review | | Overleaf + Git sync | Overleaf premium | Best of both — live collab with version history |

Section ownership: Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency.

Author Coordination Checklist:
- [ ] Agree on section ownership (who writes what)
- [ ] Set up shared workspace (Overleaf or git repo)
- [ ] Establish notation conventions (before anyone writes)
- [ ] Schedule internal review rounds (not just at the end)
- [ ] Designate one person for final formatting pass
- [ ] Agree on figure style (colors, fonts, sizes) before creating figures

LaTeX conventions to agree on early:

• \method{} macro for consistent method naming
• Citation style: \citet{} vs \citep{} usage
• Math notation: lowercase bold for vectors, uppercase bold for matrices, etc.
• British vs American spelling

---

Phase 1: Literature Review

Goal: Find related work, identify baselines, gather citations.

Step 1.1: Identify Seed Papers

Start from papers already referenced in the codebase:

# Via terminal:
grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py"
find . -name "*.bib"

Step 1.2: Search for Related Work

Load the arxiv skill for structured paper discovery: skill_view("arxiv"). It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation.

Use web_search for broad discovery, web_extract for fetching specific papers:

# Via web_search:
web_search("[main technique] + [application domain] site:arxiv.org")
web_search("[baseline method] comparison ICML NeurIPS 2024")

# Via web_extract (for specific papers):
web_extract("https://arxiv.org/abs/2303.17651")

Additional search queries to try:

Search queries:
- "[main technique] + [application domain]"
- "[baseline method] comparison"
- "[problem name] state-of-the-art"
- Author names from existing citations

Recommended: Install Exa MCP for real-time academic search:

claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp"

Step 1.2b: Deepen the Search (Breadth-First, Then Depth)

A flat search (one round of queries) typically misses important related work. Use an iterative breadth-then-depth pattern inspired by deep research pipelines:

Iterative Literature Search:

Round 1 (Breadth): 4-6 parallel queries covering different angles
  - "[method] + [domain]"
  - "[problem name] state-of-the-art 2024 2025"
  - "[baseline method] comparison"
  - "[alternative approach] vs [your approach]"
  → Collect papers, extract key concepts and terminology

Round 2 (Depth): Generate follow-up queries from Round 1 learnings
  - New terminology discovered in Round 1 papers
  - Papers cited by the most relevant Round 1 results
  - Contradictory findings that need investigation
  → Collect papers, identify remaining gaps

Round 3 (Targeted): Fill specific gaps
  - Missing baselines identified in Rounds 1-2
  - Concurrent work (last 6 months, same problem)
  - Key negative results or failed approaches
  → Stop when new queries return mostly papers you've already seen

When to stop: If a round returns >80% papers already in your collection, the search is saturated. Typically 2-3 rounds suffice. For survey papers, expect 4-5 rounds.

For agent-based workflows: Delegate each round's queries in parallel via delegate_task. Collect results, deduplicate, then generate the next round's queries from the combined learnings.

Step 1.3: Verify Every Citation

NEVER generate BibTeX from memory. ALWAYS fetch programmatically.

For each citation, follow the mandatory 5-step process:

Citation Verification (MANDATORY per citation):
1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords
2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef)
3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory)
4. VALIDATE → Confirm the claim you're citing actually appears in the paper
5. ADD → Add verified BibTeX to bibliography
If ANY step fails → mark as [CITATION NEEDED], inform scientist

# Fetch BibTeX via DOI
import requests

def doi_to_bibtex(doi: str) -> str:
    response = requests.get(
        f"https://doi.org/{doi}",
        headers={"Accept": "application/x-bibtex"}
    )
    response.raise_for_status()
    return response.text

If you cannot verify a citation:

\cite{PLACEHOLDER_author2024_verify_this}  % TODO: Verify this citation exists

Always tell the scientist: "I've marked [X] citations as placeholders that need verification."

See [references/citation-workflow.md](references/citation-workflow.md) for complete API documentation and the full CitationManager class.

Step 1.4: Organize Related Work

Group papers by methodology, not paper-by-paper:

Good: "One line of work uses X's assumption [refs] whereas we use Y's assumption because..." Bad: "Smith et al. introduced X. Jones et al. introduced Y. We combine both."

---

Phase 2: Experiment Design

Goal: Design experiments that directly support paper claims. Every experiment must answer a specific question.

Step 2.1: Map Claims to Experiments

Create an explicit mapping:

| Claim | Experiment | Expected Evidence | |-------|-----------|-------------------| | "Our method outperforms baselines" | Main comparison (Table 1) | Win rate, statistical significance | | "Effect is larger for weaker models" | Model scaling study | Monotonic improvement curve | | "Convergence requires scope constraints" | Constrained vs unconstrained | Convergence rate comparison |

Rule: If an experiment doesn't map to a claim, don't run it.

Step 2.2: Design Baselines

Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?"

Standard baseline categories:

• Naive baseline: Simplest possible approach
• Strong baseline: Best known existing method
• Ablation baselines: Your method minus one component
• Compute-matched baselines: Same compute budget, different allocation

Step 2.3: Define Evaluation Protocol

Before running anything, specify:

• Metrics: What you're measuring, direction symbols (higher/lower better)
• Aggregation: How results are combined across runs/tasks
• Statistical tests: What tests will establish significance
• Sample sizes: How many runs/problems/tasks

Step 2.4: Write Experiment Scripts

Follow these patterns from successful research pipelines:

Incremental saving — save results after each step for crash recovery:

# Save after each problem/task
result_path = f"results/{task}/{strategy}/result.json"
if os.path.exists(result_path):
    continue  # Skip already-completed work
# ... run experiment ...
with open(result_path, 'w') as f:
    json.dump(result, f, indent=2)

Artifact preservation — save all intermediate outputs:

results//
  /
    /
      final_output.md          # Final result
      history.json             # Full trajectory
      pass_01/                 # Per-iteration artifacts
        version_a.md
        version_b.md
        critic.md

Separation of concerns — keep generation, evaluation, and visualization separate:

run_experiment.py              # Core experiment runner
run_baselines.py               # Baseline comparison
run_comparison_judge.py        # Blind evaluation
analyze_results.py             # Statistical analysis
make_charts.py                 # Visualization

See [references/experiment-patterns.md](references/experiment-patterns.md) for complete design patterns, cron monitoring, and error recovery.

Step 2.5: Design Human Evaluation (If Applicable)

Many NLP, HCI, and alignment papers require human evaluation as primary or complementary evidence. Design this before running automa

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mateaix
• Source: mateaix/mateclaw
• License: Apache-2.0
• Homepage: https://claw.mate.vip

Install and usage instructions live in the source repository linked above.