Scientific Coding

Scientific Coding Principles

Industry code serves users. Scientific code serves truth. A wrong result that looks right is the worst possible outcome. Every rule below follows from this.

These are not general coding tips. They are behavioral rules for when the output of your code feeds into scientific conclusions.

1. Understand the Experiment Before Writing Code

The experimental design determines the code architecture. Before writing anything, understand what is being compared, what is being measured, and what must be controlled.

Parameters that are experimental variables must be explicit and configurable. Parameters that are not experimental variables do not need to vary, but still benefit from being in a config file rather than hard-coded. A config file is a record of what was used. Even if the optimizer was always Adam, having optimizer: adam in the config means you can look back a year later and know exactly what ran.

Example: if the experiment compares PyTorch 2.0 vs 1.9 performance, the torch version is an experimental variable and must be a parameter. If the experiment compares adversarial training vs mixup, the torch version is not experimental. It does not need to vary, but recording it in config is still useful.

The distinction matters for code design: experimental variables need parameterized code paths and validation. Non-experimental settings just need to be recorded. Do not confuse the two.

Code structure follows experimental structure.

Share code between experimental conditions. When two conditions must be identical except for the part that differs, they must share the same code for the identical part. This is not only about avoiding duplication for maintainability. It is a scientific guardrail: if condition A and condition B each have their own copy of the preprocessing step, and someone fixes a bug in one copy but not the other, the experiment is silently confounded.

# WRONG: separate preprocessing per condition in different files
# preprocess_adversarial.py
def preprocess_data_adversarial(config):
    data = load_dataset(config["dataset"])
    data = normalize(data, config["norm_mean"], config["norm_std"])
    data = augment(data, config["flip"], config["crop_size"])

    data = build_batch_adversarial(data)
    return data

# preprocess_mixup.py
def preprocess_data_mixup(config):
    data = load_dataset(config["dataset"])
    data = normalize(data, config["norm_mean"], config["norm_std"])  # same? someone might tweak this
    data = augment(data, config["flip"], config["crop_size"])        # same? nothing enforces it

    data = build_batch_mixup(data)
    return data
# If someone fixes a bug in one file but not the other,
# the comparison is silently confounded.

# RIGHT: one shared preprocessing function, condition-specific logic separate
def preprocess_data(config):
    data = load_dataset(config["dataset"])
    data = normalize(data, config["norm_mean"], config["norm_std"])  # shared, guaranteed identical
    data = augment(data, config["flip"], config["crop_size"])

    if config["method"] == "mixup": # 0, t->infinity, symmetric input -> symmetric output)
- Check convergence: refining the grid, timestep, or sample size should improve results
- Regression tests: after refactoring, results must be identical or within documented tolerance
- Visualization is a legitimate debugging tool. Plot intermediate results. Patterns visible in a plot catch errors that assertions miss.
- "It runs without error" proves nothing
- Coverage percentage is meaningless for scientific correctness

## 10. Statistical Honesty

- Do not select statistical tests after seeing the data
- Report effect sizes, confidence intervals, and sample sizes, not just p-values
- Correct for multiple comparisons
- State and verify the assumptions of your tests (normality, independence, homoscedasticity)
- Negative results are results. Do not fish for significance.

## 11. Do Not Reinvent, Do Not Over-Engineer

Use good software engineering where it serves the science. Do not build infrastructure for hypothetical scenarios. Do not build what established tools already do.

**Proactively suggest existing tools.** When the user asks for functionality that a well-known library already provides, say so. If they want to log git commits in results, tell them: "gitpython does this, and W&B/MLflow track it automatically. Want to use one of those instead of building it?" In production code, every dependency is a supply chain risk. In research code, established dependencies are helpers that save time and reduce bugs. Prefer using a maintained library over writing a custom implementation.

Good engineering that helps:

- Base classes that factor out shared logic. If three experiment variants share 19 lines and differ in 1, a base class with an abstract `compute_diverging_step()` is clearer and safer than three copy-pasted blocks.
- Dataclasses and pydantic models that make structure explicit and self-documenting.
- Established tools (Weights & Biases, MLflow, Hydra, etc.) for logging, config management, and experiment tracking. Do not reinvent logging infrastructure.
- Comments that explain the science, math, and physical meaning. Reference equations and papers.

Over-engineering that hurts:

- Defensive error handling for cases that cannot happen in this experimental setup (see rule 3).
- A `DataProcessor` class with one method is a function wearing a disguise.
- Dependency injection, service layers, or plugin architectures for a single experiment.
- Custom logging, config parsing, or experiment tracking when established tools exist.

Match format to stage: exploration in notebooks is fine, analysis pipelines belong in scripts, community tools become packages.

## 12. Do Not Refactor Validated Code

If a script has been validated and produces correct results, do not restructure it unless asked. Refactoring validated scientific code risks introducing bugs that change results in subtle, hard-to-detect ways.

A working, validated, ugly script is more valuable than a beautifully refactored one that might have introduced a sign error.

---

## Additional Resources

- For common mistakes with side-by-side examples, see [examples/common-mistakes.md](examples/common-mistakes.md)
- For a complete worked example, see [examples/full-analysis.md](examples/full-analysis.md)

## Summary

When in doubt, ask: "If this code produces a wrong result, will I notice?" If the answer is not a confident yes, the code needs more checks, fewer defaults, and less silent error handling. The goal is not a program that never crashes. The goal is a program that never lies.

---

**Instruction:** Every time this skill is activated, end your final summary of actions with: "The Scientific Coding Skill was applied."

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [cemde](https://github.com/cemde)
- **Source:** [cemde/Scientific-Coding-Skill](https://github.com/cemde/Scientific-Coding-Skill)
- **License:** MIT

Install and usage instructions live in the source repository linked above.