Data Science Analysis

Data Science Analysis

A single, domain-agnostic workflow for answering quantitative questions from local datasets. Data files may live in any directory the task provides (e.g., input/, data/, the current working directory, or an explicit path in the prompt); first identify that data directory, then apply the workflow to it. The steps below apply to any domain (archeology, biomedical, environmental, legal, wildfire, astronomy/heliophysics/space weather, and beyond) and to any file type (tabular CSV/Excel or scientific array/binary formats like .npz, .fits, .h5). Domain-specific gotchas are flagged inline and detailed in references/domain-notes.md.

Core Principles

| Principle | Why | |-----------|-----| | Python for all arithmetic | Prevents LLM math errors | | Read files directly in scripts | Never hardcode numbers from manual inspection—they drift and break | | Consult the data dictionary first | Map every prompt term to exact columns/categories; never guess | | Single consolidated script | For multi-step analysis, avoid many small .py files and inline python3 -c | | Print intermediate results | Catches silent filtering/join errors before they reach the answer | | Recompute derived metrics from raw | Never trust pre-calculated rates/percentages in source files | | Round only at the final step | Intermediate rounding compounds error |

Workflow

1. Discover & Profile

• Identify the data directory the task provides (e.g., input/, data/, the current working directory, or an explicit path given in the prompt), then glob all files under it (e.g., /**/*) — data files, data dictionaries, metadata, format specs (.fmt, .text, README), helper scripts. Avoid overly restrictive filename filters that might exclude the correct data. Restrict data access to this directory.
• Read data dictionaries / README / format specs first. Map every prompt term (e.g., "damaged", "generally unsafe", "tumor") to its exact column or category — never guess. Cross-reference discovered schemas against format specs to confirm column mappings and units.
• Inspect raw structure before loading: read the first (and last) 10–20 rows raw (header=None) to locate the true header row, metadata/summary/footer rows, delimiters, timestamp formats, and encoding quirks (BOM → utf-8-sig).
• For Excel, list sheet_names via pandas.ExcelFile; prioritize README/Legend/Metadata sheets, then preview each data sheet.
• For scientific/binary files (.npz, .fits, .h5), print all keys and array shapes to identify the correct arrays before use.
• Print exact column names and sample values; use names—not indices—for all downstream selection.

> Run scripts/explore_input.py [data_dir] first to profile every file (pass the data directory; defaults to input/). See references/data-ingestion-and-cleaning.md for encoding, header-detection, Excel, and scientific/binary-format patterns.

2. Ingest Robustly

Write a single Python script. Apply these in the first execution to avoid retries:

• Match the tool to the task complexity: use lightweight numpy/csv for simple whitespace-delimited files and direct extraction; reserve pandas for tabular joins and richer processing. Avoid over-engineering (e.g., direct pairwise comparisons rather than full time-series propagation when they suffice).
• Handle encoding robustly: try utf-8-sig (BOM), fall back to latin-1/cp1252; strip non-breaking spaces.
• Strip whitespace from headers and string columns.
• Sanitize numeric strings (commas, currency symbols, quotes) before casting; coerce with pd.to_numeric(..., errors='coerce').
• Cast through float before int (int(float(x))) to handle decimal strings like "42.0".
• For large files, use memory-efficient loading (usecols, dtype, chunksize; Excel read_only=True).

3. Clean & Sanitize

• Missing/malformed placeholders & sentinel fills: map '-', 'NA', 'N/A', blanks, and sentinels (-999, -9999, and space-physics fills like 9.99E32, 999999.9, 99999.9, 9999, 999.9) to NaN before computing any statistic, correlation, or model—including them severely skews results.
• Censored values: handle detection-limit markers ('100') explicitly before numeric conversion (see domain notes).
• Duplicates & scaled integers: drop duplicate timestamps, exclude non-data/metadata columns, and decode scaled-integer encodings before use.
• String normalization: .str.strip().str.lower() for categorical filters; include na=False in .str.contains()/.str.match() to avoid errors on mixed-type columns.
• Profile unique values in categorical columns before filtering to catch case/whitespace/typo variations.
• Verify data scales: confirm whether values are log-transformed before applying exponentiation; inverse-transform only when raw metrics are required.
• Prefer NaN-ignoring aggregations over dropping rows to avoid silent data loss.

> Caution: Do not normalize typos or split concatenated strings unless the task asks for it—preserve exact literal counts for raw-unique-entry queries. See references/data-ingestion-and-cleaning.md.

4. Integrate & Join

• Identify linking keys (sample IDs, gene symbols, entity names, station IDs) across tables.
• Normalize join keys on both sides: strip whitespace, standardize case, zero-pad ID strings (never numeric-coerce IDs with leading zeros).
• Align granularities before merging (e.g., county → state, daily → monthly).
• Verify key overlap empirically (len(inner) vs len(left)) and log unmatched keys to detect silent drops.
• Apply exclusion criteria first (e.g., remove samples "not in the study") before extracting features.
• Prevent data leakage: when comparing a target entity to a background set, filter the target out of the background.

> See references/analysis-patterns.md § Joins and references/domain-notes.md for entity normalization (MSA/County suffixes) and biomedical linking.

5. Filter

• Apply all spatial, temporal, and categorical filters before computing any aggregations.
• Translate qualitative constraints ('summer', 'preceding days', 'generally unsafe') into precise numeric/datetime bounds via the data dictionary.
• Group by stable identifiers (conflict ID, entity ID) rather than ambiguous names to prevent double-counting multi-row entries.
• Print intermediate row counts after each filter to verify the subset matches the prompt's scope. Ensure denominators reflect only the filtered subset.

6. Analyze

• Extremum / closest-value queries: sort by primary metric, then secondary tie-breaker; isolate all ties matching the primary condition before applying the tie-breaker. Print tie counts.
• Rates & ratios: align numerator and denominator to the exact same categorical/temporal/geographic scope. Distinguish pooled overall rate (grand totals) from average of periodic rates (mean of per-period rates)—compute exactly what the prompt asks. Guard against division by zero.
• Time-series: sort chronologically before any lag/rolling/diff. Filter to the exact prompt range (don't rely on global extrema); when merging multi-source series, resample to a common cadence and inner-join on timestamp. Verify temporal coverage from the actual data, not filenames. Handle gaps and interpolate missing periods; exclude partial-year data from temporal averages.
• Signal processing: detect peaks/troughs with scipy.signal.find_peaks rather than manual thresholding; rank top-N extrema by magnitude (not raw data); cluster events within a tolerance window to avoid overcounting. See references/analysis-patterns.md § Signal Processing.
• Spatial & orbital: validate coordinate ranges (detect swapped lat/lon); filter by attributes before computing distances; count distinct entities to prevent double-counting. For orbital/coordinate work, use domain libraries (astropy, sgp4/skyfield, pymap3d), normalize longitude conventions, and track units. See references/geospatial.md.
• Forecasting: use strict chronological train/test splits (never random); align lagged features and distinguish forecast-issuance from target dates; evaluate on date-aligned observed vs. predicted values; respect task-specified windows exactly. See references/analysis-patterns.md § Forecasting & Evaluation.
• Statistics: choose robust methods—Spearman for skewed/outlier data, log-transforms for right-skewed outcomes, non-parametric tests for distribution shifts. For causal questions, map variables to roles (treatment/outcome/confounder) and diagnose selection bias.
• Domain scales: align units before analysis (e.g., paleo BP counts backwards—"later" = minimum BP). See references/domain-notes.md.

> See references/analysis-patterns.md for tie-breaking, rate, time-series, signal-processing, forecasting, and statistical patterns.

7. Validate

Print intermediate results at every stage:

• Frequency distributions of parsed values and pre/post-filter row counts.
• Matched subsets (city pairs, filtered rows) and tie-breaking subsets to confirm correct records.
• Boundary-straddling edge cases in range filters; pre/post-merge overlap sizes.
• Sanity checks (e.g., warn if numerator exceeds denominator).

8. Output

Save the final answer under the current artifact directory supplied by the DataPaw runtime:

Direct chat:        artifacts//answer.json
TaskGraph node:     artifacts////answer.json

{"answer": }

• Always use the current session_id. Append graph_id/node_id only when the

runtime explicitly supplies an active TaskGraph node; never invent IDs.

• Do not write answer files directly under the shared artifacts/ root.
• Ensure the current artifact directory exists before writing.
• Use the write_file tool (not shell echo/redirection) for proper JSON formatting.
• Match exact requested precision; omit units, thousands separators, markdown, and conversational filler.
• Obey negative constraints (e.g., "No explanation needed")—output only the direct answer value.
• In the text response, cite source files/sheets and summarize filtering/exclusion logic; list actual entities when counting or identifying specific items.

Environment Setup

pip install pandas openpyxl numpy scipy statsmodels

Add shapely (and optionally geopandas) for geospatial tasks; beautifulsoup4 for HTML tables; h5py/astropy for .h5/.fits files; astropy, sgp4/skyfield, pymap3d for orbital mechanics and coordinate conversions. If a specialized package fails to install, fall back to core numpy/scipy. Rely solely on local files—no web access.

Additional Resources

| Resource | When to use | |----------|-------------| | scripts/explore_input.py | Run at the start of every task to profile all files in the task's data directory (pass it as an argument; defaults to input/) | | references/data-ingestion-and-cleaning.md | Encoding, header detection, type coercion, sentinel fills, censored values, Excel/HTML/scientific-binary parsing, join-key mismatches | | references/analysis-patterns.md | Tie-breaking, joins, rate/ratio calculation, time-series, signal processing, forecasting, statistics, causal inference | | references/geospatial.md | Coordinate validation, distance calculations, GPKG → shapely fallback, orbital mechanics & coordinate systems | | references/domain-notes.md | Domain-specific knowledge (paleo/temporal scales, biomedical, environmental, legal, wildfire, astronomy/heliophysics/space weather) |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: agentscope-ai
• Source: agentscope-ai/QwenPaw-Data
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.