Data Analysis

Data Analysis Methodology

Prioritize correctness over speed — a wrong insight is worse than no insight.

Step 1: Data Ingestion

Uploaded files are located at /home/user/uploads/. Always list that directory first to discover available files:

import os
for f in os.listdir('/home/user/uploads/'):
    print(f)

Tool selection: Use code_run as the primary execution tool — it is universally supported across all sandbox providers. Use code_interpret only when you need rich output capture (e.g., inline dataframes, rendered plots); note that code_interpret may not be available in all environments.

Examine the raw file (first 20-30 lines) to understand format, delimiter, encoding, headers, and obvious quality issues before loading.

Load by file type — always specify dtypes for known columns and parse_dates for date columns:

| Extension | Loader | |---|---| | .csv | pd.read_csv('/home/user/uploads/file.csv', parse_dates=[...]) | | .tsv | pd.read_csv('/home/user/uploads/file.tsv', sep='\t', parse_dates=[...]) | | .xlsx / .xls | pd.read_excel('/home/user/uploads/file.xlsx', engine='openpyxl') | | .json | pd.read_json('/home/user/uploads/file.json') | | .parquet | pd.read_parquet('/home/user/uploads/file.parquet') |

Step 2: Mandatory EDA

Run this diagnostic block on every dataset before any analysis:

print(f"Shape: {df.shape}")
print(f"\nDtypes:\n{df.dtypes}")
print(f"\nMissing values:\n{df.isnull().sum()[df.isnull().sum() > 0]}")
print(f"\nDuplicate rows: {df.duplicated().sum()}")
print(f"\nNumeric summary:\n{df.describe()}")

for col in df.select_dtypes(include='object').columns:
    n_unique = df[col].nunique()
    print(f"\n{col}: {n_unique} unique values")
    if n_unique 5 categories) |
| Relationship between two numeric variables | Scatter plot |
| Trend over time | Line chart |
| Part-of-whole | Stacked bar (NOT pie chart) |
| Correlation matrix | Heatmap |

Sandbox-specific rules:
- Use `matplotlib.use('Agg')` — there is no display
- Always call `plt.close()` after saving to free memory
- Save plots to `/home/user/output/` at 150 dpi — create the directory first: `os.makedirs('/home/user/output/', exist_ok=True)`
- When using `code_run`, pass saved file paths via the `output_files` parameter so artifacts are tracked. When using `code_interpret`, rich outputs (plots, dataframes) are auto-captured — but for file-based outputs (saved PNGs), prefer `code_run` with `output_files`
- Use colorblind-friendly palettes (`tab10`, `Set2`)

## Step 6: Report Structure

Dataset Overview: [rows × columns, date range if applicable]

Key Findings:

1. [Quantified finding — "Revenue increased 23% QoQ"]
2. [Finding with statistical backing — "Correlation r=0.82, p<0.001"]

Data Quality Notes: [Cleaning applied, missing data, caveats]

Methodology: [Tests/techniques used and why]

**Never say "X causes Y" from observational data alone.**

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [droxer](https://github.com/droxer)
- **Source:** [droxer/Synapse](https://github.com/droxer/Synapse)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.