# Pyfixest Grid Sharding

> |

- **Type:** Skill
- **Install:** `agentstack add skill-kennethkhoocy-applied-micro-skills-pyfixest-grid-sharding`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [kennethkhoocy](https://agentstack.voostack.com/s/kennethkhoocy)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [kennethkhoocy](https://github.com/kennethkhoocy)
- **Source:** https://github.com/kennethkhoocy/applied-micro-skills/tree/main/plugins/applied-micro/skills/pyfixest-grid-sharding

## Install

```sh
agentstack add skill-kennethkhoocy-applied-micro-skills-pyfixest-grid-sharding
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# pyfixest Grid Sharding

## Problem

A regression grid (e.g. 2 measures x 3 FE structures x pooled+per-domain x 3
label variants ~ 70 models) on a 327k-row panel with high-cardinality director
FE ran ~55 s/model sequentially — ~65 min wall — on an RTX 5080 machine with
`demeaner_backend="cupy64"` on every call. The GPU was NOT the bottleneck.

## Context / Trigger Conditions

- Measured signature (verified 2026-07-21, H5 seat-loss rerun): job process at
  ~1.4 cores CPU (37.7 CPU-min in 27 wall-min), `nvidia-smi` 0% utilization
  with ~4 GB resident (cupy context loaded, idle), one pyfixest singleton
  warning per completed model ticking by in the log.
- Any orchestration prompt that asks a worker to "rerun every headline cell
  under variants A/B/C" without specifying execution structure.

## Solution

1. Diagnose before blaming the GPU: check process CPU-minutes vs wall-clock
   (~1 core => serial CPU-bound) and GPU utilization (near 0% => demeaning is
   not the constraint). The cupy64 kwarg is still correct; it just cannot fix
   a CPU-dominated pipeline.
2. Shard the GRID, not the data: split the model list across N OS processes
   (`--shard i --nshards N` over the model index, one output part-file each,
   merge step at the end), N ~ cores-4. Models are independent — this is the
   Execution Style process-sharding pattern applied to regressions.
3. Amortize fixed costs inside a shard: build the panel/interactions ONCE per
   variant and reuse; where specs share RHS/FE, use pyfixest multiple-
   estimation syntax (multiple depvars / sw()/csw() stepwise) so one model
   matrix serves several reported cells.
4. Orchestrator rule: put the sharding mandate IN the worker prompt for any
   grid larger than ~10 models. Workers default to sequential loops otherwise.
5. Mid-flight call: if a sequential grid is already >1/3 done with no
   per-model checkpoint, let it finish — restart+shard usually nets slower.
   Grids launched fresh should checkpoint per model (append-only part file) so
   this trade-off never binds again.

## Verification

Sharded reruns of the same grid should show near-linear speedup up to
memory/RAM limits; per-model results must be byte-identical to the sequential
run (same seeds not needed — feols is deterministic).

## Measured GPU-saturation verdict (2026-07-21 escalation experiment)

A controlled escalation loop (same 327k-row seat-loss grid, N concurrent OS
shard processes, nvidia-smi sampled every 2 s, RTX 5080) settled the question
empirically: mean GPU utilization was **0.7% at N=4, 0.6% at N=8, and ~1% at
N=12** (peaks 2-5%), with total VRAM flat around 4 GB. GPU saturation is
UNATTAINABLE for pyfixest cupy64 grids — the demeaning kernel is a brief burst
inside a CPU-bound per-model pipeline — so the correct objective is CPU-core
saturation via process shards, with cupy64 kept on per project rules. Two
further measured costs: (1) kill-and-escalate restarting loses in-flight fits
(throughput FELL from 1.29 to 0.64 fits/min when escalating 4->8 mid-run) — pick
N once from cores and RAM, do not escalate live; (2) each shard holds the panel
in RAM (~1.2 GB for a 327k-row panel; scale linearly), so cap N by free RAM
before cores. Evidence: `.claude-local\specialist-directors-us\
classifier_aug_2026-07-21\stageB_v2\h5_seatloss_gpu\attempts.json`.

## Notes

- VRAM: N concurrent cupy64 processes each hold a context (~4 GB observed on
  a 327k x 40k-FE problem); on a 16 GB card cap GPU-sharing shards at ~3 or
  run overflow shards with the numba default (flag them per project rules).
- See also: [pyfixest-cupy64-absorbed-regressors] (numerical differences of
  the cupy backend — unrelated to speed), and the global CLAUDE.md Execution
  Style section (process-level parallelism; GIL makes threads useless here).

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [kennethkhoocy](https://github.com/kennethkhoocy)
- **Source:** [kennethkhoocy/applied-micro-skills](https://github.com/kennethkhoocy/applied-micro-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-kennethkhoocy-applied-micro-skills-pyfixest-grid-sharding
- Seller: https://agentstack.voostack.com/s/kennethkhoocy
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.