SpriteGameGen

AI Sprite & Game Asset Tool

A local-first character animation workbench that turns a text prompt (and optional reference image) into reusable animation clips and versioned engine-ready bundles. Azure GPT Image, Gemini, or an independently operated loopback ComfyUI server can do the generative work; a deterministic Python pipeline handles repair, palettes, sprite-sheet packing, frame ZIPs, bundle checksums, and Godot resources. Side-scroller walks also provide Gemini with deterministic eight-phase pose guides, so frames change limb geometry instead of merely restyling the same stance.

Licensed under [Apache-2.0](LICENSE). See [CONTRIBUTING.md](CONTRIBUTING.md), [CODEOFCONDUCT.md](CODEOFCONDUCT.md), and [SECURITY.md](SECURITY.md) before contributing or reporting a vulnerability.

Layout

backend/    FastAPI + deterministic image pipeline (Python 3.11+)
frontend/   React + Vite + TypeScript thin client
projects/   per-project output folders (git-ignored)

Backend setup

cd backend
uv sync --locked --extra dev
cp .env.example .env        # then fill in the values below
uv run python ../scripts/doctor.py
uv run pytest -q
uv run uvicorn app.main:app --reload

Environment / auth

This project authenticates to Gemini through Vertex AI (Google Agent Platform) using a service-account JSON key — not a GEMINI_API_KEY. Set in .env:

| Var | Meaning | |---|---| | SPRITE_ENV_FILE | Optional absolute dotenv path. If omitted, the backend uses backend/.env regardless of process CWD. Relative paths inside the selected dotenv file resolve from that file's directory. | | GOOGLE_APPLICATION_CREDENTIALS | Path to the service-account JSON key file (loaded explicitly; if unset, falls back to gcloud ADC) | | GOOGLE_CLOUD_PROJECT | GCP project ID | | GOOGLE_CLOUD_REGION | Vertex region (default global) | | GEMINI_MODEL_GENERATE | Stage 1 model (default gemini-3.1-flash-image) | | GEMINI_MODEL_EDIT | Stage 2 model (default gemini-3.1-flash-image) | | GEMINI_MODEL_TEXT | Optional prompt-preview model (default gemini-3.5-flash) | | GEMINI_TIMEOUT_SECONDS | Per-attempt Gemini request timeout (default 120) | | GEMINI_MAX_RETRIES | Maximum attempts for retryable Gemini failures (default 5) | | GEMINI_BACKOFF_SECONDS | Initial retry delay for non-quota failures; subsequent delays double (default 1) | | GEMINI_QUOTA_BACKOFF_SECONDS | Minimum delay after 429 RESOURCE_EXHAUSTED before an automatic retry (default 15) | | AZURE_OPENAI_ENDPOINT | Optional Azure OpenAI resource or /openai/v1 endpoint | | AZURE_OPENAI_API_KEY | Azure API key; local secret, never commit it | | AZURE_OPENAI_DEPLOYMENT | Azure deployment name, for example gpt-image-2-2 | | AZURE_IMAGE_QUALITY | low, medium, high, or auto (default low) | | AZURE_IMAGE_TIMEOUT_SECONDS | Per-attempt Azure request timeout (default 180) | | AZURE_IMAGE_MAX_RETRIES | Maximum Azure attempts for retryable failures (default 2) | | AZURE_IMAGE_MAX_CONCURRENCY | Maximum concurrent Azure frame edits (default 3) | | PROJECTS_DIR | Output dir (default ./projects). Relative values resolve from the selected dotenv file's directory, not the process CWD. | | ACTION_PACKS_DIR | Optional immediate directory of strict data-only JSON action packs. Relative values resolve from the selected dotenv file. | | COMFYUI_URL | Optional explicit loopback URL and port, for example http://127.0.0.1:8188. | | COMFYUI_WORKFLOW_DESCRIPTOR | Trusted operator-owned descriptor beside an API-format workflow JSON file. | | MAX_UPLOAD_BYTES | HTTP reference-image upload limit (default 10485760); reported by MCP capabilities even though direct MCP generation does not accept image uploads. |

> The service-account key must stay local — it is git-ignored (project-*.json) and > must never be pushed.

> Region note: Gemini 3.x image models (gemini-3.1-flash-image, gemini-3-pro-image) > are served from the global endpoint, not a regional one like us-central1 — a > regional endpoint returns a 404 "model not found in region".

> First run: rembg downloads its ~170 MB background-removal model on first use, so > the first generate/animate call is slow. The onnxruntime inference backend is a > declared dependency (installed via pip install -e ".[dev]").

Frontend setup

cd frontend
npm ci
npm run dev     # dev server on http://localhost:5173
npm run build

Testing

• Pipeline (backend/tests/): pure unit tests against committed fixtures — fast,

free, deterministic.

• Gemini client: tested against a mocked SDK; no real API calls in CI.
• Frontend (frontend/): npm test — Vitest covers API request shaping, project

browser resume/delete behavior, Zustand hydration, FrameStrip regenerate/curation actions, and the AnimationPlayer loop timing math.

• Live smoke test: makes real Gemini calls, manual only (kept out of pytest).

Run it from the backend venv after configuring .env:

``bash cd backend # Windows: .venv\Scripts\activate macOS/Linux: source .venv/bin/activate python ../scripts/smoke_generate.py "a knight with a sword" ``

It generates a base sprite, edits it into one walk frame, runs the pipeline, and writes the results to scripts/smoke_out/ (git-ignored) for inspection. Use this to confirm the live model IDs and SDK signatures still match after any dependency bump.

• Live model validation matrix: probes the configured prompt, generation, and edit

models by region; records availability, latency, optional safety-block behavior, and raw/processed artifacts for a scored manual quality review. It is billable, opt-in, outside CI, and writes to a disposable git-ignored directory. See [docs/live-model-validation.md](docs/live-model-validation.md) for the supported model/region table and acceptance rubric.

``powershell cd backend .venv\Scripts\python.exe ..\scripts\validate_live_models.py --repeats 3 --include-block-probe ``

Using the app

With both the backend (uvicorn) and frontend (npm run dev) running, open http://localhost:5173. The saved-project browser loads local projects first; open a healthy project to restore its prompt, sprite, animation frames, and export workflow. Then work through the character workspace:

1. Generate — describe the sprite, pick pixel/hi-res and a game camera, then choose

an allowed direction. Prompt enhancement is optional: request a visible preview, edit it, and explicitly accept it before generation. Choose Auto, Azure, or Gemini; Auto prefers Azure, then Gemini, then configured ComfyUI. Hyperagent is shown as Experimental but remains disabled until its agent-mediated image path is validated. You can also attach a reference.

1. Animate — create or replace a named clip from an action pack or custom motion,

then preview loop ranges and per-frame durations. Side-scrollers allow left/right; top-down/2.5D projects allow all eight directions.

1. Repair — rerender from retained sources, share a palette, flip, nudge,

enable/disable, reset, or regenerate without compounding transforms.

1. Export — download the sheet, atlas, individual-frame ZIP, or deterministic

character bundle. The Godot profile includes Godot 4.7 SpriteFrames and scene files.

See [bundle and Godot documentation](docs/character-bundles.md), [action packs and recipes](docs/actions-and-recipes.md), and the [ComfyUI security boundary](docs/comfyui.md).

Local MCP server

The backend installs sprite-mcp, a local stdio MCP server backed by the same synchronous SpriteService, ProviderRegistry, and project store as FastAPI. Startup is storage-only safe: initialization, capability discovery, project reads, and resource reads do not require cloud credentials. A creative tool fails with a safe tool error if its required provider is not configured.

After installing the backend, register it in an MCP client using the console script:

{
  "mcpServers": {
    "sprite-game": {
      "command": "/absolute/path/to/backend/.venv/bin/sprite-mcp",
      "env": {
        "SPRITE_ENV_FILE": "/absolute/path/to/backend/.env",
        "PROJECTS_DIR": "/absolute/path/to/projects"
      }
    }
  }
}

SPRITE_ENV_FILE must be absolute. PROJECTS_DIR should also be absolute in an MCP client configuration so behavior is explicit even when the client launches the server from a foreign working directory.

Direct MCP contract

The exact direct tool inventory is:

| Tool | Provider/billing effect | Local overwrite effect | |---|---|---| | get_capabilities | None | None | | list_projects | None | None | | get_project | None | None | | enhance_prompt | Gemini text call; may incur provider billing | None | | generate_sprite | One image-generation call through auto, azure, or gemini; may incur provider billing | Creates a new project; does not overwrite an existing project | | animate | Multiple image-edit calls through the provider stored on the project; may incur provider billing | Replaces prior animation frames and animation metadata | | regenerate_frame | One image-edit call through the provider stored on the project; may incur provider billing | Replaces the selected frame | | export_sheet | No provider call or provider billing | Replaces the matching local sprite sheet and atlas outputs | | set_render_settings | None | Rerenders retained sources | | set_frame_adjustment | None | Updates one curated frame | | update_clip / delete_clip | None | Updates or deletes clip-owned state/assets | | export_character_bundle | None | Replaces a deterministic bundle ZIP | | validate_recipe / get_project_recipe | None | Read-only; no overwrite |

auto prefers Azure when configured and otherwise uses Gemini, but only during initial generation. The resolved concrete provider is persisted. animate and regenerate_frame always use that stored provider and return an error if it is no longer configured; they never silently switch providers.

get_capabilities reports application version 0.1.0, provider availability, action presets, camera/direction combinations, and all prompt, upload, decoded-image, export, sheet, and frame-error limits. FastMCP 1.28.1 does not expose a public FastMCP constructor argument for the application version, so the application version is reported here rather than in the MCP initialization server-version field.

Project outputs are MCP-specific DTOs. They include revision, concrete provider, operation outcome, frame status/errors, absolute local paths, and sprite:// resource URIs, but never expose the persisted/browser-only Frame.url field. The server publishes two read-only resource templates:

sprite://projects/{project_id}/manifest
sprite://projects/{project_id}/assets/{filename}

Both resolve through the canonical PROJECTS_DIR store and reject unsafe IDs, filenames, symlink escapes, and traversal. The server does not accept arbitrary filesystem paths or reference-image paths.

FastMCP 1.28.1 generates useful input schemas from Annotated/Pydantic constraints, but its public tool decorator does not provide a strict-extra option. Unknown top-level tool arguments are therefore ignored by the SDK's generated argument model. This limitation is covered by a contract test; the server does not patch private SDK internals.

Capability boundaries

The tools above are exported directly by the local MCP server. Additional operations may exist internally in the framework-neutral runtime; absence from tools/list only means an operation is not a direct MCP tool.

Hyperagent remains an experimental remote-agent capability. It appears in provider capability metadata as unavailable. Its authenticated, agent-mediated image path still requires separate validation; do not infer remote-agent capability from the direct server's tool inventory.

Run the credential-free protocol smoke test from the repository root:

backend\.venv\Scripts\python.exe scripts\smoke_mcp.py

The smoke script launches the installed sprite-mcp console entrypoint from a temporary foreign CWD with absolute temporary SPRITE_ENV_FILE and PROJECTS_DIR values, removes cloud credentials, initializes stdio, asserts the exact tool inventory, calls get_capabilities, and prints only its parent-process success line.

Agent skill

A portable Agent Skills-compatible guide is available at [skills/sprite-game-gen/](skills/sprite-game-gen/). Install that complete directory in your agent host's skill path and connect the sprite-mcp server. The skill teaches agents to discover live capabilities, distinguish billable and destructive operations, preserve partial animation results, verify project revisions, and use sprite:// resources safely. Its setup reference includes a client-neutral stdio configuration example.

Versioning and releases

The package, project manifest, bundle, action-pack, recipe, and batch-state versions evolve independently. See [version contracts](docs/version-contracts.md), the [release process](docs/release-process.md), and [CHANGELOG.md](CHANGELOG.md).

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Nether403
• Source: Nether403/SpriteGameGen
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.