# Local Model Selection

> local-model-selection — Choose and recommend local LLM models for Hermes Agent — VRAM-tier recommendations, uncensored/abliterated variants, quant selection, model family naming conventions, dual-GPU setups, and auxiliary model selection.

- **Type:** Skill
- **Install:** `agentstack add skill-atlasomnia-donna-starter-local-model-selection`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [AtlasOmnia](https://agentstack.voostack.com/s/atlasomnia)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [AtlasOmnia](https://github.com/AtlasOmnia)
- **Source:** https://github.com/AtlasOmnia/donna-starter/tree/main/skills/autonomous-ai-agents/local-model-selection

## Install

```sh
agentstack add skill-atlasomnia-donna-starter-local-model-selection
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Local Model Selection for Hermes

Use this skill when the user asks which local model to run, which quant to pick, which uncensored variant to use, how to pair models across GPUs, or what auxiliary model to use alongside their main model.

## When to use

Trigger on requests like:
- "what's the best Qwen model for my hardware"
- "which quant should I use for 32GB VRAM"
- "what uncensored model for 16GB"
- "what should I run as a secondary/auxiliary model"
- "compare these three models for agent use"
- "is this model still the best or has something newer dropped"
- "where are my LM Studio models"
- "move/delete these LM Studio models to free space"
- "are there any Ollama models on this Mac"
- "uninstall Ollama"

For storage operations, distinguish the machine being cleaned from any remote model host. For LM Studio, inventory exact model directories, check `lms ps` before deletion, delete only explicitly named model repositories, and verify both path removal and real free-space change; see `references/lm-studio-model-storage-operations.md`. For Ollama on macOS, cross-check `ollama list`, allocated model blobs/manifests, and large files before claiming models exist, then use the process/app/CLI/package/data verification sequence in `references/macos-ollama-storage-and-uninstall.md` for a complete uninstall.

## Core rule: verify model existence before recommending

**Never fabricate model names.** Model families have specific size tiers — not every number exists in every family. Before recommending, verify the model exists on HuggingFace or Ollama.

When a user asks for recommendations:

- Use **exact model identifiers**, e.g. "Qwen3.6-27B" not just "a Qwen 3 model."
- Never say "we can run about X parameters" without naming at least one concrete, released, GGUF-available candidate that actually fits.
- If you're unsure of the latest family/size, check via web_search or HuggingFace before answering — the user will spot vague or outdated claims immediately.

**If you mix up versions (e.g., saying "Qwen 3.5" when Qwen 3.6 is current), correct yourself directly instead of hedging.**

## Model family naming conventions

### Qwen family
| Family | Sizes | Key dates |
|--------|-------|-----------|
| Qwen3 | 0.6B, 1.7B, 4B, 8B, **14B**, 32B | Dec 2025 |
| Qwen3.5 | **0.8B, 2B, 4B, 9B**, 27B, 35B-A3B, 122B-A10B | Feb–Jul 2026 |
| Qwen3.6 | 27B, 35B-A3B, 40B (Claude distill) | Apr 2026 |
| Qwen-AgentWorld | **35B-A3B**, 397B-A17B | Jun 2026 — world-model CPT/SFT/RL atop Qwen3.5 MoE, 256K ctx |

**Common pitfall:** No Qwen3.5 14B exists. Qwen3 has 14B; Qwen3.5 has 9B. Do not combine family names with wrong sizes.

**AgentWorld positioning:** Qwen-AgentWorld-35B-A3B is best treated as a world-model / agent-worker candidate, not a drop-in replacement for Qwen3.6-27B dense as the main Hermes brain. Early community reports are promising for long, non-coding agent tasks and local tool-heavy workflows, but mixed for coding. Before judging quality, verify the chat template/tool-call formatting; a patched Qwen `chat_template.jinja` may be necessary for reliable role/tool behavior.

### Gemma 4 family (Apr 2026)
| Size | Type | Notes |
|------|------|-------|
| 12B | Dense | Vision + tool calling |
| 26B-A4B | MoE (~4B active) | Vision + tool calling, good 16GB pick |
| 31B | Dense | Vision + tool calling |

### Other relevant families
- **GPT-OSS 20B** — OpenAI MoE (20B total, 4-6 active), 128K ctx
- **Mistral Small 24B** — Dense, Apache 2.0, naturally less restricted base training
- **Dolphin 3.0** — Dataset-filtered uncensored variants (Mistral 24B, Llama 8B)

## Uncensored/abliterated landscape

Three main techniques:

| Method | How | Trade-off |
|--------|-----|-----------|
| **Heretic** | LoRA-based refusal vector extraction | Lowest KL divergence, best capability preservation |
| **HauhauCS** | Aggressive/Balanced variants | Balanced variant tuned for agentic stability. Aggressive strips preamble. |
| **Huihui** | Crude abliteration (proof-of-concept) | Can cause catastrophic degradation at larger scales (KL >3 on 4B+) |

### Key findings from community benchmarks (Nathan Sapwell, Apr 2026)
- Abliteration is NOT lossless at any scale. All techniques cause measurable benchmark drops.
- Base Qwen3.6-27B refuses 99.5% of harmful prompts. Heretic and HauhauCS both reduce to near-zero while preserving most benchmarks.
- Huihui degrades catastrophically on models >4B — avoid for serious agent work.
- **HauhauCS Balanced** is specifically recommended for agentic coding — keeps self-reasoning preamble which stabilizes long tool-call chains.

### Where to find uncensored GGUFs
- **DavidAU** (HuggingFace) — Heretic + NEO-CODE-Di-IMatrix quants for many models
- **mradermacher** (HuggingFace) — static quants of Heretic variants (Gemma 4 especially)
- **bartowski** (HuggingFace) — standard quants of most models including abliterated
- **huihui-ai** (HuggingFace + Ollama) — crude abliterated variants
- **HauhauCS** (HuggingFace) — Aggressive and Balanced uncensored variants

## VRAM tier recommendations (as of June 2026)

### 16GB VRAM tier (RTX 5070 Ti-class cards)

| Rank | Model | Repo | Quant | VRAM | Notes |
|------|-------|------|-------|------|-------|
| 1 | Gemma 4 26B-A4B Heretic | `mradermacher/gemma-4-26B-A4B-it-heretic-GGUF` | Q4_K_M | ~14GB | MoE, vision, tool calling. InsiderLLM top pick. |
| 2 | Mistral Small 24B Abliterated | various on HF | Q4_K_M | ~14GB | Dense, BSWEN #1 pick, limited ctx room |
| 3 | GPT-OSS 20B Heretic | `DavidAU/OpenAi-GPT-oss-20b-HERETIC-uncensored-NEO-Imatrix-gguf` | IQ4_NL | ~12GB | Highest benchmark scores, MoE can be quirky |
| 4 | Qwen3 14B Abliterated | `huihui-ai/Qwen3-14B-abliterated` | Q4_K_M | ~10.7GB | Same Qwen DNA as main 27B, lots of headroom |
| 5 | Qwen3.5 9B Uncensored | `HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` | Q4_K_M | ~6GB | Lightning fast, weak for complex aux tasks |

### 24GB VRAM (RTX 3090/4090)
- **Qwen3.6-27B abliterated (MTP-GGUF)** at Q6_K (~22.4GB) — headline pick
- **Gemma 4 31B Heretic** at Q4_K_M — dense with vision

### 32GB VRAM tier (RTX 5090-class cards)
- **Primary agent at ordinary context:** Qwen3.6-27B at Q6_K, or Q8_0 when the loaded context leaves enough KV/compute headroom. Heretic/HauhauCS variants are test-first options only for roles that actually require refusal removal; stock is preferred for faithful summarization.
- **Dedicated 231K–262K compressor on one 5090:** use the stock-based `LibertAIDAI/Qwen3.6-27B-NVFP4-GGUF` file `Qwen3.6-27B-NVFP4-Q8_0.gguf` with Q8 K/V cache, text-only, non-thinking, and parallelism 1. Its NVFP4 FFN + Q8 attention/embedding layout leaves room for the long KV cache while preserving more retrieval fidelity than ordinary Q4_K_M.
- Do not recommend a quant without its loaded context: a model that fits at 32K may OOM at 262K.
- Full single-GPU compression memory math, fallback, live spill diagnosis, and verification: `references/single-gpu-long-context-compression.md`.
- **Loaded-but-spilling rule:** inspect the endpoint's live `loaded_instances` configuration and the OS's active CUDA inventory before changing quants or context. A remembered or installed second GPU is not usable capacity; `nvidia-smi` must enumerate it. On Windows, a `CM_PROB_PHANTOM` display adapter is only a stale registry device, not an active accelerator.
- At 231K–262K context on one 5090, remediate system-RAM spill in this order: set parallel predictions to `1`; reduce KV precision from Q8 to Q4 if needed; reduce eval batch/workspace; lower context only last, and never below the main model's effective compression threshold without explicitly accepting earlier compaction.
- General Qwen3.6 configuration details remain in Apple Notes #272.

### 8-12GB VRAM
- **Qwen3 8B abliterated** at Q4_K_M (~5GB) — best budget pick
- **Qwen3.5 9B** at Q4_K_M (~6GB) — stronger than Qwen3 8B
- **Dolphin 3.0 Llama 3.1 8B** — reliable creative writing pick

## Dense vs MoE: which architecture for Hermes agent work

**Rule:** For a primary Hermes brain, dense models consistently outperform MoE in the same family on agentic benchmarks. The MoE speed advantage comes from activating fewer parameters per token — that's throughput, not intelligence.

| Benchmark | Qwen3.6-27B (dense) | Qwen3.6-35B-A3B (MoE) | Delta |
|-----------|---------------------|----------------------|-------|
| SWE-bench Verified | 77.2 | 73.4 | +3.8 |
| Terminal-Bench 2.0 | 59.3 | 51.5 | +7.8 |
| SkillsBench (coding agent) | 48.2 | 28.7 | **+19.5** |
| MMLU-Pro | 86.2 | 85.2 | +1.0 |
| GPQA Diamond | 87.8 | 86.0 | +1.8 |
| LiveCodeBench v6 | 83.9 | 80.4 | +3.5 |
| BenchLM aggregate | **71** | 62 | **+9** |

The MoE model (35B total, ~3B active/token) is 3-4x faster in token generation but loses quality across the board. The dense model (27B all active) processes more parameters per token — slower but smarter. For Hermes agent work where tool calling reliability, multi-step reasoning, and coding matter: **dense wins**.

MoE excels at throughput-heavy tasks: RAG pipelines, serving many concurrent requests, or when speed is the primary constraint and quality is acceptable at a lower tier. Dense is the right pick for your primary agent brain on hardware that can run it comfortably.

## Quant selection guide

| Quant | BF16 % | Size vs BF16 | Best for |
|-------|--------|-------------|----------|
| Q4_K_M | ~94-95% | ~25% | Budget, max context headroom |
| Q5_K_M | ~96% | ~30% | Good balance |
| Q6_K | ~97.4% | ~35% | Minimum for serious agent work |
| Q8_0 | ~98.5% | ~50% | Best quality, tight on VRAM |

**Community rule (r/LocalLLaMA, May 2026):** "Q6 really is the minimum. Q4_K_M amplifies a bias invisible at Q6+. Q4 looks fine on single-shot benchmarks but starts losing coherence on sequential tool calling."

**For Hermes agent workloads with heavy multi-tool use (long context, frequent tool calls, config edits, JSON schemas):**
- Treat **Q6_K as preferred** for the qwen3.6-27b-heretic family and similar 27B-class abliterated models — not just a single variant like neo-code-di-imatrix-max.
- **Q5_K_M is the practical floor** for agentic use in this class; going below it increases silent hallucinations, schema drift, and instruction loss across long sessions.
- Code-heavy / heavily finetuned variants (e.g., neo-code-di-imatrix-max) are more sensitive to quant degradation — Q6_K is strongly recommended there.

**Exception:** MoE models (GPT-OSS 20B, Gemma 26B-A4B) tolerate lower quants better. IQ4_NL on GPT-OSS 20B is described as "wild, off-the-cuff" but still functional.

### Narrow Q4 exception: cloud-orchestrated local delegates

Q4_K_M is still not the default recommendation for a standalone primary Hermes brain. It can be rational for a **bounded local delegated worker** when a stronger cloud flagship owns decomposition, architecture, integration, and final verification.

Require all of the following:

- The Q4 worker receives scoped implementation/test tasks rather than final authority.
- Saved VRAM produces a measured benefit such as a second parallel slot or removal of system-RAM spill.
- `delegation.max_concurrent_children` matches the runtime's live `parallel` count.
- Thinking stays enabled for coding/tool loops.
- The cloud main verifies actual diffs and test output, not only the delegate summary.
- A reliable cloud delegation fallback exists.
- N simultaneous tool-call smokes succeed for N configured slots.

If concurrency is not materially valuable, keep Q6/Q8. For the decision matrix, mmproj guidance, and a verified Qwen-Q4 + Gemma-Q6 example, read `references/cloud-orchestrated-q4-delegates.md`. Routing mechanics belong in the model-switching guidance.

## KV cache quantization

When the user asks about KV cache precision (Q8 vs Q4), or about splitting K-cache vs V-cache:

- Most backends (llama.cpp, LM Studio) use a single global KV quant; they do not expose independent K vs V settings in stable form today.
- Keys tolerate aggressive quant better than values; hurting V tends to degrade quality faster.
- Q8_0 KV is the default recommendation for agent work: closest thing to "free lunch."
- Q4_0 KV is acceptable when:
 - You are close to VRAM limits with high-weight quants (e.g., Q8_0 weights), or
 - You need very long context and would otherwise spill into system RAM.

Hard rule: avoid CPU/system-RAM spillover at all costs for agent workloads. A second GPU over PCIe is preferable to offloading layers/KV cache into main RAM; token speed drops hard and latency spikes once you hit CPU memory.

If Q8_0 weights + Q8_0 KV would cause LM Studio to offload:
- Prefer either:
 - Using both GPUs in the same LM Studio instance to keep everything on VRAM, or
 - Dropping KV cache to Q4_0 so the model fits fully on the primary GPU.

Asymmetric K=Q8 / V=Q4 is mostly research-level and not reliably exposed; don't recommend it as a practical solution unless there's clear backend support.

## Dual-GPU usage guidance (single model across GPUs)

When the user has two GPUs and a single large model that doesn't fit comfortably on one:

- Using both GPUs in the same LM Studio instance is appropriate when:
 - The model (including context) would otherwise spill into system RAM, or
 - They want to maximize context length at higher quants.
- For models that comfortably fit on the primary GPU at the **intended loaded context** (for example, Qwen3.6-27B Q6/Q8 at ordinary 16K–64K contexts on a 5090):
 - Prefer running them on the primary GPU only; avoid PCIe overhead and keep latency low.
- Do not generalize that fit claim to 231K–262K compression loads. At very long context, KV cache changes the answer; run the architecture-specific memory calculation and inspect real runtime allocation.

Important: LM Studio auto-balances across GPUs — it does NOT let you pin one model to one GPU.
- The "GPU priority" UI (drag to reorder) only affects allocation order, not hard isolation.
- You cannot guarantee that model A runs solely on GPU0 while model B runs solely on GPU1 in a single LM Studio instance.
- If strict per-GPU separation is required, use different runtimes or environment variables (e.g., llama-server with CUDA_VISIBLE_DEVICES).

If they also want a secondary model:
- Best practice is to dedicate one GPU/endpoint per role:
 - Primary Hermes model → dedicated endpoint (e.g., LM Studio or Unsloth on GPU0).
 - Secondary/auxiliary model → separate instance on GPU1, or same LM Studio only when needed and not both heavy models hot at once.

## Thinking mode for agent workloads

When the user asks whether to enable/disable "thinking" or internal reasoning:

- Default stance: thinking must stay enabled for Hermes agent use.
- It is used to plan multi-step tasks, manage tool calls, and maintain consistency across long loops.
- Disabling it noticeably:
 - Increases hallucinations in complex chains,
 - Weakens instruction adherence,
 - Makes tool-calling more brittle.
- Only consider disabling for simple, single-shot tasks where you want faster/shorter answers—and even then the gain is small compared to risk.

### Qwen3.6-27B local coding-agent preset

For `qwen3.6-27b-nvfp4-mtp` acting as a Hermes coding/tool agent—not as the top-level cloud orchestrator—start with:

- Thinking: **enabled**
- Temperature: **0.6**
- Top P: **0.90–0.95**
- Top K: **20**

Do not default to temperature 0 for autonomous agent loops; it can make recovery and tool selection overly rigid. If edits wander or tool behavior becomes unstable, lower temperature to 0.3–0.4 before disabling thinking. Thinking-off is reserved for disposable one-shot rewriting, classification, or summaries—not coding, file mutation, terminal work, or verification loops.

## LM Studio prompt ownership when serving Hermes

When LM Studio is acting as Hermes's OpenAI-compatible backend:

- Leave LM Studio's **system prompt blank**. Hermes builds and sends its own dynamic system message containing profile identity, tools, skills, memory, project

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [AtlasOmnia](https://github.com/AtlasOmnia)
- **Source:** [AtlasOmnia/donna-starter](https://github.com/AtlasOmnia/donna-starter)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-atlasomnia-donna-starter-local-model-selection
- Seller: https://agentstack.voostack.com/s/atlasomnia
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.