Ollama Local

ollama-local

Overview

Set up and use Ollama for running large language models locally. Ollama runs models on your machine — no API keys, no cloud, no per-token costs. The agent installs Ollama, pulls models, and shows you how to use them via the REST API or command line.

When to Use

• The user wants to run an LLM locally without paying for API access.
• The user wants privacy — no data leaves their machine.
• The user wants to use a local model with their agent or application.
• The user says "set up Ollama", "run a local LLM", or "I want offline AI".

Installation

Linux

curl -fsSL https://ollama.com/install.sh | sh

macOS

# Via Homebrew
brew install ollama

# Or download from https://ollama.com/download

Windows

Download from https://ollama.com/download and run the installer. Ollama runs as a background service on Windows.

Verify installation

ollama --version
# ollama version is 0.x.x

Model Management

Pull a model

# Small, fast model (good for testing)
ollama pull llama3.2:3b

# Medium model (good balance of speed and quality)
ollama pull llama3.1:8b

# Large model (best quality, needs 16GB+ RAM)
ollama pull llama3.1:70b

# Coding-focused model
ollama pull qwen2.5-coder:7b

# Embedding model
ollama pull nomic-embed-text

List installed models

ollama list

Run a model (interactive chat)

ollama run llama3.1:8b
>>> Tell me about quantum computing

Remove a model

ollama rm llama3.2:3b

API Usage

Ollama exposes a REST API at http://localhost:11434:

Generate a response

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain recursion in one sentence.",
  "stream": false
}'

Chat (multi-turn)

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "user", "content": "What is 2+2?"},
    {"role": "assistant", "content": "4"},
    {"role": "user", "content": "What about 3+5?"}
  ],
  "stream": false
}'

Generate embeddings

curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "The quick brown fox jumps over the lazy dog."
}'

Python client

import requests

response = requests.post('http://localhost:11434/api/generate', json={
    'model': 'llama3.1:8b',
    'prompt': 'Write a haiku about the ocean.',
    'stream': False
})
print(response.json()['response'])

Streaming responses

import requests

response = requests.post('http://localhost:11434/api/generate', json={
    'model': 'llama3.1:8b',
    'prompt': 'Tell me a story.',
    'stream': True
}, stream=True)

for line in response.iter_lines():
    if line:
        import json
        chunk = json.loads(line)
        print(chunk.get('response', ''), end='', flush=True)

Integration with Hermes

Configure Hermes to use the local Ollama instance as a provider:

# Set Ollama as a custom provider
hermes config set model.provider custom
hermes config set model.base_url http://localhost:11434/v1
hermes config set model.api_key ollama  # Ollama doesn't require a real key
hermes config set model.default llama3.1:8b

Or use Ollama for specific tasks (like auxiliary/compression) while keeping a cloud model for main reasoning:

hermes config set auxiliary.compression.provider custom
hermes config set auxiliary.compression.base_url http://localhost:11434/v1
hermes config set auxiliary.compression.model llama3.2:3b

Model Selection Guide

| Model | Size | RAM needed | Best for | |---|---|---|---| | llama3.2:3b | 2 GB | 4 GB | Fast responses, simple tasks | | llama3.1:8b | 5 GB | 8 GB | General purpose, good balance | | qwen2.5-coder:7b | 5 GB | 8 GB | Code generation, debugging | | llama3.1:70b | 40 GB | 64 GB | High quality, complex reasoning | | nomic-embed-text | 0.3 GB | 1 GB | Embeddings for RAG/search |

Performance Tips

• Use GPU if available — Ollama auto-detects NVIDIA/AMD GPUs and Apple Silicon. GPU inference is substantially faster than CPU; measure on your own hardware, since the gap depends on the model, quantisation, and VRAM.
• Match model size to your RAM — A model that doesn't fit in RAM will spill to disk and become extremely slow. Check ollama ps to see if the model is fully in memory.
• Use smaller models for simple tasks — Don't use a 70B model for a one-sentence answer. Use 3B or 8B for quick tasks.
• Keep models loaded — Ollama keeps models in memory for 5 minutes after last use by default. Increase this with OLLAMA_KEEP_ALIVE env var if you're making frequent requests.
• Quantization — Ollama uses 4-bit quantization by default, which reduces memory usage by ~70% with minimal quality loss. No configuration needed.

Common Pitfalls

1. Model larger than available RAM. Ollama will fall back to disk swap and performance becomes unusable rather than failing outright — check ollama ps to confirm the model is fully resident in memory, and use a smaller model or add RAM if not.
2. First pull looks "stuck". The first ollama pull downloads the full model file (multiple GB); this can take minutes on a slow connection. Subsequent runs use the cached model and start instantly — don't kill the process assuming it's hung.
3. Port conflict on 11434. If another service already binds Ollama's default port, the server fails to start silently in some setups. Set OLLAMA_HOST=0.0.0.0:11435 before starting and update client URLs to match.
4. GPU not detected. On Linux, missing NVIDIA drivers/CUDA toolkit means Ollama silently falls back to CPU (much slower) instead of erroring. Verify with nvidia-smi before assuming GPU is in use.
5. Prompts exceed local context comfortably. Local models advertise large context windows (e.g., 128k for Llama 3.1) but running at full context requires far more RAM than the base model size suggests. Keep prompts under ~8k tokens for 8B-class models in practice.
6. Requests queue instead of running in parallel. Ollama processes requests sequentially by default, so concurrent callers block each other. Set OLLAMA_NUM_PARALLEL if concurrency is needed.

Verification Checklist

• [ ] ollama --version succeeds and ollama list shows the pulled model
• [ ] ollama ps confirms the model is loaded and shows a reasonable memory footprint (not swapping)
• [ ] A test curl http://localhost:11434/api/generate call returns a non-empty response field
• [ ] If GPU acceleration was expected, ollama ps or system GPU monitor (nvidia-smi) confirms it's actually being used
• [ ] If wired into Hermes as a provider, hermes config get model.base_url reflects the correct local URL and a real Hermes call round-trips successfully

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: THEROCKSSS
• Source: THEROCKSSS/hermes-skills-portfolio
• License: MIT

Install and usage instructions live in the source repository linked above.