Llm Server Docs

Local LLaMA Server Setup Documentation

TL;DR: End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS, along with steps for configuring SSH, firewall, and secure remote access via Tailscale.

Software Stack:

• Inference Engine (Ollama, llama.cpp, vLLM)
• Search Engine (SearXNG)
• Model Server (llama-swap, systemd service)
• Chat Platform (Open WebUI)
• MCP Proxy Server (mcp-proxy, MCPJungle)
• Text-to-Speech Server (Kokoro FastAPI)
• Image Generation Server (ComfyUI)

Table of Contents

• [Local LLaMA Server Setup Documentation](#local-llama-server-setup-documentation)
• [Table of Contents](#table-of-contents)
• [About](#about)
• [Priorities](#priorities)
• [Prerequisites](#prerequisites)
• [General](#general)
• [Allow sudo Permissions](#allow-sudo-permissions)
• [Update System Packages](#update-system-packages)
• [Schedule Startup Script](#schedule-startup-script)
• [Configure Script Permissions](#configure-script-permissions)
• [Configure Auto-Login (optional)](#configure-auto-login-optional)
• [Docker](#docker)
• [Add User to Docker Group](#add-user-to-docker-group)
• [Nvidia Container Toolkit](#nvidia-container-toolkit)
• [Create a Network](#create-a-network)
• [Harden Docker Containers](#harden-docker-containers)
• [Helpful Commands](#helpful-commands)
• [HuggingFace CLI](#huggingface-cli)
• [Manage Models](#manage-models)
• [Download Models](#download-models)
• [Delete Models](#delete-models)
• [Search Engine](#search-engine)
• [SearXNG](#searxng)
• [Open WebUI Integration](#open-webui-integration)
• [Inference Engine](#inference-engine)
• [Ollama](#ollama)
• [llama.cpp](#llamacpp)
• [vLLM](#vllm)
• [Open WebUI Integration](#open-webui-integration-1)
• [Ollama vs. llama.cpp](#ollama-vs-llamacpp)
• [vLLM vs. Ollama/llama.cpp](#vllm-vs-ollamallamacpp)
• [Model Server](#model-server)
• [llama-swap](#llama-swap)
• [systemd Service](#systemd-service)
• [Open WebUI Integration](#open-webui-integration-2)
• [llama-swap](#llama-swap-1)
• [systemd Service](#systemd-service-1)
• [Chat Platform](#chat-platform)
• [Open WebUI](#open-webui)
• [MCP Proxy Server](#mcp-proxy-server)
• [mcp-proxy](#mcp-proxy)
• [MCPJungle](#mcpjungle)
• [Comparison](#comparison)
• [Open WebUI Integration](#open-webui-integration-3)
• [mcp-proxy](#mcp-proxy-1)
• [MCPJungle](#mcpjungle-1)
• [VS Code/Claude Desktop Integration](#vs-codeclaude-desktop-integration)
• [Text-to-Speech Server](#text-to-speech-server)
• [Kokoro FastAPI](#kokoro-fastapi)
• [Open WebUI Integration](#open-webui-integration-4)
• [Image Generation Server](#image-generation-server)
• [ComfyUI](#comfyui)
• [Open WebUI Integration](#open-webui-integration-5)
• [SSH](#ssh)
• [Firewall](#firewall)
• [Remote Access](#remote-access)
• [Tailscale](#tailscale)
• [Installation](#installation)
• [Exit Nodes](#exit-nodes)
• [Local DNS](#local-dns)
• [Third-Party VPN Integration](#third-party-vpn-integration)
• [Updating](#updating)
• [General](#general-1)
• [Nvidia Drivers \& CUDA](#nvidia-drivers--cuda)
• [Ollama](#ollama-1)
• [llama.cpp](#llamacpp-1)
• [vLLM](#vllm-1)
• [llama-swap](#llama-swap-2)
• [Open WebUI](#open-webui-1)
• [mcp-proxy/MCPJungle](#mcp-proxymcpjungle)
• [Kokoro FastAPI](#kokoro-fastapi-1)
• [ComfyUI](#comfyui-1)
• [Troubleshooting](#troubleshooting)
• [Docker](#docker-1)
• [ssh](#ssh-1)
• [Nvidia Drivers](#nvidia-drivers)
• [Ollama](#ollama-2)
• [vLLM](#vllm-2)
• [Open WebUI](#open-webui-2)
• [Monitoring](#monitoring)
• [Notes](#notes)
• [Software](#software)
• [Hardware](#hardware)
• [References](#references)
• [Acknowledgements](#acknowledgements)

About

This repository outlines the steps to run a server for running local language models. It uses Debian specifically, but most Linux distros should follow a very similar process. It aims to be a guide for Linux beginners like me who are setting up a server for the first time.

The process involves installing the requisite drivers, setting the GPU power limit, setting up auto-login, and scheduling the init.bash script to run at boot. All these settings are based on my ideal setup for a language model server that runs most of the day but a lot can be customized to suit your needs.

> [!IMPORTANT] > No part of this guide was written using AI - any hallucinations are the good old human kind. While I've done my absolute best to ensure correctness in every step/command, check everything you execute in a terminal. Enjoy!

Priorities

• Simplicity: It should be relatively straightforward to set up the components of the solution.
• Stability: The components should be stable and capable of running for weeks at a time without any intervention necessary.
• Security: The components should be able to be tightly secured and limited in their capability to damage the system in case of a known vulnerability affecting any of the components.
• Maintainability: The components and their interactions should be uncomplicated enough that you know enough to maintain them as they evolve (because they will evolve).
• Aesthetics: The result should be as close to a cloud provider's chat platform as possible. A homelab solution doesn't necessarily need to feel like it was cobbled together haphazardly.
• Modularity: Components in the setup should be able to be swapped out for newer/more performant/better maintained alternatives easily. Standard protocols (OpenAI-compatibility, MCPs, etc.) help with this a lot and, in this guide, they are always preferred over bundled solutions.
• Open source: The code should be able to be verified by a community of engineers. Chat platforms and LLMs involve large amounts of personal data conveyed in natural language and it's important to know that data isn't going outside your machine.

Prerequisites

Any modern CPU and GPU combination should work for this guide. Previously, compatibility with AMD GPUs was an issue but the latest releases of Ollama have worked through this and AMD GPUs are now supported natively.

For reference, this guide was built around the following system:

• CPU: Intel Core i5-12600KF
• Memory: 96GB 3200MHz DDR4 RAM
• Storage: 1TB M.2 NVMe SSD
• GPU: 2x Nvidia RTX 3090 (24GB)

> [!NOTE] > AMD GPUs: Power limiting is skipped for AMD GPUs as AMD has recently made it difficult to set power limits on their GPUs. Naturally, skip any steps involving nvidia-smi or nvidia-persistenced and the power limit in the init.bash script. > > CPU-only: You can skip the GPU driver installation and power limiting steps. The rest of the guide should work as expected.

> [!NOTE] > This guide uses ~/ (or /home/) as the base directory. If you're working in different directory, please modify all your commands accordingly.

To begin the process of setting up your server, you will need the following:

• Fresh install of Debian
• Internet connection
• Basic understanding of the Linux terminal
• Peripherals like a monitor, keyboard, and mouse

To install Debian on your newly built server hardware:

• Download the Debian ISO from the official website.
• Create a bootable USB using a tool like Rufus for Windows or Balena Etcher for MacOS.
• Boot into the USB and install Debian.

For a more detailed guide on installing Debian, refer to the official documentation. For those who aren't yet experienced with Linux, I recommend using the graphical installer - you will be given an option between the text-based installer and graphical installer.

I also recommend installing a lightweight desktop environment like XFCE for ease of use. Other options like GNOME or KDE are also available - GNOME may be a better option for those using their server as a primary workstation as it is more feature-rich (and, as such, heavier) than XFCE.

General

Allow sudo Permissions

To do a bunch of things in this guide, we need to be root. But we don't have the ability to act as root until root allows our user to. First, we'll switch to root and grant our user permission to run commands with sudo.

> [!TIP] > sudo stands for "superuser doer" - in Linux, it signals to the OS that you want the command you're running to be running on behalf of the root user. This should be used sparingly on a highly secure system (we want user-specific permissions for most processes) and carefully since it can affect the system with the same power as the root user. Don't overthink its use or worry when using it - just know that it can be dangerous if used incorrectly.

• Switch to root:

``bash su root ``

• Run the following command to add your user to the sudo group (which has the permissions we're looking for):

``bash sudo usermod -a -G sudo ` > Replace ` with your username.

Save and exit (Ctrl+X).

• Close your existing terminal and open a new session. This is required to see the changes.
• (Optional) Test your new permissions by running ls (command to view the contents of a directory) with sudo:

``bash sudo ls ``

Update System Packages

• Update the system by running the following commands:

`` sudo apt update sudo apt upgrade ``

Now, we'll install the required GPU drivers that allow programs to utilize their compute capabilities.

Nvidia GPUs

• Follow Nvidia's guide on downloading CUDA Toolkit. The instructions are specific to your machine and the website will lead you to them interactively.
• Run the following commands:

``bash sudo apt install linux-headers-amd64 sudo apt install nvidia-driver firmware-misc-nonfree ``

• Reboot the server.
• Run the following command to verify the installation:

``bash nvidia-smi ``

AMD GPUs

• Run the following commands:

``bash deb http://deb.debian.org/debian bookworm main contrib non-free-firmware apt install firmware-amd-graphics libgl1-mesa-dri libglx-mesa0 mesa-vulkan-drivers xserver-xorg-video-all ``

• Reboot the server.

We'll also install some packages that are not installed on Debian by default but may be required later:

sudo apt install libcurl cmake

Schedule Startup Script

In this step, we'll create a script called init.bash. This script will be run at boot to set the GPU power limit and start the server using Ollama. We set the GPU power limit lower because it has been seen in testing and inference that there is only a 5-15% performance decrease for a 30% reduction in power consumption. This is especially important for servers that are running 24/7.

• Run the following commands:

``bash touch init.bash nano init.bash ``

• Add the following lines to the script:

``bash #!/bin/bash sudo nvidia-smi -pm 1 sudo nvidia-smi -pl ` > Replace with the desired power limit in watts. For example, sudo nvidia-smi -pl 250`.

For multiple GPUs, modify the script to set the power limit for each GPU: ``bash sudo nvidia-smi -i 0 -pl sudo nvidia-smi -i 1 -pl ``

• Save and exit the script.
• Make the script executable:

``bash chmod +x init.bash ``

Adding the init.bash script to the crontab will schedule it to run at boot.

• Run the following command:

``bash crontab -e ``

• Add the following line to the file:

``bash @reboot /path/to/init.bash ` > Replace /path/to/init.bash with the path to the init.bash` script.

• (Optional) Add the following line to shutdown the server at 12am:

``bash 0 0 * * * /sbin/shutdown -h now ``

• Save and exit the file.

Configure Script Permissions

We want init.bash to run the nvidia-smi commands without having to enter a password. This is done by giving nvidia-persistenced and nvidia-smi passwordless sudo permissions, and can be achieved by editing the sudoers file.

AMD users can skip this step as power limiting is not supported on AMD GPUs.

• Run the following command to edit the sudoers file:

``bash sudo visudo ``

• Add the following lines to the file:

`` ALL=(ALL) NOPASSWD: /usr/bin/nvidia-persistenced ALL=(ALL) NOPASSWD: /usr/bin/nvidia-smi ` > Replace ` with your username.

• Save and exit the file.

> [!IMPORTANT] > Ensure that you add these lines AFTER %sudo ALL=(ALL:ALL) ALL. The order of the lines in the file matters - the last matching line will be used so if you add these lines before %sudo ALL=(ALL:ALL) ALL, they will be ignored.

Configure Auto-Login (optional)

When the server boots up, we may want it to automatically log in to a user account and run the init.bash script. This is done by configuring the lightdm display manager.

• Run the following command:

``bash sudo nano /etc/lightdm/lightdm.conf ``

• Find the following commented line. It should be in the [Seat:*] section.

`` # autologin-user= ``

• Uncomment the line and add your username:

`` autologin-user= ` > Replace ` with your username.

• Save and exit the file.

Docker

📖 Documentation

Docker is a containerization platform that allows you to run applications in isolated environments. This subsection follows Docker's guide to install Docker Engine on Debian. The commands are listed below, but visiting the guide is recommended in case instructions have changed.

• If you already have a Docker installation on your system, it's a good idea to re-install so there are no broken/out-of-date dependencies. The command below will iterate through your system's installed packages and remove the ones associated with Docker.

``bash for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt purge $pkg; done ``

• Run the following commands:

```bash # Add Docker's official GPG key: sudo apt update sudo apt install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources: sudo tee /etc/apt/sources.list.d/docker.sources [!CAUTION] > Being in the docker group grants significant system access - effectively root-equivalent privileges to the Docker daemon. Only add trusted users to this group.

1. Add your user to the Docker group:

``bash sudo usermod -aG docker $USER ``

1. Apply the changes:

``bash newgrp docker ``

1. Verify your new permissions by checking running containers:

``bash docker ps -a ``

If you encounter "permission denied" and Docker commands still require sudo, log out and log back i

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: varunvasudeva1
• Source: varunvasudeva1/llm-server-docs
• License: MIT

Install and usage instructions live in the source repository linked above.