How to Run an LLM Locally in 2026: Tools, Hardware and Models

To run an LLM locally, install a runner such as Ollama or LM Studio, download an open-weight model that fits in your memory, and chat with it or call it through a local API. On macOS or Linux the whole thing is two commands: curl -fsSL https://ollama.com/install.sh | sh, then ollama run gemma4. The real decision is the model: a 4-bit model needs a little over half a gigabyte of RAM or VRAM per billion parameters, plus room for context.

Quick answer: which tool to use

If you are not sure, start with Ollama. If you prefer clicking to typing, start with LM Studio. We compare those two in detail in Ollama vs LM Studio.

Why run an LLM locally?

• Privacy. Prompts, documents and code never leave your machine. Ollama's FAQ says local use isn't visible to Ollama at all.
• No per-token bill. Once you own the hardware, heavy use costs electricity.
• Offline and stable. It works on a plane, and the model only changes when you change it.
• Control. You pick the exact model, quantization, context length and system prompt.

The trade-off is honest: the best closed models still lead on the hardest tasks, and a laptop will not match a data center's speed. Local models are excellent for private drafting, summarizing, coding help, classification and agents that work on your own files. For a sense of where open models sit against the frontier, see our roundup of the best LLMs.

The five main options

Ollama

Ollama is the most widely used runner. It downloads models from its library, manages GPU offload, and exposes a REST API on 127.0.0.1:11434, plus an OpenAI-compatible API at /v1. Its README lists llama.cpp as the supported backend. Recent versions also have ollama launch, which wires local models into coding agents such as Claude Code, Codex and OpenCode.

LM Studio

LM Studio is a desktop app for finding, downloading and chatting with models, with a local server for other apps. It runs GGUF models through llama.cpp on every platform and MLX models on Apple Silicon. It also acts as an MCP client, so local models can use MCP servers. Since 8 July 2025, LM Studio says it is "free to use both at home and at work".

llama.cpp

llama.cpp is the C/C++ engine that made local LLMs practical, and it is what Ollama and LM Studio build on. It supports Apple Silicon ("a first-class citizen"), CUDA, AMD via HIP, Vulkan, SYCL and CPU+GPU hybrid inference for models bigger than your VRAM. The current quick start uses one llama command:

curl -LsSf https://llama.app/install.sh | sh
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF      # chat in the terminal
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF    # OpenAI-compatible server and web UI

vLLM

vLLM is a high-throughput serving engine for production. Its quickstart targets Linux with Python 3.11 to 3.14, installs with uv pip install vllm --torch-backend=auto, and serves an OpenAI-compatible API on port 8000. It shines when several users or agents hit one GPU server. For a single person on a laptop, it is more than you need.

Jan

Jan is an Apache-2.0 desktop app that "runs 100% offline", downloads models from Hugging Face, supports MCP, and offers an OpenAI-compatible server at localhost:1337. It can also connect to cloud providers if you want one app for both.

What hardware do you need?

Memory is the constraint that matters. The model's weights must fit in GPU memory (VRAM) for good speed, or in unified memory on a Mac, with room left for the KV cache that holds your conversation.

The arithmetic

Weights (GB) ≈ parameters (billions) × bits per weight ÷ 8.

llama.cpp publishes the effective bits per weight of its formats: about 4.89 for Q4_K_M, 5.70 for Q5_K_M, 6.56 for Q6_K, 8.50 for Q8_0 and 16 for F16. So a 27B model at Q4_K_M is about 27 × 4.89 ÷ 8 ≈ 16.5 GB. Ollama lists its qwen3.8:27b download at 18 GB, which matches once you add the vision encoder and metadata.

Google's Gemma 4 docs give official weights-only figures:

Two things to notice. First, 4-bit cuts memory to about a quarter of 16-bit, which is why almost everyone runs 4-bit locally. Second, a mixture-of-experts model like the 26B A4B only computes about 4B parameters per token, but you still have to load all 26B.

Then add the KV cache, which grows with context length. Long contexts for agents and coding can add several gigabytes. Our local LLM hardware guide works through the KV cache formula and a full sizing table.

Rough tiers

GPU or Mac?

NVIDIA GPUs are the default on Windows and Linux. Ollama supports cards with compute capability 5.0 or higher. Consumer cards top out at 32 GB (RTX 5090) and many popular ones have 16 GB, so VRAM, not speed, decides what you can run.

Apple Silicon Macs share one pool of unified memory between CPU and GPU, so a Mac with 64 GB can give a model far more memory than most single consumer GPUs. Apple's current Mac Studio offers up to 128 GB with M5 Max and up to 512 GB with M5 Ultra. LM Studio requires Apple Silicon and macOS 14 or newer and recommends 16 GB or more.

AMD GPUs work through ROCm on Linux and on selected cards on Windows, and Vulkan covers more hardware in Ollama and llama.cpp.

Do you need a GPU?

No. Small models run on a CPU, and llama.cpp is heavily optimized for it. But generation slows as models grow, so a CPU-only machine is best kept to models of a few billion parameters. Check what Ollama is actually using with ollama ps: the Processor column shows 100% GPU, 100% CPU or a split. A split means part of the model spilled into system RAM and will be much slower.

Step by step: run an LLM locally with Ollama

1. Install Ollama

# macOS and Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows (PowerShell)
irm https://ollama.com/install.ps1 | iex

There are also manual installers (Ollama.dmg, OllamaSetup.exe) on ollama.com/download, and an official ollama/ollama Docker image.

2. Run your first model

ollama run gemma4

The first run downloads the default tag (gemma4:latest, the E4B model, 6.6 to 9.5 GB depending on variant) and opens a chat. Pick a size explicitly with a tag, for example ollama run gemma4:12b or ollama run qwen3.8:27b. Running plain ollama opens a menu that can also connect Ollama to agents and apps.

3. Check it is on the GPU

ollama ps

If you see a CPU/GPU split, choose a smaller model or a smaller quantization.

4. Set a sensible context length

Ollama's default context depends on VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more. Its docs say agents, web search and coding tools "should be set to at least 64000 tokens". Set it when starting the server:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Bigger context costs memory. To save some, Ollama can quantize the KV cache with OLLAMA_KV_CACHE_TYPE=q8_0 (about half the memory of the default f16) when flash attention is on.

5. Call it from code

Native API:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{"role": "user", "content": "Summarize RAG in one sentence."}],
  "stream": false
}'

Or point any OpenAI SDK at Ollama:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
reply = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Write a haiku about VRAM."}],
)
print(reply.choices[0].message.content)

The API key is required by the SDK but ignored locally. There are also official ollama packages for Python and JavaScript.

6. Know where things live

Models are stored in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows. Set OLLAMA_MODELS to move them to a bigger disk. Models unload after 5 minutes idle by default.

Keep the server bound to 127.0.0.1 unless you mean to share it. If you expose it on your network with OLLAMA_HOST, put authentication or a firewall in front of it.

Which models to start with

The Ollama library's most-pulled models are still older families (Llama 3.1, DeepSeek-R1, Qwen2.5) because of years of downloads. For new setups in October 2026, start with current models that fit your memory:

For coding specifically, see best local LLM for coding. For embeddings in a local RAG pipeline, the library's top embedding models include nomic-embed-text, mxbai-embed-large and bge-m3.

Do your own evaluation: pick three real tasks from your work, run each candidate, and compare quality and speed on your hardware. That beats any leaderboard.

Can you run Claude or ChatGPT locally?

Claude: no. Claude is a closed-weight model; Anthropic serves it through its apps, API and cloud partners and does not publish the weights. What you can do is use Anthropic's Claude Code tool with a local model. Ollama documents ollama launch claude, or setting ANTHROPIC_BASE_URL=http://localhost:11434 with a tool-capable model and at least 64k context. The interface is Claude Code; the model is whatever open model you run. The same goes for ChatGPT, although OpenAI does publish the separate open-weight gpt-oss models, which you can run.

Troubleshooting

• Very slow replies: run ollama ps. If part of the model is on the CPU, use a smaller model or quantization, or a shorter context.
• Out of memory on load: the model plus context does not fit. Drop to a 4-bit tag or reduce OLLAMA_CONTEXT_LENGTH.
• Agent forgets instructions halfway: the context is probably at the 4k default. Raise it.
• Tool calls fail: use a model the library marks with the tools capability.
• GPU not detected on Linux after sleep: Ollama's docs suggest reloading the nvidia_uvm module.

Once a model runs, the next step is usually connecting it to your files and tools. Read what is MCP to see how local models can use the same tool servers as cloud ones.