How to Build a Local AI Chatbot with Ollama (No Cloud, No Cost)

Every week there's a new AI tool asking for your credit card. Usage limits, rate limits, API costs that spiral once you go past the free tier — it adds up fast, and it means everything you type goes through someone else's servers.

Ollama is a different approach entirely. It lets you download open-source language models and run them directly on your laptop or desktop, with a dead-simple command-line interface and an optional web UI that looks almost identical to ChatGPT. No account. No key. Nothing sent anywhere.

This guide walks you through the whole setup from zero — installing Ollama, choosing and pulling a model, adding a proper browser-based chat interface, and optionally connecting it to Python so you can build on top of it.

What is Ollama and why does it matter?

Ollama is an open-source tool that packages large language models into a simple runtime you can install like any other application. Under the hood it handles all the heavy lifting: downloading model weights in a standardized format (.gguf), loading them efficiently into memory, and exposing a local HTTP API that's compatible with the OpenAI API format.

That last part is more important than it sounds. Because Ollama mimics the OpenAI API interface, any tool or library that works with ChatGPT can be pointed at your local Ollama instance instead — just change the base URL. Libraries like openai for Python, LangChain, LlamaIndex, and dozens of others work out of the box.

What hardware do you actually need?

This is the question everyone asks first — and the honest answer is: probably less than you think. Ollama runs on CPU if it has to, but a dedicated GPU makes a significant difference in speed.

Mac users with Apple Silicon (M1, M2, M3, M4) are in a sweet spot — the unified memory architecture means the GPU and CPU share the same RAM pool, so a 16 GB M2 MacBook can run a 13B model comfortably at very good speed.

Step 1 — Install Ollama

Installation is genuinely one command on Mac and Linux. Windows support is now stable too.

macOS and Linux

curl -fsSL https://ollama.com/install.sh | sh

This downloads the binary, installs it to /usr/local/bin, and starts the Ollama service automatically. On macOS it also adds an icon to your menu bar.

Windows

Download the installer from ollama.com/download and run it. It installs like any Windows application and starts a background service. NVIDIA GPU support works out of the box if your drivers are up to date.

Verify it's running

ollama --version
# ollama version 0.x.x

# Or check the API directly
curl http://localhost:11434
# Ollama is running

Step 2 — Pull and run your first model

With Ollama running, you can pull any model from the Ollama library with a single command. Let's start with Llama 3.2 — Meta's latest small model, fast and capable even on modest hardware.

# Pull the model (downloads ~2 GB)
ollama pull llama3.2

# Start chatting immediately in the terminal
ollama run llama3.2

After the download completes, you'll get an interactive terminal prompt where you can type messages directly. It's a quick way to verify everything is working before you set up the browser interface.

>>> Tell me a fun fact about the ocean
The deepest point in the ocean is the Challenger Deep in the 
Mariana Trench, reaching approximately 36,000 feet (11,000 meters)...

>>> /bye   # exits the chat

You can also use the REST API directly — Ollama exposes an endpoint athttp://localhost:11434:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "What is the capital of Morocco?",
  "stream": false
}'

Step 3 — Add a chat UI with Open WebUI

The terminal chat is fine for testing, but for real use you want a proper browser interface. Open WebUI is the best option available right now — it's a polished, feature-rich front end that connects to Ollama automatically, looks nearly identical to ChatGPT, and runs entirely locally via Docker.

Option A: Docker (recommended)

If you have Docker installed, this is one command:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Open your browser at http://localhost:3000. You'll be asked to create a local admin account (stored only on your machine), and then you're in. Select your model from the dropdown at the top and start chatting.

Option B: pip (no Docker required)

pip install open-webui
open-webui serve

Same result — the UI runs at http://localhost:8080. The pip version is slightly easier to install but Docker gives you better isolation and easier updates.

Step 4 — Talk to it from Python (optional)

If you want to build something on top of Ollama — a script, a tool, an agent — the easiest approach is to use the ollama Python library. It wraps the REST API in a clean interface.

pip install ollama

Basic chat

import ollama

response = ollama.chat(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Explain recursion in plain English."}
    ]
)

print(response["message"]["content"])

Streaming responses

import ollama

# Stream the response token by token
for chunk in ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write me a haiku about Python."}],
    stream=True,
):
    print(chunk["message"]["content"], end="", flush=True)

Using it with the OpenAI library

Because Ollama's API is compatible with OpenAI's format, you can use the officialopenai Python library just by changing the base URL — useful if you're migrating code or using a library that expects the OpenAI interface:

from openai import OpenAI

# Point the client at your local Ollama instead of OpenAI
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but ignored locally
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is machine learning?"}],
)

print(response.choices[0].message.content)

Which model should you pick?

Ollama's library has hundreds of models. Here are the most practical choices for 2026, matched to common use cases:

You can have multiple models installed at the same time and switch between them freely. List what you have with ollama list and remove one with ollama rm model-name.

Tips for better performance

Use quantized models

Model files come in different quantization levels — essentially how aggressively the weights are compressed. More compression means smaller size and faster speed at a small quality cost. The default Ollama downloads are usually Q4 (4-bit quantization), which is a good balance. If you have more RAM to spare, try :Q8 variants for better output quality:

# Q4 = smaller, faster (default)
ollama pull llama3.2

# Q8 = larger, slightly better quality
ollama pull llama3.2:latest-q8_0

Set a system prompt with Modelfile

You can create a custom version of any model with a persistent system prompt — useful for keeping it focused on a specific task:

# Create a file called Modelfile
FROM llama3.2

SYSTEM """
You are a senior software engineer who gives concise, 
practical answers. You always show code examples. 
You never give vague or overly long responses.
"""

# Build it as a custom model
ollama create my-dev-assistant -f Modelfile

# Run it
ollama run my-dev-assistant

Keep Ollama running as a service

By default Ollama starts when you launch it. On Linux, to make it start automatically on boot:

sudo systemctl enable ollama
sudo systemctl start ollama

Increase the context window

By default, Ollama uses a 2048-token context window for most models. For longer conversations or documents, increase it via an environment variable:

# In your Modelfile
FROM llama3.2
PARAMETER num_ctx 8192

# Or via API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Your long prompt here...",
  "options": { "num_ctx": 8192 }
}'

FAQ