How do you run an LLM locally with Ollama?

Ollama lets you download and run open-weight LLMs like LLaMA 3, Mistral, and Gemma on your own machine. It exposes an OpenAI-compatible API so your existing code works without changes.

AILLM

Overview

Cloud LLM APIs are convenient but come with cost, latency, data privacy, and rate limit concerns. Ollama is an open-source tool that makes it trivially easy to run capable open-weight models locally. It handles model downloading, GPU/CPU inference, and exposes an API that is compatible with the OpenAI SDK so you can switch between local and cloud models with one line change.

Key Points

•Install Ollama from ollama.com, then pull a model with `ollama pull llama3.2` - that is the entire setup.
•Models run on CPU but are much faster on a GPU. A modern MacBook M-series chip handles 7B models at 30+ tokens/second.
•Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1 - point the OpenAI SDK at it.
•Popular open models: LLaMA 3.2 (Meta), Mistral (Mistral AI), Gemma 2 (Google), Phi-3 (Microsoft), Qwen2.5.
•Model sizes: 3B (fast, lower quality), 7B (good balance), 13B (better, needs 16GB RAM), 70B+ (best, needs GPU).
•Great for: local development, privacy-sensitive data, cost-free experimentation, offline use.

Examples

Install and run a model (terminal)

bash

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model (downloads once, cached locally)
ollama pull llama3.2          # 3B model, ~2GB, fast on any machine
ollama pull mistral           # 7B model, ~4GB, great quality
ollama pull qwen2.5-coder:7b  # code-optimised, excellent for dev tasks

# Chat directly in the terminal
ollama run llama3.2

# List models you have downloaded
ollama list

# Ollama runs as a background service, API available at:
# http://localhost:11434

Use Ollama with the OpenAI SDK (zero code changes)

typescript

import OpenAI from 'openai';

// Just change the baseURL - rest of your code is identical
const local = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey:  'ollama',  // Ollama does not need a real key
});

const response = await local.chat.completions.create({
  model: 'llama3.2',   // must match a model you have pulled
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user',   content: 'Explain async/await in JavaScript.' },
  ],
});

console.log(response.choices[0].message.content);

Switch between local and cloud with an environment variable

typescript

import OpenAI from 'openai';

const isLocal = process.env.LLM_PROVIDER === 'local';

const client = new OpenAI(
  isLocal
    ? { baseURL: 'http://localhost:11434/v1', apiKey: 'ollama' }
    : { apiKey: process.env.OPENAI_API_KEY }
);

const MODEL = isLocal ? 'llama3.2' : 'gpt-4o-mini';

// The rest of your application code is unchanged
const response = await client.chat.completions.create({
  model:    MODEL,
  messages: [{ role: 'user', content: userMessage }],
});

Modelfile: create a custom model with a persona

bash

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM llama3.2

SYSTEM """
You are Alex, a friendly TypeScript tutor.
Always include a short code example in your answer.
Keep explanations under 100 words.
"""

PARAMETER temperature 0.4
PARAMETER num_ctx 8192
EOF

# Build the custom model
ollama create ts-tutor -f Modelfile

# Run it
ollama run ts-tutor "What is a union type?"

Pro Tip

Use `qwen2.5-coder:7b` or `deepseek-coder-v2` for coding tasks - they are specifically trained on code and outperform the general-purpose models of the same size. For a chat assistant, `llama3.2:3b` is fast enough for real-time responses on almost any laptop while costing nothing to run.