Ollama Guide: Run DeepSeek-R1 and Llama 3 Locally Offline

Entity Definition: What is Ollama?

Ollama (ollama/ollama) is an open-source runtime and packaging framework built in Go and C++ that enables developers to download, manage, customise, and serve open-weight Large Language Models (LLMs) locally. By bundling C++ inference backends (primarily llama.cpp) into a clean CLI and REST API daemon, Ollama allows high-performance inference of models like DeepSeek-R1, Llama 3.3, and Qwen 2.5 on macOS, Linux, and Windows without sending data to external cloud APIs.

---

The Problem: Cloud Dependence and Local Inference Friction

Sending every prompt to cloud endpoints is expensive, raises privacy issues, and breaks down completely when working offline. For months, local inference meant wrestling with raw C++ compilations, manually mapping GGUF tensor files, and configuring hardware acceleration flags by hand.

Ollama fixes this by applying Docker-like ergonomics to LLM management. It abstracts hardware detection, model weight downloading, quantisation mapping, and GPU offloading into simple commands.

+------------------------------------------------------------------+
|                           YOUR APPLICATION                       |
|         (Python / Web UI / LangChain / OpenAI SDK Client)       |
+------------------------------------------------------------------+
                                 |
                     OpenAI-Compatible REST API
                       (http://localhost:11434)
                                 |
+------------------------------------------------------------------+
|                          OLLAMA DAEMON                           |
|  +--------------------+  +--------------------+  +------------+  |
|  |  Model Manager     |  |  Modelfile Parser  |  | API Server |  |
|  +--------------------+  +--------------------+  +------------+  |
+------------------------------------------------------------------+
                                 |
                    Inference Abstraction Engine
                                 |
+------------------------------------------------------------------+
|                   LLAMA.CPP INFERENCE ENGINE                     |
|         (GGUF Quantised Weights & Tensor Parallelism)            |
+------------------------------------------------------------------+
                                 |
                 Hardware Acceleration Layer
      [ Apple Metal ]    [ NVIDIA CUDA ]    [ AMD ROCm / CPU ]

---

Key Architectural Details

1. Integrated llama.cpp Backend: Ollama embeds a modified version of llama.cpp for core tensor computation. It handles GGUF quantisation formats, dynamically offloading model layers to available VRAM while leaving overflow tensors in system RAM.

2. Unified REST API & OpenAI Emulation: Ollama runs a lightweight background daemon that serves native endpoints alongside an OpenAI-compatible interface (/v1/chat/completions). Existing tools built for cloud endpoints work locally by changing the base_url.

3. Modelfile Paradigm: Borrowing syntax from Dockerfile, Ollama introduces the Modelfile. This allows developers to declare base weights, set system instructions, configure context window sizes (num_ctx), and lock temperature parameters in a single file.

---

Feature Matrix: Local vs Cloud Workflows

Feature	Cloud API Providers	Raw llama.cpp	Ollama
Data Privacy	Zero (Processed externally)	100% Local	100% Local
Setup Complexity	Low (Requires API Key)	High (Manual build & flags)	Minimal (Single binary / CLI)
Model Management	Managed by Vendor	Manual file downloads	Built-in registry (ollama pull)
API Interface	Vendor Proprietary	Custom bindings / Server	OpenAI-compatible REST API
Hardware Optimisation	Handled by Server	Manual compilation flags	Automatic (Metal, CUDA, ROCm)

---

Quickstart Setup and Installation

1. Installation

macOS & Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download the native installer directly from the official Ollama repository release page.

2. Fetch and Run a Model

Run a compact open-source model directly from the command line:

# Pull and chat with DeepSeek-R1 (8B reasoning model)
ollama run deepseek-r1:8b

To list active models loaded in VRAM:

ollama ps

---

Customising Models with a Modelfile

You can build tailored agents locally by writing a Modelfile. This example creates a deterministic, concise Python refactoring assistant.

Create a file named Modelfile:

FROM qwen2.5:14b

# Set higher context window (8k tokens)
PARAMETER num_ctx 8192

# Lower temperature for deterministic coding tasks
PARAMETER temperature 0.2

# Set custom system prompt
SYSTEM """
You are a senior Python software engineer specializing in PEP 8 compliance and type hinting.
Respond strictly with optimised code blocks and minimal commentary.
"""

Build and run the custom model:

# Build the model container
ollama create py-refactor -f ./Modelfile

# Run your new custom agent
ollama run py-refactor "def add(a, b): return a+b"

---

Programmatic Integration (Python & OpenAI SDK)

Because Ollama exposes an OpenAI-compatible endpoint, switching from cloud LLMs to local models requires only updating the host URL.

import openai

# Point client to the local Ollama instance
client = openai.OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Required by SDK, ignored by local daemon
)

response = client.chat.completions.create(
    model="py-refactor",
    messages=[
        {"role": "user", "content": "Write an async function to fetch JSON from an URL."}
    ]
)

print(response.choices[0].message.content)

---

Developer Consensus & Real-World Hardware Insights

Hardware benchmarking discussions across GitHub threads and developer channels highlight clear resource trade-offs:

• VRAM is King: While Ollama can split tensors between VRAM and system RAM, running entirely within GPU VRAM provides significantly higher token generation rates.
• Quantisation Standards: The community consensus favours Q4_K_M (4-bit medium quantisation) GGUF weights. This balances minimal perplexity degradation with a roughly 60% reduction in memory overhead compared to 16-bit precision.
• Apple Silicon Efficiency: On M-series Macs, unified memory allows models like llama3.3:70b (quantised to 4-bit) to run smoothly across system RAM, bypassing traditional consumer VRAM limits.

---

Why Ollama Belongs in Your Stack

Ollama bridges the gap between raw open-weight LLM binaries and developer-ready infrastructure. It provides full control over your inference pipeline, eliminates cloud API fees, keeps sensitive codebases strictly offline, and standardises local model deployment through a clean, containerised approach.

• Repository: github.com/ollama/ollama
• Licence: MIT