← Back to all spotlights

Tinygrad: The Minimalist Deep Learning Framework

Explore tinygrad, George Hotz's lightweight neural network compiler that strips deep learning down to its bare essentials in under 10,000 lines of Python.

P24
By Pickwise24 Editorial Team
Verified Open-Source Review

Modern deep learning frameworks feel like Victorian steam engines: immensely powerful, but held together by millions of lines of legacy C++, brittle CUDA bindings, and enough bespoke abstractions to make your IDE weep. If you have ever tried to trace an error deep inside PyTorch’s autograd engine, you know the sensation of staring into an abyss that stares right back.

Enter tinygrad (github.com/tinygrad/tinygrad), the brainchild of George Hotz (geohot) and the tiny corp team. Its premise is delightfully insolent: rebuild an entire deep learning framework and optimising neural network compiler in under 10,000 lines of readable Python.

No bloat. No hidden proprietary blobs. Just pure, unadulterated linear algebra lowered directly to hardware.


What Is tinygrad?

tinygrad is an open-source, minimalist deep learning framework and just-in-time (JIT) neural network compiler. It sits neatly between high-level frameworks like PyTorch and low-level domain-specific compilers like XLA or TVM.

Instead of treating hardware backends as completely alien beasts requiring custom operator libraries (like cuDNN), tinygrad reduces every tensor operation into a handful of primitive primitives, fuses them into computational graphs, and compiles them directly into kernel code for your target hardware.


+-------------------------------------------------------+
|                High-Level Python API                  |
|          (Tensor operations, autograd, optim)         |
+---------------------------+---------------------------+
                            |
                            v
+-------------------------------------------------------+
|                    LazyBuffer / AST                   |
|       (Graph construction, dead-code elimination)     |
+---------------------------+---------------------------+
                            |
                            v
+-------------------------------------------------------+
|                     Linearizer                        |
|        (Kernel fusion, memory layout optimisation)    |
+---------------------------+---------------------------+
                            |
         +------------------+------------------+
         |                  |                  |
         v                  v                  v
     [Metal/OSX]        [OpenCL/GPU]       [CUDA/C]

The 26 Primitive Operations

While standard frameworks maintain thousands of bespoke kernels, tinygrad reduces all operations across deep learning to roughly 26 primitive ops split across four categories:

  1. UnaryOps: Element-wise mathematical transformations (e.g., EXP2, LOG2, SQRT, SIN).
  2. BinaryOps: Element-wise operations between two tensors (e.g., ADD, MUL, CMPLT).
  3. ReduceOps: Dimensional collapse operations (e.g., SUM, MAX).
  4. MovementOps: Zero-copy tensor shape and stride transformations (e.g., RESHAPE, PERMUTE, SLICE, EXPAND).

Convolution? That is merely a stride manipulation (MovementOp) followed by an element-wise multiplication (BinaryOp) and a summation (ReduceOp). Because strides handle spatial transforms virtually without moving memory, memory bandwidth stays lean.


Architectural Comparison: PyTorch vs tinygrad

FeaturePyTorchtinygrad
Codebase SizeMillions of lines (C++, Python, CUDA)< 10,000 lines of Python
Execution ModelEager by default (TorchDynamo for compile)Lazy execution with unified JIT compiler
Backend SupportHardware-specific manual kernelsJIT-compiled C, OpenCL, Metal, CUDA, AMD (HIP/LLVM)
Operator Count2,000+ distinct operators~26 primitive mathematical operations
Mental ModelMonolithic frameworkHackable, transparent compiler pipeline

Getting Started: Installation and Setup

Getting started does not require wrestling with half-broken container configurations. You can install tinygrad directly from PyPI or clone the repository to poke around its internals.


# Clone the repository to study the compiler
git clone https://github.com/tinygrad/tinygrad.git
cd tinygrad
python3 -m pip install -e .

Verify your setup by running the built-in test suite:


python3 -m pytest test/test_ops.py

Practical Example: Training a Model

Here is how straightforward it is to build, train, and inspect a simple multi-layer perceptron. Notice how closely the user-facing syntax mirrors the PyTorch idioms you already know, yet without the extraneous runtime baggage.


from tinygrad import Tensor, nn
from tinygrad.engine.jit import TinyJit

class TinyClassifier:
    def __init__(self):
        self.l1 = nn.Linear(784, 128)
        self.l2 = nn.Linear(128, 10)

    def __call__(self, x: Tensor) -> Tensor:
        return self.l2(self.l1(x).relu())

model = TinyClassifier()
optim = nn.optim.Adam(nn.state.get_parameters(model), lr=0.001)

# Compile the training step using the JIT decorator
@TinyJit
def train_step(x: Tensor, y: Tensor) -> Tensor:
    optim.zero_grad()
    out = model(x)
    loss = out.sparse_categorical_crossentropy(y)
    loss.backward()
    optim.step()
    return loss

# Dummy data: Batch of 32 images (flattened to 784)
x_train = Tensor.randn(32, 784)
y_train = Tensor.randint(32, low=0, high=10)

# Run a sample forward & backward pass
loss = train_step(x_train, y_train)
print(f"Step Loss: {loss.numpy():.4f}")

Inspecting the Generated Kernel

Want to see the raw GPU code tinygrad creates for your machine? Simply prefix your script with the DEBUG environment variable:


DEBUG=3 python train_script.py

tinygrad will dump the exact Metal, CUDA, or OpenCL kernels it generated, alongside memory read/write statistics and runtime metrics in gigasamples per second.


Why tinygrad Matters in the Current AI Landscape

The developer consensus across tech forums and compiler engineering channels is clear: deep learning stacks have grown unsustainably complex. As companies race to run open-weight models locally—on everything from high-end workstations to consumer laptops—we cannot afford runtimes that require gigabytes of supporting dependencies just to compute matrix multiplications.

tinygrad strips away this complexity. It proves that you do not need proprietary driver magic or sprawling runtime layers to achieve competitive compute throughput. Whether you want an ultra-clean library to understand how backpropagation actually maps to GPU threads, or you need an embeddable, hackable engine for custom hardware accelerators, tinygrad is one of the most refreshing projects in modern computing.

🛡️ Editorial Standards & Methodology

Every repository featured on Pickwise24 undergoes testing on local workstation hardware before publication. We verify CLI installation steps, review open-source repository licensing, benchmark computational footprint, and evaluate architectural trade-offs to provide genuine, high-utility developer intelligence.