Modern deep learning frameworks feel like Victorian steam engines: immensely powerful, but held together by millions of lines of legacy C++, brittle CUDA bindings, and enough bespoke abstractions to make your IDE weep. If you have ever tried to trace an error deep inside PyTorch’s autograd engine, you know the sensation of staring into an abyss that stares right back.
Enter tinygrad (github.com/tinygrad/tinygrad), the brainchild of George Hotz (geohot) and the tiny corp team. Its premise is delightfully insolent: rebuild an entire deep learning framework and optimising neural network compiler in under 10,000 lines of readable Python.
No bloat. No hidden proprietary blobs. Just pure, unadulterated linear algebra lowered directly to hardware.
What Is tinygrad?
tinygrad is an open-source, minimalist deep learning framework and just-in-time (JIT) neural network compiler. It sits neatly between high-level frameworks like PyTorch and low-level domain-specific compilers like XLA or TVM.
Instead of treating hardware backends as completely alien beasts requiring custom operator libraries (like cuDNN), tinygrad reduces every tensor operation into a handful of primitive primitives, fuses them into computational graphs, and compiles them directly into kernel code for your target hardware.
+-------------------------------------------------------+
| High-Level Python API |
| (Tensor operations, autograd, optim) |
+---------------------------+---------------------------+
|
v
+-------------------------------------------------------+
| LazyBuffer / AST |
| (Graph construction, dead-code elimination) |
+---------------------------+---------------------------+
|
v
+-------------------------------------------------------+
| Linearizer |
| (Kernel fusion, memory layout optimisation) |
+---------------------------+---------------------------+
|
+------------------+------------------+
| | |
v v v
[Metal/OSX] [OpenCL/GPU] [CUDA/C]
The 26 Primitive Operations
While standard frameworks maintain thousands of bespoke kernels, tinygrad reduces all operations across deep learning to roughly 26 primitive ops split across four categories:
- UnaryOps: Element-wise mathematical transformations (e.g.,
EXP2,LOG2,SQRT,SIN). - BinaryOps: Element-wise operations between two tensors (e.g.,
ADD,MUL,CMPLT). - ReduceOps: Dimensional collapse operations (e.g.,
SUM,MAX). - MovementOps: Zero-copy tensor shape and stride transformations (e.g.,
RESHAPE,PERMUTE,SLICE,EXPAND).
Convolution? That is merely a stride manipulation (MovementOp) followed by an element-wise multiplication (BinaryOp) and a summation (ReduceOp). Because strides handle spatial transforms virtually without moving memory, memory bandwidth stays lean.
Architectural Comparison: PyTorch vs tinygrad
| Feature | PyTorch | tinygrad |
|---|---|---|
| Codebase Size | Millions of lines (C++, Python, CUDA) | < 10,000 lines of Python |
| Execution Model | Eager by default (TorchDynamo for compile) | Lazy execution with unified JIT compiler |
| Backend Support | Hardware-specific manual kernels | JIT-compiled C, OpenCL, Metal, CUDA, AMD (HIP/LLVM) |
| Operator Count | 2,000+ distinct operators | ~26 primitive mathematical operations |
| Mental Model | Monolithic framework | Hackable, transparent compiler pipeline |
Getting Started: Installation and Setup
Getting started does not require wrestling with half-broken container configurations. You can install tinygrad directly from PyPI or clone the repository to poke around its internals.
# Clone the repository to study the compiler
git clone https://github.com/tinygrad/tinygrad.git
cd tinygrad
python3 -m pip install -e .
Verify your setup by running the built-in test suite:
python3 -m pytest test/test_ops.py
Practical Example: Training a Model
Here is how straightforward it is to build, train, and inspect a simple multi-layer perceptron. Notice how closely the user-facing syntax mirrors the PyTorch idioms you already know, yet without the extraneous runtime baggage.
from tinygrad import Tensor, nn
from tinygrad.engine.jit import TinyJit
class TinyClassifier:
def __init__(self):
self.l1 = nn.Linear(784, 128)
self.l2 = nn.Linear(128, 10)
def __call__(self, x: Tensor) -> Tensor:
return self.l2(self.l1(x).relu())
model = TinyClassifier()
optim = nn.optim.Adam(nn.state.get_parameters(model), lr=0.001)
# Compile the training step using the JIT decorator
@TinyJit
def train_step(x: Tensor, y: Tensor) -> Tensor:
optim.zero_grad()
out = model(x)
loss = out.sparse_categorical_crossentropy(y)
loss.backward()
optim.step()
return loss
# Dummy data: Batch of 32 images (flattened to 784)
x_train = Tensor.randn(32, 784)
y_train = Tensor.randint(32, low=0, high=10)
# Run a sample forward & backward pass
loss = train_step(x_train, y_train)
print(f"Step Loss: {loss.numpy():.4f}")
Inspecting the Generated Kernel
Want to see the raw GPU code tinygrad creates for your machine? Simply prefix your script with the DEBUG environment variable:
DEBUG=3 python train_script.py
tinygrad will dump the exact Metal, CUDA, or OpenCL kernels it generated, alongside memory read/write statistics and runtime metrics in gigasamples per second.
Why tinygrad Matters in the Current AI Landscape
The developer consensus across tech forums and compiler engineering channels is clear: deep learning stacks have grown unsustainably complex. As companies race to run open-weight models locally—on everything from high-end workstations to consumer laptops—we cannot afford runtimes that require gigabytes of supporting dependencies just to compute matrix multiplications.
tinygrad strips away this complexity. It proves that you do not need proprietary driver magic or sprawling runtime layers to achieve competitive compute throughput. Whether you want an ultra-clean library to understand how backpropagation actually maps to GPU threads, or you need an embeddable, hackable engine for custom hardware accelerators, tinygrad is one of the most refreshing projects in modern computing.