Introduction: Why SAM 2 is Disrupting Computer Vision
If you have ever spent a bleary-eyed Tuesday night meticulously rotoscoping a moving object frame by frame in video editing software, you will understand the deep, soulful exhaustion that plagues visual effects artists and computer vision developers alike. Historically, video segmentation has been a nightmare: frame-by-frame annotation is painfully slow, and tracking models tend to lose their minds the moment an object gets occluded or changes lighting.
Enter Meta AI and their heavy-hitting open-source repository: facebookresearch/sam2. Building upon the massive success of the original Segment Anything Model, SAM 2 extends zero-shot segmentation capabilities directly into video streams. It processes video frames in real-time with stunning accuracy, treating time as just another spatial dimension.
In this deep dive, we will unpack how SAM 2 solves high-performance video tracking, examine its clever streaming architecture, walk through a local installation, and run practical Python inference code.
What Problem Does SAM 2 Solve?
Traditional segmentation models generally fall into two camps: image models that treat video as a disjointed sequence of independent snapshots, and heavy video object segmentation (VOS) models that require expensive per-video fine-tuning.
SAM 2 solves the zero-shot video segmentation bottleneck. It allows developers to prompt a model with a single click, box, or mask on any frame, and automatically tracks that object across both past and future frames without prior training on the target video.
+-----------------------------------------------------------------+
| SAM 2 Architecture |
| |
| [ Video Frames ] ---> [ Image Encoder ] ---> [ Memory Bank ] |
| | |
| [ User Prompt ] ---> [ Prompt Encoder ] ---> [ Transformer ] |
| | |
| [ Mask Decoder ]|
| | |
| v |
| [ Segmented Video ] |
+-----------------------------------------------------------------+
Key Architectural Details
- Streaming Architecture: Instead of processing an entire video clip all at once (which causes VRAM usage to explode), SAM 2 processes frames sequentially through a streaming memory mechanism.
- Memory Attention Mechanism: The model features a memory encoder, a memory bank, and a memory attention module. When an object is segmented in a frame, its features are written to the memory bank. Subsequent frames attend to this historical memory to maintain tracking consistency, even through heavy occlusions.
- Hierarchical Image Backbone: Built on a powerful Hierarchical Image Transformer (Hiera) backbone, SAM 2 extracts multi-scale features, enabling it to catch both fine-grained edge details and broad semantic shapes.
Feature Walkthrough
The facebookresearch/sam2 repository comes equipped with everything a developer needs to start parsing pixels immediately:
- Zero-Shot Generalisation: Out-of-the-box performance on unseen domains, from wildlife documentaries to microscopic cell imagery and fast-paced sports.
- Interactive Refinement: Ability to add positive and negative clicks across any frame to dynamically correct masks on the fly.
- Multiple Model Sizes: Ranging from tiny configurations (
sam2_hiera_tiny) designed for edge devices up to large models (sam2_hiera_large) built for heavy-duty server inference.
| Model Variant | Backbone | Speed (FPS) | Best Use Case |
|---|---|---|---|
sam2_hiera_t | Hiera-Tiny | Fast (~45 FPS) | Edge devices, real-time webcam feeds |
sam2_hiera_s | Hiera-Small | Balanced (~30 FPS) | General developer prototyping |
sam2_hiera_l | Hiera-Large | Heavy (~15 FPS) | High-fidelity offline production rendering |
Local Setup and Installation Guide
Let us get SAM 2 running locally. You will need an NVIDIA GPU with adequate VRAM (8GB+ recommended) and a clean Python environment (Python 3.10+).
Step 1: Clone the Repository
git clone https://github.com/facebookresearch/sam2.git
cd sam2
Step 2: Install Dependencies
It is best to install the package in editable mode along with its dependencies:
pip install -e .
Step 3: Download Pre-trained Checkpoints
Meta hosts several model checkpoints. Download the large or tiny checkpoint into the checkpoints/ directory:
mkdir -p checkpoints
cd checkpoints
# Example for the Hiera-Large checkpoint
wget https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt
cd ..
Practical Code & CLI Usage Examples
Here is a clean, minimal Python snippet to initialise SAM 2 and track an object across a sequence of video frames loaded as individual images.
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
from sam2.build_sam import build_sam2_video_predictor
# Select device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Paths
sam2_checkpoint = "./checkpoints/sam2_hiera_large.pt"
model_cfg = "sam2_hiera_l.yaml"
# Initialise video predictor
predictor = build_sam2_video_predictor(model_cfg, sam2_checkpoint, device=device)
# Directory containing sequential video frames (frame_0000.jpg, etc.)
video_dir = "./path_to_video_frames"
frame_names = [
p for p in os.listdir(video_dir)
if p.endswith((".jpg", ".jpeg", ".png"))
]
frame_names.sort()
# Initialise inference state
inference_state = predictor.init_state(video_dir)
# Add a click prompt on the first frame (frame index 0)
# Object ID 1, point coordinate [x, y], label 1 (1 for foreground, 0 for background)
ann_frame_idx = 0
ann_obj_id = 1
points = np.array([[210, 350]], dtype=np.float32)
labels = np.array([1], np.int32)
_, out_frame_idx, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
inference_state=inference_state,
frame_idx=ann_frame_idx,
obj_id=ann_obj_id,
points=points,
labels=labels,
)
# Propagate through the video stream
video_segments = {}
for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
video_segments[out_frame_idx] = {
out_obj_id: (out_mask_logits[i] > 0.0).cpu().numpy()
for i, out_obj_id in enumerate(out_obj_ids)
}
print(f"Successfully tracked objects across {len(video_segments)} frames!")
Why SAM 2 Stands Out
SAM 2 completely changes the economics of video processing pipelines. Before its release, building a reliable object tracker required training custom bounding-box networks or stitching brittle optical flow algorithms together. By turning video segmentation into a prompt-driven, memory-augmented transformer task, Meta has handed developers a Swiss Army knife for computer vision. Whether you are building automated sports analytics, security monitoring tools, or next-generation video editing suites, facebookresearch/sam2 is an essential repository to clone, test, and integrate today.