We Built Video Search on a Jetson

NVIDIA's VSS Blueprint needs an H100 with 128GB RAM. We wanted video search on a device that fits in your hand. So we built one.

The problem

At GTC 2026, NVIDIA announced VSS 3, a blueprint for searching and summarizing massive volumes of video. The NHL uses it. It processes footage 30x faster than real-time. It is impressive.

It also requires an H100 or L40S GPU, 128GB of RAM, and Ubuntu 24.04 on x86. The Jetson AGX Orin in our lab has 64GB of unified memory, a 57GB eMMC drive, and an ARM processor. VSS does not run on it. NVIDIA confirmed this on their forums.

What we built instead

VSS-Lite. Four Python files. Same idea, ingest video, analyze frames with a vision model, index the results, search with natural language. Different scale, same utility.

How it works

The pipeline has three stages:

Ingest. Extract frames from video at configurable FPS. A motion filter skips static frames, no point analyzing a camera pointed at an empty court. Frames with significant motion go to the VLM.

Analyze. Each frame goes to llava:7b running on Ollama. The prompt asks for a structured JSON response: what happened, where is the ball, is it in or out, confidence level. On the Orin, this takes about 6 seconds per frame.

Index. Results go into SQLite with the frame metadata, timestamp, detections, VLM analysis, call type, confidence. Thumbnails saved to the SD card. The search layer maps natural language queries to structured filters.

$ python3 run.py
=== INGESTING ON JETSON ===
Frames analyzed: 5
Events found: 2
Ingest time: 57.1s
Speed: 1.1x real-time

=== SEARCH: "show me every out call" ===
Found 2 results
  4.0s: out (75%)
  28.0s: out (70%)

What we learned

Motion filtering matters. Without it, the VLM analyzes 15 frames of nothing happening. With it, we analyzed 5 frames that mattered. That is a 3x reduction in inference cost for the same useful output.

llava:7b is good enough for rough calls. It sees the court, identifies players, tracks ball position relative to lines. It is not accurate enough for sub-centimeter line calls. But for flagging moments worth reviewing, it works.

The real architecture is hybrid. DeepStream runs at 672 FPS for detection and tracking. The VLM runs at 0.16 FPS for reasoning. The trick is knowing which frames to send to the VLM. That is where the value is, not in making the VLM faster, but in being smarter about what you ask it.

Disk is the bottleneck on edge. Not compute. The Jetson has an Ampere GPU with 2048 CUDA cores. It has a 57GB eMMC drive. We spent more time managing disk space than debugging inference.

What is next

CLIP embeddings for visual similarity search, find frames that look like a specific play. A YOLO sports model to replace the generic traffic detector. And wiring it into the Sideline dashboard so coaches can search match footage from their phone.

The code is in the sports-ai repo. The approach is replicable on any Jetson device with an SD card.