Picking the Model: A Field Guide by Modality and Device

There is no best model. There is only the best model for this modality, on this device, under this latency budget. Here is the decision I actually run — and what happened when I took it to Apple Silicon with MLX.

Every few months someone asks me which model they should use. The honest answer is a question back: what are you putting in, what is it running on, and how long can it take? Answer those three and the field narrows from hundreds of models to about four.

Get them wrong and you end up where most teams end up — a frontier API doing work a 500-million-parameter model would do faster, cheaper, and without the data leaving the building.

The three questions, in order

Modality first. Text, vision, video, audio, and structured extraction are genuinely different problems with different winners. A model that tops the text leaderboards may be mediocre at reading a scanned form.

Device second. Unified memory on a Mac, discrete VRAM on an RTX card, and shared memory on a Jetson behave differently. The number that matters is not parameter count — it is whether the weights plus the KV cache fit in the memory you have, at the quantization you can tolerate.

Latency budget third. This is the one people skip, and it is the one that eliminates the most options. "Under 100 ms" and "under 10 seconds" are different products.

Text

The easiest case, and the most over-served. For classification, routing, extraction and summarisation over a bounded domain, a 3B–8B open-weight model fine-tuned on your own examples beats a frontier model at the task. Llama 3.x, Qwen 3, Gemma 3 and Nemotron Nano are the shortlist. Nemotron Nano is my default when the model has to drive tools.

Undervolt runs Nemotron Nano 8B locally over Ollama, structuring 2.2 million Austin building permits. The questions are bounded — "what has been permitted near this lat/lng" is not an open-ended request — so the small model is genuinely sufficient, and the per-token bill is electricity.

Reach for a frontier model when the task is open-ended reasoning over unfamiliar material. That is a real category. It is a much smaller one than the industry's API spend implies.

Vision and video

This is where device choice starts dominating, because frames arrive on a clock and the model has to keep up.

Detection and tracking are not the same problem as understanding. If you need boxes and identities, a YOLO-class detector is the right tool and a vision-language model is the wrong one — orders of magnitude apart in cost for a job the detector does better. If you need to know what is happening, that is a VLM, and you should expect seconds, not milliseconds.

The pattern that works is layered: run the cheap detector on every frame, and call the expensive model only on the frames that matter. On RefereAI, YOLO26 does per-frame detection and the interpretation layer sits above it — sport inference, score-source discipline, overlays. Cosmos Reason 2 on a Jetson is what I reach for when the question is genuinely about the scene rather than the objects in it, and that runs at a handful of frames per second, not thirty.

Audio

Speech-to-text is the most solved problem on this list and the one most often over-provisioned. A 110M-parameter Parakeet CTC model transcribes English faster than real time on a laptop CPU. Whisper is the safe default when you need multilingual or robustness to bad audio; the distilled variants get most of the quality at a fraction of the cost.

If you are sending audio to a cloud API for English transcription of clean speech, you are paying for something a small local model does well, on hardware you already own.

Then the device decides

Apple Silicon — unified memory is the advantage

The reason a Mac Studio is a serious inference box is not raw compute. It is that 192 GB of unified memory means the GPU addresses the whole model with no host-to-device copy. On a discrete card you are bounded by VRAM and paying a transfer tax; on Apple Silicon you are bounded by total RAM, which is much larger and much cheaper per gigabyte.

Jetson — when local also has to mean low-power

An AGX Orin is not fast compared to a 5090. It draws 15–60 W and can sit in a venue, a vehicle, or under a desk. Choose it when the constraint is power, physical placement, or the data genuinely cannot move — not when it is cost.

Discrete GPU — throughput per dollar

An RTX 4090 or 5090 remains the best throughput per dollar for training and batched serving. If nobody is asking you to unplug it and carry it somewhere, this is usually the right box.

The MLX part

Which brings me to what I actually learned building RefereAI on Apple Silicon.

MLX is Apple's array framework for its own silicon, and the thing that makes it interesting is not speed in isolation — it is lazy evaluation plus unified memory. Arrays live in memory both the CPU and GPU address. There is no .to(device), because there is nowhere else to move it. For a video pipeline that alternates between frame handling and inference dozens of times a second, removing the copy removes a real cost, not a theoretical one.

Porting YOLO26 to MLX for RefereAI was less about a faster model and more about the pipeline stopping being the bottleneck. The lesson that surprised me: most of my early wins were not in the model at all. They were in serialization — how frames got in and results got out. I spent longer on the boundary between the render worker and the inference path than on inference itself, and that is where the latency went from roughly 200 ms to comfortably under 100.

That is the general shape. On a well-chosen small model, the model is rarely the slow part. Preprocessing, format conversion, and moving data between processes usually are. Profile before you swap models.

What I would tell someone starting on MLX

Lazy evaluation is a real trap the first time. Nothing computes until you force it, so a naive benchmark measures graph construction and reports an impossibly good number. Call mx.eval() where you think the work happens, or you will believe something wrong for a day.

Not every operation has an MLX kernel. When you hit a gap, the choice is writing one or falling back — and the fallback can cost more than the op saved. Check coverage before committing a pipeline.

And quantization is where Apple Silicon gets generous. 4-bit weights on unified memory let a machine run models that a similarly priced discrete card cannot hold at all. The quality loss on a narrow, fine-tuned task is usually smaller than people expect — but measure it on your data, because "usually" is doing real work in that sentence.

The short version

Start from the modality, not the leaderboard. Let the device set the memory ceiling and the latency budget set the floor. Layer cheap models under expensive ones so the expensive one runs rarely. And before you conclude a model is too slow, profile the pipeline around it — in my experience that is where the time actually goes.

The moat was never the model. It is your data, and a model small enough to fit where the data already lives.