Open models: where to get them, how to run them, how to make them yours.
A short guide for the person who runs BrainBox. It covers registries, file formats, engines, licenses, and fine-tuning. Each claim is common knowledge or has a link.
Where models live
Five places hold almost every open model you will use.
Hugging FaceThe registry. Weights, model cards, and licenses. The community posts GGUF and MLX conversions under names like mlx-community, bartowski, and unsloth.Ollama libraryA curated list. One command pulls a model. Tags name the variant, for example gemma4:e4b or :q4_K_M.ModelScopeAlibaba's registry. The first stop for Qwen and GLM releases.NVIDIA NGCNVIDIA's catalog. Nemotron and Cosmos models, plus containers for their hardware.Formats and what the names mean
The file format tells you which engine can load it and how much memory it needs.
| Format | What it is | Runs on |
|---|---|---|
| safetensors | The original weights as the lab released them. Full precision or bf16. | PyTorch, vLLM, Transformers. Needs a framework and a GPU with enough memory. |
| GGUF | The llama.cpp format. Usually quantized. One file holds weights and metadata. | llama.cpp, Ollama, LM Studio. Mac, Linux, Windows, CPU or GPU. |
| MLX | Weights laid out for Apple's MLX framework. Quantized in 4 or 8 bit. | Apple Silicon only. mlx-lm and lily. |
| NVFP4 / FP8 | Low precision number formats that NVIDIA Blackwell GPUs run in hardware. | vLLM, SGLang, TensorRT-LLM on Blackwell. DGX Spark. |
| AWQ / GPTQ | GPU quantization methods. Both keep the weights in 4 bit and calibrate on sample text. | vLLM, SGLang, Transformers on NVIDIA GPUs. |
Quantization in plain words
A model is a large set of numbers. Quantization stores each number in fewer bits. The file gets smaller and the model runs faster. Some quality is lost.
- Q8 keeps 8 bits per weight. Near lossless. Twice the size of Q4.
- Q4_K_M keeps about 4.5 bits per weight. The default trade for most people.
- QAT means quantization aware training. The lab trained the model to expect 4 bit weights. In our bench the Gemma 4 12B QAT build got every fact right but dropped 3 of 12 citations. Test it on your own questions.
Memory rule: parameters times bytes per weight, plus room for context. A 12B model at Q4 is about 8 GB for weights. Add 1 to 3 GB for a long context. Leave headroom for the system.
How to run them
Pick the engine that matches your machine. BrainBox talks to any of them through Ollama or an OpenAI compatible URL. Set engine: openai and engine_url in brainbox.yaml.
OllamaThe simplest path. Mac and Linux. One command to pull, one to run. BrainBox uses it by default.llama.cppLoads any GGUF. llama-server gives an OpenAI style endpoint. The most knobs: context size, GPU layers, sampling.MLXApple Silicon native. Uses unified memory well. Pick it when you want the fastest tokens on a Mac and do not need GGUF.lilyPerplexity's open Metal server. Built for Qwen3.6-35B-A3B on an M5. Our desk-m5-lily profile uses it for chat and keeps Ollama for embed and vision.vLLM / SGLangGPU servers. Many users at once, batched requests, NVFP4 and FP8 on Blackwell. Pick these for a DGX Spark or a rack.LM StudioA desktop app. Browse Hugging Face, download, chat, serve on a local port. Pick it when you want a window instead of a terminal.Licenses, in plain words
Open weights means you can download the weights. It does not mean the training data or the training code are open. Read the model card before you ship a model to a client.
| License | What you can do | What to watch |
|---|---|---|
| Apache-2.0 and MIT | Use, change, sell, keep private. | Keep the license notice with the code or the weights. |
| Gemma terms | Use and distribute, commercial use allowed. | Read the prohibited use policy. It travels with any derivative. |
| Llama community license | Use and distribute, commercial use allowed. | Products over 700 million monthly users need a separate license from Meta. Name the model in your product. |
Make it yours: fine-tuning
A base model knows the world. A fine-tuned model knows your words, your formats, and your rules.
UnslothThe practical path. LoRA and QLoRA on one GPU. Notebooks for Gemma, Qwen, and Llama. Exports to GGUF so Ollama can load the result.MilesOpen RL post-training for LLMs and VLMs at scale. Produces LoRA adapters that SGLang serves. Pick it when you have a reward signal and a GPU cluster.MLX LoRAMac only training with mlx_lm.lora. Slower than a GPU. Good for small runs on the same machine that serves the model.What you need
- A few hundred good examples from your own data. Quality beats count.
- A benchmark that proves the tune helped. The BrainBox usage log and bench are built for this.
- A rule to stop. When the number does not move, stop.
We do this as a service. See Sovereign AI services.
Get familiar: a short path
Two weeks. One machine. Your own notes.
Day 1Install Ollama. Pull gemma4:e4b. Ask it ten questions about your own notes.Day 2Run the BrainBox bench on your machine. Read the numbers.Week 1Try two more models on the same questions. Keep the winner.Week 2One LoRA run on Unsloth with your own examples. Run the bench again.