slotstream
What it is
Runs Qwen3.8-Flash-Next — a 125B MoE that is 104GB even at 4-bit — on Apple Silicon with a fraction of that in RAM, by streaming mixture-of-experts weights from SSD into a fixed cache of expert slots. MLX and Swift, a CLI plus Ollama- and OpenAI-compatible APIs, vision input included; a one-line installer pulls the ~88GB model. The README's own numbers: about 12 tokens/sec warm on a 48GB M5 Pro, and a technically-it-runs 1.41 tokens/sec on a 16GB M2 mini.
Featured in
Issue #11 — Mark Your Own Homework · — the full issue has that week's other tools, reads, and big-lab releases.
More local AI tools we've covered
- Llmspeed Pick a GPU or Mac and a model, and it plays tokens back at the speed you'd actually get.
- Janas A C engine that streams mixture-of-experts weights from SSD, so a 48 GB, 80B model runs on a 32 GB laptop.
- Magnitude Local inference engine for coding agents that tunes its kernels to your exact hardware before running a model.
- Cactus Needle 3 A foundation model for phones, wearables, and microcontrollers, built for automation rather than chat: tool calls, structured extraction, and embeddings. The trick is an 'intelligence ladder' — one set of weights where every depth from 2 to 20 layers is a usable model of its own, shipped as 2-bit binaries between roughly 9 and 29 MB — so you choose accuracy versus footprint per device without training a family. Open weights on Hugging Face under Apache-2.0; Cactus sells a one-off $19 hosted fine-tune.