On July 29, iOS and Metal engineer Andrey Mikhaylov posted TurboFieldfare on Show HN: an open-source inference engine (Apache-2.0) written from scratch in Swift and Metal that runs Gemma 4 26B-A4B in roughly 2 GB of RAM on any Apple Silicon Mac — including machines with only 8 GB of unified memory. The thread climbed to 919 points with over three hundred comments. The project is named after the fieldfare, the author’s favorite thrush, and the README is explicit about two things: it is not affiliated with Google, and it is not a wrapper around MLX or llama.cpp.
Why a 26B Model Fits in 2 GB
The trick starts with the mixture-of-experts architecture: Gemma 4 26B-A4B has about 26 billion total parameters but only about 3.88 billion active per token. TurboFieldfare simply refuses to put the whole model in memory. Only the shared core — about 1.35 GB — and an FP16 KV cache stay resident. For each generated token, the router picks the top-8 expert IDs, and the engine fetches those expert weights from SSD via bounded parallel pread calls into a 16-slot LFU cache, where they get reused across tokens.
The quantization split is deliberate: embeddings, attention, and expert weights use MLX affine 4-bit with group 64, while the router stays at 8-bit, since routing mistakes cost more than weight precision. Prefill runs in chunks of up to 128 tokens so one fetched expert serves multiple rows. The KV cache is handled per layer type: circular storage for the 25 sliding-window layers, linear storage for the 5 full-attention layers.
Worth being clear about why only a MoE can do this. A dense 26B model touches every parameter for every token, so there is nothing to leave on disk. The 26B-total, 3.88B-active ratio is what makes the streaming arithmetic work: each token reads a few hundred megabytes of cold experts at most, and the LFU cache absorbs the locality in real workloads.
Measured: 5 to 6 Tokens per Second on an 8GB Air
The numbers the author measured: 5.1 to 6.3 tokens per second decode on an 8GB M2 MacBook Air, and 31 to 35 tokens per second on a 24GB M5 Pro. The price is storage — the text model takes about 14.3 GB. The installer streams byte ranges directly from a pinned Hugging Face revision and repacks them into a .gturbo layout on the fly, never staging a full checkpoint on disk. An optional 1.1 GB vision pack requires M2 or later; text-only mode runs on M1. Requirements are macOS 26 and Metal 4.
The repo ships more than an engine: six Swift products, including the core library, a native Mac app, a one-shot decode service, a CLI, and a loopback OpenAI-compatible Chat Completions server with function tools — no auth or TLS, strictly local — plus the streaming installer itself. The repo also documents 103 audited experiments. That number probably says more about how the thing was tuned than any headline benchmark would.
Why This Approach Is Worth Watching
Most local runtimes treat RAM as a hard wall: if the model does not fit, it does not run. TurboFieldfare treats SSD as part of the memory hierarchy and trades random-read latency for RAM, using the sparse activation of MoE to keep the trade survivable. The top Hacker News comment asked exactly the right question — why do we insist on shoving the entire model into memory at all? The replies were just as honest: per-token knowledge relevance is still an unsolved research problem, and MoE is only a partial answer to it.
For product teams there are two lessons. First, 8GB entry-level machines become a viable target again, which nudges the minimum hardware bar for on-device AI down another notch. Second, this is an engineering trade-off laid out in the open: you pay with disk space and random-read latency to save memory. Whether that trade works depends on whether your product needs to run at all, or run fast — an interactive assistant and a background summarizer will answer that question differently.
It also points at an obvious follow-up: the same expert-streaming scheme applied to larger MoE checkpoints, where the resident core stays fixed but the disk footprint grows. The iPhone and iPad ports listed as future work aim the same direction — memory-constrained devices are exactly where this trade-off pays best.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.