Bonsai 27B: The Tiny AI Giant That Fits Your Phone

See All Articles

5 Key Takeaways

• Bonsai 27B uses extreme ternary and binary quantization (1.71 and 1.125 bits per weight) to shrink a 27B-parameter model to 5.9 GB and 3.9 GB while retaining 94.6% and 89.5% of original reasoning ability.
• It fits and runs efficiently on consumer devices like laptops and smartphones without cloud connectivity, enabling true on-device AI.
• The post-training quantization method avoids the accuracy collapse seen in traditional sub-4-bit approaches, especially on math, coding, and reasoning benchmarks.
• Memory demands are further reduced by quantizing the KV cache to 4-bit, allowing long contexts (e.g., 100k tokens) within mobile RAM constraints.
• The model is open source under Apache 2.0, works with llama.cpp and MLX, and achieves practical throughput (e.g., 11 tok/s on iPhone 17 Pro Max) for real-time applications.

---

Bonsai 27B: The Breakthrough That Puts a 27-Billion-Parameter AI on Your Laptop and Phone

Imagine an AI model that understands language, follows complex instructions, generates code, and even sees images – all while running directly on your laptop or smartphone without a cloud connection. That reality has just taken a giant leap forward. PrismML has released Bonsai 27B, a dramatically compressed version of the powerful Qwen3.6-27B model, that fits in as little as 3.9 gigabytes and keeps up to 94.6% of the original's reasoning ability. The release marks the first time a 27-billion-parameter model is genuinely usable on a phone, and it is available today under the open Apache 2.0 license.

This isn't a new AI trained from scratch. Bonsai 27B is the same architecture as Qwen3.6-27B, but repackaged using an extreme form of quantization. Quantization is the process of reducing the precision of the numbers that make up a neural network's "weights" – the core parameters that capture what the model has learned. Think of it as compressing a high-resolution photograph into a smaller file. Done well, the image is nearly indistinguishable; done poorly, artifacts creep in. PrismML's achievement is making that compression so effective that a model once requiring a server-class GPU can now live comfortably on a consumer device.

Why Compression Matters

Modern large language models often contain tens of billions of parameters. Storing each parameter as a 16-bit floating-point number (FP16) means a 27-billion-parameter model demands around 54 gigabytes of memory just for the weights. That excludes the additional memory needed to process text – the so‑called KV cache that grows with every token the model reads or writes. For comparison, a high-end smartphone may have 12 GB of RAM, and an operating system will typically cap an app to about half that. A 54 GB model is not even in the same ballpark.

The obvious path is to lower the number of bits per parameter. The field already has established 4-bit and 2-bit quantization methods, but these often hide a secret: their average bit‑per‑weight (bpw) is higher than the label suggests, and they tend to break down on tasks that require deep reasoning, such as advanced math or agentic tool use. Bonsai 27B takes a different route. It deploys true extreme compression – ternary (1.71 bpw) and binary (1.125 bpw) – while actively mitigating the accuracy collapse that plagues conventional sub‑4‑bit builds.

How Bonsai 27B Was Built

Under the hood, each weight in the model is not stored as a standalone number. Instead, it is represented by a code drawn from a tiny set of possible values, plus one shared scaling factor for every group of 128 weights. In the ternary variant, a weight can only be −1, 0, or +1; in the binary variant, it is either −1 or +1. The effective weight is calculated as the code multiplied by the group's scaling factor, which is kept in higher precision (FP16).

This scheme yields a true average bit-per-weight of 1.71 for ternary (because choosing one of three values carries about 1.585 bits of information, plus a 16‑bit scale amortized over 128 weights) and 1.125 for binary. Compared with the original FP16, ternary delivers a roughly 9.4‑fold reduction in weight memory; binary achieves a 14.2‑fold reduction. The math is neat, but the real test is whether the model still works.

Crucially, Bonsai 27B compresses the entire matrix-heavy backbone: embeddings, attention projections, the multilayer perceptron layers, and the final language‑model head. Only a tiny set of normalization and scaling parameters remains in higher precision. The vision tower – the component that lets the model understand images – is kept separately at 4‑bit precision using the HQQ method. This design ensures the whole system remains multimodal while still being small enough for edge deployment.

Unlike previous attempts such as BitNet, which require training a model from scratch with quantisation baked in, Bonsai 27B takes a finished high‑precision model and post‑trains it into a low‑bit representation. That makes the technique applicable to a wide range of existing models.

Performance: Surprising Accuracy at a Fraction of the Size

PrismML evaluated the model across 15 benchmarks in "thinking mode" – the setting where the model is encouraged to reason step‑by‑step. Testing used EvalScope with the vLLM inference engine on H100 GPUs. The headline numbers are striking.

The full FP16 Qwen3.6-27B scores an average of 85.07 on those 15 thinking benchmarks. The ternary version of Bonsai 27B, at just 5.9 GB, retains 80.49 – that is 94.6% of the original performance. The even smaller 1‑bit build, at 3.9 GB, still manages 76.11, holding on to 89.5% of the full model's capability.

Benchmark Category	FP16 Original	Ternary Bonsai (5.9 GB)	Binary Bonsai (3.9 GB)
Overall (15 benchmarks)	85.07	80.49 (94.6%)	76.11 (89.5%)
Math	95.3	93.4	91.7
Coding	88.7	86.0	81.9
Knowledge & Reasoning	83.2	77.0	73.4
Agentic & Tool-calling	80.0	74.0	66.0
Vision Understanding	72.6	65.2	59.6

These numbers are impressive in isolation, but what makes them remarkable is how they contrast with conventional low‑bit quantization. Traditional 2‑bit builds often maintain robust scores on multiple‑choice benchmarks like MMLU‑Redux, yet fail catastrophically when the task demands genuine problem‑solving depth. For example, the IQ2_XXS variant of Qwen3.6-27B (labeled "2‑bit" but actually averaging 2.8 bpw) scores just 57.5 on the AIME26 math competition set and 56.4 on LiveCodeBench, even though it manages 88.9 on MMLU‑Redux. Short‑form benchmarks mask the collapse; Bonsai 27B's evaluation explicitly avoids that pitfall. A similar pattern was observed on Gemma-4-31B's Q2_K_XL build, reinforcing that extreme quantization without careful post‑training can break a model's reasoning core.

Memory Budgets and Running on Phones

Making a model fit a phone isn't just about weight size. Smartphones have fixed, shared memory, and operating systems enforce strict limits. On iOS, for instance, a single app cannot reliably use more than about half of the physical RAM. A 12 GB iPhone exposes roughly 6 GB to an app – and that budget must cover model weights, the KV cache, and all runtime overhead.

The KV cache is a hidden memory hog. When an AI generates text, it stores the keys and values of previous tokens so that later tokens can attend to them. In Qwen3.6-27B, only 16 of the 64 transformer layers use a full, growing attention cache; the rest rely on linear attention, which is constant in memory. That design choice keeps the KV cache at about 64 KiB per token in FP16. Over a full 262,000‑token context window, the cache would balloon to roughly 17.2 GB. PrismML quantifies this and offers a 4‑bit KV cache, which shrinks that demand to around 4.3 GB. This is a critical knob for on-device deployment.

With those figures, the memory peaks become manageable. At 100,000 tokens using an FP16 cache, the 1‑bit Bonsai build peaks at 11.6 GB; the ternary peaks at 14.7 GB. By contrast, the ordinary Q4_K_XL ("4‑bit") build would need 25.6 GB. The math works out: the ternary Bonsai 27B not only uses fewer bits per weight but also benefits from the same efficient KV cache strategy.

PrismML also measured how much the output distribution changes when the KV cache is quantized to 4‑bit. Using forward KL divergence, a measure of how much the model's predictions deviate from the FP16‑KV baseline, the ternary Bonsai shifts by a mere 0.0011 nats on MATH‑500. The regular Q4_K_XL build shifts by 0.0146 nats – more than ten times as much. This suggests Bonsai's low‑bit weights are robust even when the cache is compressed, keeping generation quality nearly identical.

Throughput: From Data Centers to Your Pocket

Getting a model to fit is one thing; making it usable is another. Because text generation is largely memory‑bandwidth‑bound, the fewer bytes that need to be moved per token, the faster the output. Bonsai's extreme compression directly translates into speed gains on consumer hardware.

PrismML reported throughput numbers for generating 128 tokens after a prompt of 512 tokens. On an Apple M5 Max chip, the binary version reaches 66.5 tok/s for generation; on an M5 Pro, ternary manages 26.2 tok/s. On an iPhone 17 Pro Max, the binary build pushes out 11.0 tok/s – enough for a snappy chat experience. In a data‑centre setting with an H100 GPU, binary generation hits nearly 105 tok/s.

Prefill – the initial processing that reads the prompt – is compute‑bound and benefits less from the memory savings, but on‑device the small footprint means prefill completes quickly enough to be imperceptible.

To push generation speed even higher, PrismML ships a DSpark speculative decoder trained specifically for Bonsai 27B. Speculative decoding works by using a lightweight draft model to predict several future tokens, then letting the full model verify them all at once. With a draft depth of 4 on an H100, the binary Bonsai achieves an average acceptance length of 3.6 tokens, resulting in 143.8 tok/s – a 1.37‑fold speedup that costs nothing in output quality because verification is lossless. On Apple Silicon the drafter is off by default for single‑stream generation to avoid unnecessary overhead.

How to Use Bonsai 27B

Everything ships under Apache 2.0 on popular inference frameworks. The ternary build is the default in the demonstration repository. Starting the server is as simple as running a script to bring up an OpenAI‑compatible API with a chat and vision interface on port 8080. For direct command‑line generation, a few lines of code suffice:

./llama-cli -m ./Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf \
  --mmproj ./Ternary-Bonsai-27B-gguf/mmproj.gguf -c 0 \
  -p "Explain KV cache growth."

On Apple hardware via MLX, the command is similarly straightforward. Tool calling uses the standard OpenAI‑style tools array, with responses placed in choices[0].message.tool_calls. Thinking mode is enabled by default, but can be toggled per request with the thinking_budget_tokens parameter.

The model retains the original Qwen3.6-27B's 262K‑token context window, making it suitable for long‑form document understanding, repository‑scale code reasoning, and extended conversations.

Where It Fits

Bonsai 27B unlocks four practical deployment patterns that were simply not possible with a full‑precision 27B model:

1. Laptop‑local AI agents can use the ternary build for full‑repository code analysis and tool‑assisted workflows across a quarter‑million tokens – all offline.
2. Phone‑local reasoning becomes a reality with the 1‑bit build; internal tests suggest it consumes roughly 672 tokens per 1% of iPhone battery, making it viable for sustained use.
3. Privacy‑sensitive applications can keep all data on the device by construction, never sending prompts to a cloud service.
4. Resource‑conscious server setups benefit too: a single 24 GB GPU can now host a 27B‑class model when combining the 1‑bit weights with a 4‑bit KV cache.

Key Takeaways and What's Next

Bonsai 27B is not a new model trained from scratch; it is a demonstration that high‑quality post‑training quantization can shrink an enormous AI into a package that fits in your pocket while preserving almost all of its intelligence. The ternary build retains 94.6% of the original FP16 thinking score in just 5.9 GB, and the 1‑bit build retains 89.5% in 3.9 GB. Both are Apache 2.0 and run on llama.cpp (CUDA, Metal) and MLX. Conventional sub‑4‑bit quantization often fails on hard reasoning tests, but Bonsai's method avoids that cliff.

For everyday users, this means the line between cloud AI and on‑device AI is blurring. Tasks that once required sending data to a remote server – code generation, complex instruction following, even basic vision understanding – can now happen entirely on a laptop or phone, with no latency, no per‑token fees, and complete privacy. For developers, it offers a blueprint for shrinking other large models without retraining them.

As on‑device hardware continues to improve and quantization techniques mature, expect more models of this class to appear. Bonsai 27B may well be remembered as the moment a 27‑billion‑parameter AI really did fit in your pocket.

Read more