Windows · NVIDIA + AMD/Intel · open source · numbers first
The inference engine
designed for one machine.
Cloud inference is stateless, batched, and generic — because it serves strangers. Your machine serves one person, forever. Whiskurr is what running a model looks like when you design for that: ~1.8× the leading desktop LLM app's decode on the identical model file, same GPU, measured by the same harness.
MIT licensed. Every performance claim on this page has a measured before/after — the scoreboard is the index, and reproductions (or refutations) are contributions.
Why it's faster
Five pillars — and a sixth they earned.
Each one exists because a personal machine has properties a datacenter can't have: one user, persistent state, idle capacity, and a history worth learning from. A pillar ships only when it has a measured before/after number.
Sessions, not requests
Your system prompt and conversation history are prefilled once, ever. Every turn appends to a live KV cache — per-turn cost stays flat as the chat grows, instead of reprocessing everything like a stateless server. p50 501 → 381 ms on a 477-token prefix.
A drafter that knows you
Speculative decoding drafts from your own history — your phrases, your habits — not a generic small model. 1.7× on self-similar output, outputs bit-identical, free when it misses. Acceptance climbs the longer you use it.
Ensembles for free
Single-user decode is bandwidth-bound, so the compute lanes sit mostly idle: 8 candidate answers cost ~2.7× one, not 8× — 391 → 133 ms per candidate at n=8. Quality for free.
Verify, don't generate
Judging two finished candidates takes 54 ms; generating one takes ~390. Wherever checking beats writing, Whiskurr checks.
Arrival-time compute
While input is still arriving — you're still talking, still typing — the prompt is already being prefilled: 341 ms fed-while-arriving vs 386 all at once. By the time you hit send, the model has already read it.
VRAM as a cache
A 20.8 GB mixture-of-experts model on a 12 GB GPU: attention stays on the GPU, experts stream from RAM. 12.6 → 26.3 tok/s — 2.1× over the naive split, from one placement decision. This one wasn't in the plan until the measurements said yes — and then it carried a 284B model.
Numbers
Same file. Same GPU. Same harness.
Qwen3.6-35B-A3B (20.8 GB) on an RTX 4080 Laptop with 12 GB of VRAM — a model that "shouldn't fit" — both apps serving an OpenAI-compatible API, defaults vs defaults:
| metric | leading desktop app (defaults) | Whiskurr Chat | difference |
|---|---|---|---|
| decode speed | 16.3–18.5 tok/s | 29.9–32.5 tok/s | ~1.8× faster |
| whole task (200 tokens) | ~12.6 s | ~6.1 s | ~2× faster |
| warm first-token | 0.5–1.7 s | 0.12–0.15 s (7B class) · 0.4–0.7 s (hybrid 35B) | flipped |
That first-token row was originally published as a loss — then engineered away
(token-level prompt sync + prefix-state snapshots restoring in 9–26 ms) and re-measured with
output-correctness gates on every run, not hand-waved. Raw runs live in the repo
(compare-results.jsonl); the full ledger is
SCOREBOARD.md.
- Contextual-sparsity speculation ("read only the neurons that matter") failed on stock
models — the data, the scripts, and the verdict are in
sparsity/. - Quantizing the KV cache to q8 is not free speed: decode −5% at short context. It's a memory tool, so it isn't the default.
- Hybrid linear-attention models can't rewind — partial rewinds report success, then fail at decode. Shipped mitigation: prefix-state snapshots, 9–26 ms to restore.
Reference hardware for every number on this page: RTX 4080 Laptop (12 GB VRAM), 96 GB RAM, Windows 11. Your magnitudes will differ; the signs should not.
The ceiling test · measured 2026-07-06
A 284-billion-parameter model. This laptop.
parameters · one 145.29 GiB file · the same 12 GB GPU as every number above
DeepSeek V4 Flash (MXFP4) is bigger than this machine's RAM and VRAM combined. The same placement that earned the sixth pillar — attention on the GPU, experts streaming from 96 GB of RAM — plus memory-mapped overflow paging from NVMe, and it runs: prefill 2.92 tok/s, decode 1.73 tok/s. Plainly: that is reading pace, not chat pace — an oracle you consult, not a conversation. But a 284B model answering at all, on hardware you can buy in a mall, changes the question from "can it run?" to "when do you call the mountain?" — the small model does the talking, and "check that" hands the hard part upstairs.
llama-bench, experts streamed (-ngl 99 -ot exps=CPU), overflow mmap-paged
from NVMe. Next on the bench: prefix-snapshotting the session so short warm questions skip the
minutes-long prefill. Receipts in the repo.
Vision · since 0.2
It can see now.
Drop an image into the chat and ask. Works with any GGUF that ships an
mmproj sibling — Whiskurr finds the projector next to the weights, and the
Discover page badges every vision-capable model. It reads text off screenshots and names shapes
in drawings, fully local — the screenshot never leaves the machine that took it.
Whiskurr Chat
An app you already know how to use.
Load any GGUF from a dropdown of what's on disk, or download one from Hugging Face — model pages render right in the app, quants readable, fits-your-hardware badges included. Chat with streaming, tabs and split view, per-chat system prompts, edit-&-resend, a context-fill meter, a Stop button that actually stops, and a cat that purrduces while it thinks. Thinking models get an honest "🐾 Pawndering…" frame; the answer arrives clean.
Several models, resident at once
The API routes each request to its model by name — cold ones load into a spare slot without evicting what's warm. A 3B answered math 199 ms after a dictation-cleanup model finished, both loaded.
Receipts on every reply
tokens · time · tok/s · "pre-guessed N of your tokens (X% and learning)" — the flywheel is visible on each message, not asserted in marketing.
Downloads that survive
Hugging Face downloads run on their own threads — chat never blocks — with live progress, cancel, and resume from partial files via HTTP Range.
The "MoE experts in RAM" checkbox is how 35B-class models run on 12 GB — big models, small VRAM, one toggle. The full inventory — every feature, each labeled measured, verified, or shipped — is FEATURES.md. Trust is the product, so the labels are the point.
New in 0.3 · Whiskurr Coder
It codes now — carefully.
A coding agent built into the app: open a folder, say what you want changed. It runs on whatever model is loaded — it shares the engine, so it costs zero extra VRAM — and it's folder-scoped with path guarding: the agent works inside the folder you opened and nowhere else.
Plan first, then edits
Plan-first mode drafts a step plan you can edit before anything runs — reorder it, cut steps, rewrite them. The agent executes the plan you approved, not the one it imagined.
Nothing writes without you
In Manual, every change is a per-file diff review — nothing touches disk without your approval. Auto mode exists, with hard ceilings — and commands still ask.
Undo is unconditional
Every change is checkpointed; undo and revert are always there, and Stop always works. The safety net isn't a mode — it's on regardless.
One switch, one port
An OpenAI-compatible API for everything on your machine.
Flip one switch — or run headless — and anything that speaks OpenAI points at
http://127.0.0.1:1234/v1 unchanged. /v1/models lists every GGUF on disk;
naming one in a request routes to it. Continue.dev, Open WebUI, any OpenAI SDK —
one-line recipes in
INTEGRATIONS.md. Typurr dictates through the same port.
# headless — model + API, no window (PowerShell) $env:WHISKURR_MODEL="C:\models\your-model.gguf"; $env:WHISKURR_API="1" .\whiskurr-chat.exe # add WHISKURR_CPU_MOE=1 for big MoE models # then, from anything that speaks OpenAI: curl http://127.0.0.1:1234/v1/chat/completions -d '{"messages":[{"role":"user","content":"hello"}]}'
Requests that share a system prompt keep their prefix warm across clients — that's the 0.12–0.15 s warm first token, for whoever asks.
Download
Two builds. Every GPU.
Same app, same engine, same numbers-first attitude — pick the build that matches
your graphics card. Unzip, run whiskurr-chat.exe, click the starter-model button if
you don't have a GGUF yet.
NVIDIA — CUDA build
The build every number on this page was measured on. Fastest path on GeForce/RTX cards; the "MoE experts in RAM" trick for oversized models was benched here.
whiskurr-chat-portable.zip — or, hash-verified through Scoop:
scoop install https://raw.githubusercontent.com/typurrapp/whiskurr/main/whiskurr.jsonAMD · Intel · others — Vulkan build
The same engine on the GPU you already have: Radeon, Arc, and integrated graphics. Built on clean CI runners from the same source, attached to the same release — live now.
Both are Windows portable zips — no installer, no admin. SHA-256 checksums live on the
release page. Models download to
%APPDATA%\Typurr\models and are shared with Typurr, so nothing gets downloaded twice.
The family
Built for Typurr, released for everyone.
Whiskurr is the engine under Typurr — the local dictation app where your voice never leaves the machine. The dictation app taught the engine what one-user inference really needs; the engine gives every app the speed. Both open source, both MIT, both 100% indoor — and the self-trained dictation models are free on Hugging Face.