VRAM Guide: What Local Coding Models Fit 8GB, 16GB, and 24GB (2026)
A practical VRAM budget for offline AI coding: which GGUF model sizes fit 8GB, 16GB, and 24GB GPUs, how KV cache eats memory, and how to pick a Quietly-ready stack that stays fast.
Awareness · 12 min
VRAM Guide
A practical VRAM budget for offline AI coding: which GGUF model sizes fit 8GB, 16GB, and 24GB GPUs, how KV cache eats memory, and how to pick a Quietly-ready stack that stays fast.
Definition
VRAM for local coding LLMs is the GPU memory that must hold model weights plus the KV cache for your context window—if either overflows, inference slows dramatically or fails, even when system RAM looks fine.
Buying a “coding LLM” is really buying a memory budget. Parameter count on the box means little until you map it to quantization, context length, and leftover VRAM for the editor, browser, and OS.
This guide gives Quietly’s practical VRAM map for 8GB, 16GB, and 24GB cards in 2026: what fits, what feels fast enough for pair-programming, and what to avoid so you do not spend a weekend fighting swap thrash.
Numbers below assume GGUF-style runtimes (llama.cpp-class) with a coding chat/agent session open—not a bare CLI hello-world at 512 tokens.
The Quietly VRAM Budget Formula
Treat VRAM as three buckets. If you only size for weights, long chats and repo context will still OOM mid-session.
note
Usable VRAM ≈ Total − 0.8–1.5GB overhead
Drivers, desktop compositor, and Quietly’s UI typically leave ~0.8–1.5GB less than the marketing sticker. Plan with usable VRAM, not the box label.
- Weights: file size of the GGUF (or loaded HF shards) that must stay resident.
- KV cache: grows with context tokens × layers × heads—often the silent killer at 16k–32k context.
- Runtime headroom: 1–2GB buffer so a longer reply or a second tool call does not evict layers to CPU.
Rule-of-thumb weight footprints for common coding sizes (Q4_K_M class). Exact files vary by architecture; treat ±10–15% as normal.
| Model class | Approx. weights (Q4) | Comfortable min usable VRAM | Typical role |
|---|---|---|---|
| ~7–8B dense | 4–5.5 GB | 7–8 GB | Fast chat, light refactors |
| ~9–14B dense | 6–9 GB | 12–16 GB | Daily pair-programming |
| ~22–32B dense | 12–18 GB | 20–24 GB | Stronger reasoning / multi-file |
| MoE “large” (few active params) | Varies widely | 16–24 GB+ | Agentic coding when quant fits |
warning
Context is not free
Doubling context from 4k→8k or 8k→16k can add hundreds of MB to multiple GB of KV cache. A model that “fits” at 4k may thrash at 32k on the same card.
8GB VRAM: what actually works
8GB is an entry lane—not a dead end. It is honest about limits: prioritize speed and discipline over chasing leaderboard giants.
8GB practical map (usable ~6.5–7.2GB). Prefer Q4/Q5; leave ≥1GB free.
| Fit? | Model size / quant | Context target | Quietly workflow tip |
|---|---|---|---|
| Yes | 7–8B Q4_K_M | 4k–8k | Best daily driver on 8GB |
| Tight | 7–8B Q5 / Q6 | 4k | Higher quality, less headroom |
| Risky | 9–14B Q3/Q4 | 2k–4k | Often CPU-offload; chat feels laggy |
| No (daily) | 22B+ at usable quant | — | Save for a 16GB+ card |
- Best jobs: explain selection, write small functions, draft tests, rename/refactor within one file.
- Weak jobs: huge multi-file agents, 100k-token repo dumps, “rewrite the whole service.”
- Cards in this class: many RTX 4060 8GB / laptop 8GB designs—capacity-limited more than “slow cores.”
tip
8GB survival kit
Cap context, close unused GPU apps (browsers with hardware acceleration, games, other local servers), and prefer one coding model at a time. In Quietly, start with a 7–8B coding instruct GGUF before importing a larger HF folder.
16GB VRAM: the daily-driver sweet spot
16GB is where local coding stops feeling like a demo and starts replacing cloud assistants for most solo developers. You can run mid-size models with enough context for real PRs.
16GB practical map (usable ~14–15GB).
| Fit? | Model size / quant | Context target | Quietly workflow tip |
|---|---|---|---|
| Excellent | 7–8B Q5/Q6 or Q8 | 8k–16k | Near-lossless quality, very fast |
| Excellent | 9–14B Q4_K_M | 8k–16k | Primary “Copilot-like” local tier |
| Good | 14B Q5 | 4k–8k | Quality up; watch KV growth |
| Possible | 22–27B Q3/Q4 | 4k–8k | Usable if you keep context tight |
| Avoid | 32B+ high quant + 32k ctx | — | Starves headroom; pick 24GB |
- Best jobs: multi-file refactors with surgical context, Project Brain retrieval + chat, agent proposals you still approve action-by-action.
- Hardware examples: RTX 4060 Ti 16GB, many 16GB workstation/laptop SKUs—VRAM capacity matters more than chasing peak TFLOPS for chat latency.
- Quietly angle: keep Llama.cpp for GGUF speed; reserve heavier engines when you intentionally load larger HF models.
note
Bandwidth vs capacity
Some 16GB cards have less memory bandwidth than older 12GB cards. For coding chat, fitting a better model (capacity) usually beats a slightly higher tok/s on a smaller model—until latency exceeds ~your typing rhythm.
24GB VRAM: room for strong models and longer context
24GB unlocks the “I barely compromise” tier: stronger dense models, healthier context, and fewer forced trade-offs when Quietly’s Project Brain attaches repo snippets.
24GB practical map (usable ~22–23GB).
| Fit? | Model size / quant | Context target | Quietly workflow tip |
|---|---|---|---|
| Excellent | 14B Q5–Q8 | 16k–32k | High quality + long sessions |
| Excellent | 22–32B Q4_K_M | 8k–16k | Serious multi-file reasoning |
| Good | 32B Q5 | 4k–8k | Quality first; trim context |
| Selective | Large MoE GGUF | 8k–16k | Only if active params + quant fit |
| Still no | Frontier cloud-only sizes | — | Local ≠ infinite; stay realistic |
- Best jobs: agentic edit plans, deeper debugging, longer design chats without constantly summarizing history by hand.
- Hardware examples: RTX 3090 / 4090-class 24GB—often the best price/performance for heavy local coding if you already own or buy used.
- Team note: one 24GB machine can be a dedicated local inference box; Quietly still keeps prompts on-device instead of a SaaS path.
tip
Do not waste 24GB on vanity context
32k–64k context is useful only when you feed high-signal code. Dumping an entire monorepo into the window burns VRAM and accuracy. Prefer Quietly Project Brain retrieval over raw paste.
No discrete GPU? CPU, unified memory, and Mac notes
Quietly runs on Windows, macOS, and Linux without requiring a datacenter GPU—but physics still applies. Unified memory (Apple Silicon) and large system RAM change the math.
- CPU-only / iGPU: expect 7–8B Q4 as the realistic daily class; treat 14B as “batch help,” not instant autocomplete.
- Apple Silicon unified memory: model weights compete with the OS and IDE—budget like VRAM plus a bigger safety margin (often 20–30% free).
- 64GB+ system RAM: enables larger GGUF via RAM inference or hybrid offload; latency still trails a true 16–24GB GPU for interactive coding.
warning
Swap is not VRAM
If the OS starts swapping model layers to disk, token generation can fall to a crawl. Downsize the model or close memory hogs—do not “fix” it by raising context.
Quietly model picker: choose by tier, not hype
Decision matrix for Quietly setups in 2026.
| Your GPU | Start here | Upgrade path | Avoid first week |
|---|---|---|---|
| 8GB | 7–8B coding instruct Q4 | Same size Q5 if stable | 14B+ with 16k context |
| 16GB | 9–14B coding Q4 | 14B Q5 or 22–27B Q4 tight | Two models loaded at once |
| 24GB | 14B Q5 or 22–32B Q4 | Longer context / MoE experiments | Max context + max model together |
In Quietly’s Model Hub, download one coding-oriented GGUF, verify chat latency with Wi‑Fi off, then increase size only if replies stay interactive. Engines (Llama.cpp, AirLLM, Frontier/Colibri) change loader behavior—but they do not invent VRAM.
tip
Airplane test
After download, disconnect the network and run a 20-turn coding chat at your chosen context. If it stays smooth, your VRAM budget is honest. If it dies mid-thread, you oversized the model or the context—not Quietly’s offline claim.
Related product pages: quietlycode.org/local-ai-coding and quietlycode.org/offline-ai-ide. Download: quietlycode.org/download.
Seven VRAM mistakes that waste weeks
- Sizing only for weights and ignoring KV cache growth.
- Buying 8GB cards expecting 32B “almost fits” quality.
- Running a second local server (another UI + same model) that doubles residency.
- Leaving 32k context on by default for tiny edits.
- Comparing tok/s on empty prompts to real IDE sessions with RAG attachments.
- Assuming cloud “private mode” is equivalent to local VRAM isolation.
- Chasing every new MoE release before your 14B daily driver is tuned.
FAQ
How much VRAM do I need for a local coding LLM in 2026?
8GB works for 7–8B quantized models with modest context. 16GB is the practical daily driver for 9–14B coding models. 24GB comfortably runs stronger 22–32B Q4 models and longer sessions. Always reserve headroom for KV cache—not only weights.
Can I run a 14B coding model on 8GB VRAM?
Sometimes with aggressive quantization and CPU offload, but interactive pair-programming often feels too slow. Prefer a 7–8B Q4 model on 8GB, or move to 16GB for a true 14B daily driver.
Does system RAM replace VRAM for Quietly?
System RAM and unified memory can host models, especially on Apple Silicon or large-RAM PCs, but latency and contention differ from a dedicated GPU. Quietly still keeps inference local either way—choose a model size that stays responsive.
Why does my model fit at start but crash after a long chat?
KV cache grows with conversation and attached code context. Lower max context, clear history, attach fewer files, or use a smaller quant so long sessions keep a VRAM buffer.
What should I download first in Quietly?
Start with one coding-instruct GGUF sized for your tier (7–8B on 8GB, ~14B Q4 on 16GB, 14B–32B Q4 on 24GB). Confirm offline chat speed, then experiment—do not load three models on day one.
Related guides
Awareness
Local RAG for Repos: Private Project Context Without Cloud
How local RAG gives an offline AI IDE surgical repo context without uploading code: indexing under .quietly/, .gitignore/.quietignore, cursor-aware retrieval, and a privacy checklist for teams.
Comparison
Best Local Coding Models You Can Run Offline in 2026
A hardware-honest shortlist of open coding LLMs for offline IDEs: which model classes win for 8GB, 16GB, and 24GB, how to judge agent readiness, and how to load them in Quietly without cloud APIs.
Awareness
ChatGPT Offline Alternative for Non-Developers (Private Local Chat)
A practical ChatGPT offline alternative for students, writers, and professionals: private on-device chat without coding, what hardware you need, and how Quietly Chat keeps prompts off the cloud.