How much VRAM do you need for local LLMs

Some links on this site may be affiliate links. As an Amazon Associate I earn from qualifying purchases. How this works.

An 8 GB graphics card is worth trying with a small, quantized local model. A 12 GB or 16 GB card gives you more room for model weights and longer conversations. A 24 GB card makes some larger models plausible, but a model download smaller than 24 GB can still exceed its memory budget.

Choose the model and the conversation length first. Then choose the hardware. If you already own a supported GPU, try it before shopping. A model that fits is only a candidate: it still needs to answer your questions well and run at a speed you find useful.

Evidence: sourced research, not a hardware benchmark. The original Qwen3 examples were checked on 10 October 2026; the Qwen3.8-27B update was checked on 1 November 2026. The original file sizes come from Qwen’s official repositories; the update uses Unsloth’s published conversion sizes. The context-memory figures are calculations, not measured VRAM readings. We have not tested these models on the card sizes listed here, and we make no price, speed or quality ranking from their memory requirements.

Start with the model file

Parameter count alone is not a memory requirement. The exact model and its quantization matter. Quantization stores weights with reduced precision; the file becomes smaller, with possible effects on answer quality that need testing.

Here are three concrete downloads in Qwen’s official Q4_K_M GGUF format. These are examples for planning, not a recommendation that every reader use Qwen.

Here, a GB is 1,000,000,000 bytes and a GiB is 1,073,741,824 bytes. We converted the published byte counts and rounded to two decimal places. Compare the same units when checking your GPU’s reported memory. Do not automatically apply a decimal-to-binary conversion to a graphics card’s advertised “GB” label; check what the hardware and driver actually report.

Download size is not total VRAM use. A llama.cpp maintainer’s allocation explanation describes model-weight memory as usually close to file size, with separate context and compute buffers. The file is a useful starting estimate for weights, not a promise about the final allocation on every backend.

Leave room for more than weights

Four things compete for memory when you run a local chat model:

• Model weights: the stored numbers the model uses to generate answers.
• Conversation memory: commonly a key-value, or KV, cache holding attention information from processed tokens so it can be reused.
• Working buffers: temporary space used by the inference software.
• Other GPU use: the display, driver and other applications also need room.

There is no universal “add 1 GB” rule. Model architecture, backend, context, batching and cache settings change the result. Some caches grow as a conversation grows; others reserve space ahead of time. Hugging Face explains these cache strategies. A short prompt that loads successfully does not establish that a long session will fit.

Use VRAM tiers as starting points

This is a shortlist for testing, not a compatibility chart or a quality ladder. Confirm the actual card, operating system, driver and inference backend using the application’s hardware support documentation. A supported GPU with enough memory still needs a workload test.

November 2026 update: Qwen3.8-27B on 16 GB versus 24 GB

If you own a supported 16 GB GPU, try it before upgrading. For Qwen3.8-27B, 24 GB gives you more memory to work with. Neither capacity guarantees long context, fast responses or useful answers. Choose the exact file and check that your runtime supports it.

Qwen’s official model card identifies the model as Qwen/Qwen3.8-27B: a 27-billion-parameter language model with a vision encoder for images and video. It lists a native context window of 262,144 tokens. That describes the model’s supported context, not the memory available on your graphics card.

What the actual downloads tell you

Unsloth’s Qwen3.8-27B GGUF repository lists several quantized conversions of Qwen’s weights. The table links to their file metadata, checked on 1 November 2026. We converted the published byte counts to decimal GB and binary GiB, rounding to two decimal places. These are download sizes, not measured GPU use.

Exact Unsloth file	Download size, GB	Download size, GiB	Planning note
Qwen3.8-27B-UD-Q3_K_XL.gguf	13.15	12.24	A smaller candidate to test on an existing 16 GB card
Qwen3.8-27B-UD-IQ4_XS.gguf	14.25	13.27	Less room for cache and working memory in a 16 GiB budget
Qwen3.8-27B-UD-Q4_K_M.gguf	16.46	15.33	Very little room beyond weights in a 16 GiB budget
Qwen3.8-27B-UD-Q5_K_M.gguf	19.77	18.41	A candidate to investigate on 24 GB; still needs memory beyond weights

Compare the same units and check the driver’s reported memory. If it reports 16 GiB total, the Q4_K_M file’s 15.33 GiB leaves less than 0.7 GiB before cache, working buffers and other GPU use. Even the smaller files are only candidates for testing. File size alone cannot establish fit or answer quality.

Vision use also needs a separate image projector, offered in the Unsloth repository. Its allocation needs to be included wherever your runtime places it; the four downloads above do not include that file.

Qwen’s model configuration confirms a hybrid layout: 48 Gated DeltaNet (linear-attention) layers and 16 gated full-attention layers. Its memory use needs a model-specific estimate from the runtime. Do not reuse the 0.25 MiB-per-token formula below for Qwen3.8-27B. That calculation assumes Qwen3-32B’s different attention layout and an FP16 KV cache.

Treat owner reports as anecdotes

One 16 GB coding report describes using UD-Q2_K_XL, a different conversion from those in the table. A 16 GB AMD walkthrough specifies UD-IQ4_XS, quantized cache settings and an image projector kept off the GPU.

These are self-reported configurations, not verified fit results. They offer settings to investigate, but do not establish speed, answer quality or reliability on another machine. We have not reproduced either setup.

The buying decision

For an existing 16 GB card, start with a supported smaller conversion and a modest context for one real task. Note the exact file, runtime, cache setting, CPU/GPU split and whether the answers hold up. Consider a different model before paying for another card.

For an existing 24 GB card, a supported Q4 conversion is a reasonable starting candidate. Increase context only when the task requires it. A successful load does not establish that a long coding session, image workload or two simultaneous users will fit.

For a new purchase, we would investigate 24 GB before 16 GB for this model because it leaves more room beyond the weights. Compare the whole upgrade: card condition, power, cooling and the existing PC. Our used RTX 3090 buying notes cover those checks. This is a memory-planning recommendation, not a current price comparison. Very long context, vision or concurrent users still need a workload-specific memory check.

Research note, 1 November 2026: This section uses Qwen’s documentation and Unsloth’s file metadata, with owner anecdotes labeled separately. Considered Gear has not run Qwen3.8-27B on these cards, measured response speed or tested quantization quality. The original Qwen3 examples elsewhere in this guide retain their 10 October research date.

Why a 24 GB card can struggle with a 20 GB download

For a worked example, take Qwen3-32B. Its official configuration specifies 64 layers, eight key-value heads and a head dimension of 128, with sliding-window attention disabled.

Assume one sequence and an FP16 KV cache, using two bytes per value. Counting both keys and values gives this calculated cache storage per token:

2 × 64 layers × 8 KV heads × 128 head dimension × 2 bytes

= 262,144 bytes per token, or 0.25 MiB per token.

The formula follows the per-layer key and value tensor dimensions described in Hugging Face’s cache explanation. It is specific to these assumptions, not a formula for every model architecture.

These subtotals exclude working buffers and other GPU use. The last row already exceeds a 24 GiB budget. The first two rows are not fit guarantees. Different cache precision, allocation strategy or CPU/GPU placement changes what is held in VRAM. The Q4_K_M label describes the model weights; it does not mean this example’s cache is also four-bit.

Reserve context for the answer

Context includes more than the text you paste. System instructions, chat history, retrieved material, tool output and generated tokens all take space. With an 8,192-token budget, reserving 2,048 tokens for an answer leaves 6,144 for everything that comes before it. That is arithmetic, not a prediction of answer length or a word-count conversion.

Parallel requests can raise the memory requirement too. Do the first check with one request, then test the concurrency you actually intend to use.

Check the machine you already own with Ollama

Use a local model and a representative task. This is a diagnostic procedure; we have not run it as a benchmark for this article.

1. Record the exact local model, quantization, Ollama version, GPU, driver and operating system. A matching parameter count is not enough to reproduce a run.
2. In an ollama run session, set the context you intend to use. For example, /set parameter num_ctx 8192 requests 8,192 tokens; it is not a universal recommended size.
3. Send a representative prompt, then run ollama ps in another terminal while the model is loaded. Inspect PROCESSOR for the CPU/GPU split and CONTEXT for the allocated context reported by your version.
4. Exercise the full task, including a longer conversation or document and the other applications you normally keep open. A successful initial load is only the beginning of the check.
5. Record the result: allocation, CPU/GPU split, errors and whether the response time and answers are useful to you. Change one setting at a time when comparing.

Ollama documents context inspection and the interactive setting and processor split. Its documentation pages currently describe defaults differently, so this guide does not assume one default for every version. Set and inspect yours explicitly.

On NVIDIA hardware, nvidia-smi provides another view of memory use. NVIDIA distinguishes total, reserved, used and free framebuffer memory. Free memory matters more to a new allocation than the number printed on the box. A snapshot still does not show the peak of a whole session.

If the model does not fit

Try a shorter context if it still serves the task. Otherwise, try a smaller model or a different weight quantization and compare the answers. Neither is a free improvement: you are changing what the application can do.

Cache quantization is a separate setting from weight quantization. Support and quality tradeoffs depend on the backend and model. Read the applicable Ollama cache guidance before changing it.

CPU/GPU offloading can let a model use system RAM as well as VRAM. Software such as llama.cpp supports hybrid inference, but there is no fixed slowdown we can honestly promise. Measure the workload. More system RAM does not simply become more dedicated GPU memory.

Apple silicon uses unified memory, so the operating system, applications and model share a budget. Do not treat the whole advertised capacity as an exclusive model allocation. Our M4 Mac mini research notes cover that different buying choice.

Choose the whole computer after the memory check

If 24 GB of dedicated graphics memory looks appropriate, the used RTX 3090 buying notes cover compatibility and used-card checks. The card still needs a suitable desktop around it.

Use the SER8 versus used RTX 3090 complete-cost comparison to separate an upgrade to a computer you already own from buying everything. For the broader choices, see the home-AI hardware guide.

Sources and method

Original Qwen3 research checked 10 October 2026; Qwen3.8-27B update sources checked 1 November 2026. Qwen’s official Git LFS records give 5,027,783,488 bytes for the 8B file, 9,001,752,960 bytes for the 14B file and 19,762,149,024 bytes for the 32B file. The download table converts those values; the cache table uses the stated Qwen3-32B configuration and FP16 assumptions. Totals were calculated before rounding. Documentation and files can change after this date.

No GPU inference test, power measurement, throughput measurement or quality evaluation was performed for this article. There are no measured fit results or current hardware prices here. Use the calculations to choose what to test, then make a buying decision from your actual workload and complete-system cost.