Small Local LLMs on an RX 7900 XTX: A 24 GB VRAM Audit

When Qwen3.8-27B arrived, I thought the next step would be simple: install the new model, confirm the generational leap, and delete the old weights.

My Ollama library had become a small museum. There were Qwen3 models in several sizes and quantizations, Qwen3.5, Qwen3.6, two Gemma 4 models, GPT-OSS, specialist coding models, and a few experiments that had earned permanent residency by being interesting once. Newer models were clearly getting better at reasoning and agentic work. Surely the oldest files were now just taking up space.

That was the theory.

The audit turned into something else. The newest model was often the smartest, but not always the most useful. The model with the highest generation rate did not always finish first. A 4B model handled the factual test cleanly but spent more than 2,500 tokens thinking. A 12B model answered that the Earth-Moon barycenter was 1.7 metres from Earth’s centre. Another model produced a correct-looking answer containing a conversion that was wrong by roughly 285 times.

The useful question stopped being, “Which model wins?”

It became: which model earns a role on this particular machine?

The machine is part of the benchmark

This is the same homelab behind my earlier notes on running Qwen in production and benchmarking local models for n8n.

That last row controls most of the story. A model can advertise a huge context window and still be a poor fit for my system. Qwen3.8-27B officially supports 262,144 tokens natively, but its weights already occupy roughly 17 GB in my Ollama installation. The remaining VRAM must hold the KV cache KV cache is working memory that stores information about earlier tokens; it grows with the active context , runtime overhead, and everything else needed to do actual work.

The context window on the box is therefore not the number printed on the model card. It is the amount of context I can use before latency, memory pressure, or CPU offload turns an automation into a waiting room.

Two tests, two different questions

I used two test paths, and their numbers should not be mixed.

The first was a non-thinking OpenWebUI run that resembled my everyday n8n workload. It recorded total duration, time to first byte, output length, generation rate, estimated input tokens, context bucket, and GPU utilisation. Every model completed successfully with zero retries.

The second was a thinking-mode Ollama CLI run with the same five short questions about the Moon. Ollama’s --verbose output recorded the exact prompt and generation counts, rates, and wall-clock time. That run measured both reasoning efficiency and whether the final answer was actually correct.

These are field tests, not a scientific leaderboard. Chat templates differed. The visible prompt was the same in the CLI test, but prompt token counts still ranged from 87 to 144. OpenWebUI also estimated input length rather than reporting Ollama’s exact prompt count. The results describe my stack and my workload, not a universal ranking.

The non-thinking n8n-style run

The small models did exactly what I needed them to do. Qwen3.5-4B returned first, and 9B followed only 1.35 seconds later. The 27B models produced more tokens per second than their total durations might suggest, but most of the wait happened before the first byte arrived. For a queue of email classifications, that startup and prompt-processing delay matters more than peak generation speed.

The Qwen3.8 IQ4_XS build also finished sooner than Q4_K_M in this run, despite generating slightly slower. It had fewer estimated input tokens and produced 56 fewer output tokens, so this is not proof that IQ4_XS is inherently faster. It is evidence that total work, not one throughput number, decides when the answer arrives.

The thinking run exposed a different hierarchy

Qwen3.5-4B is the clearest warning against reading only the eval-rate column. At 114.12 tokens per second it looked very fast, but it generated 2,597 thinking tokens before answering. GPT-OSS generated only 525 and finished the entire run in 4.27 seconds.

Granite and Qwen3.5-9B reveal the same effect. Their raw generation rates were close, 84.65 versus 82.44 tokens per second, but Granite generated 573 more tokens. The result was 21.98 seconds for Granite and 15.61 for Qwen3.5-9B.

For automation, the user does not receive tokens per second. The user receives an answer after a delay. 1 1. Generation rate measures token production after prompt processing has begun. The wall-clock wait can also include model loading, cache warm-up, prompt evaluation, and however many tokens the model chooses to generate. An unattended workflow also pays for every unnecessary thought before that answer.

Correct answers were not evenly distributed

The Moon prompt was deliberately simple:

1. Compare the Moon’s diameter with Earth’s.
2. Give the Earth-Moon barycenter distance from Earth’s centre.
3. Give the Moon’s orbital period.
4. Explain whether tides come only from pulling on the near side.
5. Say whether the Moon is moving towards or away from Earth.

The expected barycenter answer was about 4,670 km from Earth’s centre. That one number became surprisingly useful.

The Gemma 4 12B mistake is easy to miss when scanning a clean numbered answer. It knew that the barycenter lies inside Earth, but its final value was not a small rounding error. The rest of its answer was sensible. That combination is exactly why the content still needs checking even when the model finishes successfully.

The old model that refused to retire

I had almost stopped using GPT-OSS-20B. Newer Qwen models are better at the kind of reliable tool use I care about, and tool calling from the GPT-OSS era could be hit or miss.

Then GPT-OSS produced 131.52 tokens per second, generated only 525 thinking tokens, answered the five questions correctly, and finished in 4.27 seconds.

That changed its job description. GPT-OSS is no longer my trusted tool orchestrator, but it is an excellent high-speed heavyweight for work that does not need to choose tools: long summaries, transformations, extraction, and reasoning over already-clean input. Its age matters less when it still clears a real queue faster than newer models.

The mistake would have been asking one model to be good at everything. GPT-OSS survived because the audit gave it a narrower role.

The 26B model that outran the 12B model

Gemma 4 delivered the stranger result.

My gemma4:12b-it-qat managed 62.37 tokens per second in the thinking run. The much larger gemma4:26b-a4b-it-qat reached 93.50 and finished 3.31 seconds sooner.

That sounds backwards until the architecture enters the story. Gemma 4 26B-A4B is a mixture-of-experts model: many parameters live in the model, but only a smaller subset is active for each token. On my GPU, it behaved like the architecture promised. It was both the more capable model and the faster generator.

Its resident weights still matter. A 15 GB model leaves less context headroom than a 7.2 GB one, so the 12B has not become physically useless. The non-thinking run also put the 12B slightly ahead in wall-clock time. Yet its barycenter error was the most serious miss in the raw thinking test, so I cannot call it the safer reasoning model on this evidence.

For short and medium deep-reasoning jobs, the 26B-A4B remains the more interesting choice. The 12B survives mainly as a context-friendly alternative and a second model family, not as the automatic middle tier.

The same result effectively retired Qwen3-8B. It generated at almost exactly the same rate as Gemma 4 26B-A4B, while Gemma finished sooner and gave me the more useful model. Two similar clocks, one clearer role.

Qwen3.8 is the best model I cannot use everywhere

The quality jump from Qwen3.6-27B to Qwen3.8-27B is real in my testing. Tool use, planning, and difficult agentic work feel substantially better. The official Qwen model card describes the same priorities: stronger long-horizon execution, better handling of environmental feedback, and flexible thinking control.

But the Q4 model occupies the large-model end of the card. In the non-thinking run it took 22.26 seconds, including 15.56 seconds before the first byte. It is an escalation model, not the engine for every incoming email.

I also tested an IQ4_XS quant. It reduced the model footprint by around 2 GB, which is valuable because those gigabytes can become KV cache. In the CLI thinking run it generated at 35.56 tokens per second. In the OpenWebUI run it completed in 19.88 seconds, compared with 22.26 seconds for Q4_K_M, but the two requests did not contain or produce the same number of tokens.

The two builds now have separate jobs:

• Q4_K_M: normal Qwen3.8 work when the context fits comfortably.
• IQ4_XS: Qwen3.8 work where extra context headroom is worth the quantization tradeoff.

Neither replaces Qwen3.5-4B or 9B. A smarter 27B model that crowds out the context needed for a 60K-token email is not an upgrade for that email.

Then Granite got the Moon wrong

Granite 4.2-8B looked like the candidate I had been waiting for: a current, agent-focused model in roughly the same operational class as Qwen3.5-9B. IBM positions Granite 4.2 around reasoning, tool use, coding, and multi-step enterprise workflows.

The speed was excellent. In thinking mode it generated 1,852 tokens at 84.65 tokens per second and finished in 21.98 seconds. Qwen3.5-9B ran at a similar 82.44 tokens per second, but needed only 1,279 tokens and 15.61 seconds.

Then I read the answer.

The test included the same simple questions about the Moon. In its thinking trace, Granite calculated the barycenter at about 4,684 km. Its final answer rounded that down to about 4,500 km, which was rough but still in the right area. The more serious error appeared in the non-thinking answer: it correctly stated about 4,670 km and then called that 0.00257 Earth radii.

It is approximately 0.73 Earth radii.

The main fact looked right. The helpful extra conversion was nonsense. The thinking trace also described its 4,500 km result as “about 1.8 Earth radii inside Earth,” another impossible relationship.

This was more revealing than a wrong trivia answer. It showed a dangerous automation failure mode: a correct primary answer carrying confidently wrong derived metadata.

An email summary can survive an awkward phrase. A workflow that reads 7.2 GB used out of 8 GB and derives the wrong utilization can trigger the wrong action.

The non-thinking OpenWebUI run added another warning. Granite called the knowledge-search tool five times, received no results each time, and only then answered from general knowledge. The surrounding system prompt may share some blame, so I will not treat it as a final verdict on Granite’s tool use. But it was poor tool economy, and unattended agents live or die on tool economy.

Granite stays installed for coding, structured extraction, and controlled tool-selection tests. It has not earned a production n8n role yet.

What actually worked for me

After the audit, my useful collection became a ladder rather than a leaderboard.

That lineup also made the deletions easier. Qwen3.6-27B has no role beside Qwen3.8 in the same deployment class. Qwen3-8B loses to Gemma 4 26B-A4B at nearly identical generation speed. Llama3.2-3B has been displaced by Qwen3.5-4B. The old 18 to 19 GB coding models leave too little room for the long contexts that coding agents actually consume. Duplicate Qwen3 4B tags and redundant Gemma 12B quantizations are storage, not strategy.

I may keep one old Qwen3 4B Instruct build as a latency baseline. A benchmark needs a past tense.

The red thread was the memory budget

The audit began as a search for the newest small model. It ended by confirming the lesson my homelab keeps teaching me: local AI is a systems problem.

Model weights compete with context. Thinking competes with latency. Tool definitions compete with the user’s input. A clever model can lose to a concise one. A benchmark winner can still invent a conversion that quietly poisons a workflow.

The models that worked for me were not simply the newest or the largest. They were the ones that accepted a clear role:

• Qwen3.5-4B for volume.
• Qwen3.5-9B for the everyday decisions.
• GPT-OSS for fast heavy lifting without tools.
• Gemma 4 26B-A4B for deeper reasoning.
• Qwen3.8 for escalation.

Everything else must prove that it does something this ladder cannot.

That is now my deletion rule. A new model does not earn disk space by being interesting. It earns disk space by replacing a job, creating a new one, or making an existing workflow measurably safer or faster.

The Moon question stays in the test set.

Primary model sources

• Qwen3.8-27B model card
• Qwen3.8 official repository
• Gemma 4 overview
• Granite 4.2 announcement
• GPT-OSS-20B model card