Local AI comes from China and fits on a consumer GPU

Six days ago, closing out a piece on the memory shortage, I said I would try running AI locally this fall. I haven't installed anything yet. I started with the part nobody writes about, the part where you try to work out what you can actually load into your machine before downloading 11 gigabytes on a hunch. The answer came back in one sentence I hadn't seen coming. Almost every model I can run at home comes from China.

What fits on a consumer GPU

My machine is a gaming laptop with 24 GB of video memory, a card you buy in a store, nothing like the accelerators stacked in the data centers I wrote about last week. That's the part that interests me, because an AI that only runs on data center hardware isn't local AI. It's a subscription with extra steps.

"Open" and "hostable" turn out to be different things, and nobody says so plainly. Kimi K3, the open model everyone talked about this summer, weighs 2.8 trillion parameters and wants something like 1.5 TB of video memory, roughly 60 times what I have. The other two big Chinese names in open weights, DeepSeek and GLM, publish theirs too, and their best models are just as far out of reach for anyone without an air-conditioned rack in the living room. These models are open in the sense that you can download them, and closed in the sense that you can't do anything with them.

What actually fits in 24 GB, once the model is compressed the standard way, sits around 30 billion parameters, with a 27B model taking roughly 17 GB including context. That's the tier I'm going to install, and it happens to be the tier where the competition is fiercest.

The download numbers name the winner

Hugging Face, where these models live, is the central warehouse of open AI, the same one a swarm of OpenAI agents let themselves into in July. You upload a model's weights there, meaning the files that are the model, along with its documentation and its license, and anyone can download them, alter them, and republish their own version. Roughly what GitHub is to code, for files that run to tens of gigabytes. The company was founded in 2016 by three French entrepreneurs and keeps its offices in New York, which makes it the only more or less neutral referee in this story. It publishes a state of the union every summer, and the 2026 edition reads like a victory lap written in Hangzhou. In GGUF, the format local inference runs on, variants of Qwen, Alibaba's open family, pull 39.6 million downloads a month. Google's Gemma manages 20.8 million, and Meta's Llama drags along at 7.5 million.^[1] On the ground that matters here, people running a model on their own hardware, Meta barely registers.

The gap widens when you look at what people build on top. Qwen counts 151,448 derivative models on the platform, two and a half times Meta's entire footprint across every family it ships.^[1:1] A model you fine-tune and republish is a model you're betting on, and the ecosystem has picked its horse.

The same report carries the number that puts everyone back in their place, because models under one billion parameters account for 83% of all-time downloads, models above 100 billion for barely 1%, and in 2026 anything above 70 billion draws just 3%.^[1:2] The race to build giants fills the press releases while the real world runs small models on ordinary machines, and the center of gravity in local AI looks nothing like what the trade press describes.

That neutrality may not survive the year. The Information reported on August 26 that Nvidia had agreed to buy Hugging Face for $12.9 billion, a story Bloomberg and CNBC picked up the next day, with no signed agreement and no comment from either company.^[2] If it closes, it will be by far the largest acquisition in the chipmaker's history, and the marketplace where you download open models will belong to the company selling the cards you run them on.

A 27B model that argues with Opus

Alibaba shipped Qwen3.8-27B on August 14, and it's the first thing I plan to install. Twenty-seven billion parameters, dense rather than mixture-of-experts, an Apache 2.0 license that allows commercial use with nothing asked in return, a native context window of 262,144 tokens that stretches to a million, and text, image, and video input.^[3] It fits my card with room left over.

The published numbers are worth reading all the way down. On SWE-bench Pro, which measures whether a model can close real tickets in real repositories, Qwen3.8-27B reports 61.7 against 53.4 for Claude Opus 4.6 Max. On Terminal-Bench 2.1, Opus takes it back with 78.2 against 73.0.^[3:1] Those measurements come from the Qwen team, run partly on Claude Code's own harness, and no one had reproduced them independently at launch. I've watched several write-ups announce that a 27B model "beats Opus on 16 of 24 benchmarks," a phrasing with the virtue of fitting in a headline and the flaw of picking its benchmarks.

The honest version impresses me more than the loud one. A model you download for free, run on a graphics card you already own, and use commercially without asking anyone now holds a conversation with a frontier proprietary model on some coding work. Two years ago that sentence would have been laughable.

The student copying off the top of the class

That leaves the question of how a 27-billion-parameter model gets there at all, when training a frontier model from scratch swallows hundreds of millions of dollars and years of research. The answer is distillation, and the principle fits in three sentences.

A large trained model, the teacher, produces answers and, more to the point, the reasoning that leads to them. A small model, the student, trains not on raw web text but on those answers, until it reproduces the teacher's behavior. DeepSeek distilled its R1 reasoning model into 7-billion and 32-billion variants from 800,000 hand-picked examples, which is how a small model ends up performing well above its weight class.^[4]

The economics are startling, since the student never repeats the research and simply buys the result for the price of a few million API calls. That also explains the ceiling, because whoever copies doesn't outrun the model they copied, at least not on the part they copied. Chinese open models catch up within months without ever quite taking the lead, and they shine exactly where the frontier models were shining last year.

The method carries a legal bill as well. Back in June, Anthropic accused Alibaba of querying Claude nearly 29 million times through 25,000 fake accounts to train Qwen. So I'm about to install a model that Anthropic suspects of having learned by copying the very one I plan to hand my plans to, which lends my arrangement an irony I hadn't budgeted for.

Claude writes the plan, my laptop does the typing

I'm not canceling my subscription, which I've already admitted costs me real money, because that isn't where local models have something to offer. What I want to keep online is the steering, the part where you break a problem apart, choose between two approaches, and read back what came out. A few points of benchmark gap cost you dearly there, since a bad plan burns a day while a bad line burns thirty seconds.

Execution can come home. A local model needs far more direction than a frontier one, which is its well-known weakness and I have no interest in pretending otherwise. But I don't have to write that direction myself. When the big model drafts the instructions, splits the task, and spells out what the result should look like, the small model gets exactly the fuel it was missing. The weakness of local gets patched by the strength of remote, which is an elegant way of refusing to choose.

The tooling already exists. Cline paired with Ollama, or Continue.dev in agent mode, were holding down real work without constant supervision as early as spring.^[5] Kilo Code splits Plan, Code, and Debug into separate modes and gives each its own provider, so you hand planning to Claude and execution to a model running an inch below the keyboard.^[6] I'll be skipping Roo Code, which has announced it's shutting down.

Two details made me laugh along the way. LM Studio, the app I'd recommend to anyone who doesn't want to open a terminal, sends anonymous usage statistics by default, in software whose entire pitch is that nothing leaves your machine. One checkbox, but still. And Ollama, the best-known tool in the field, now sells cloud subscriptions at $20 and $100 a month.^[7] The tool for AI that runs on your own hardware ended up renting servers, and I can't decide whether that's irony or just a business model.

Stripping the censorship strips everything else

A Chinese model arrives with whatever refusals its publisher built in, and the question comes up the moment you decide to install one at home. A tool exists for that, and it is well named. Heretic, published on GitHub, bills itself as "fully automatic censorship removal" and claims its community has already republished more than 5,000 models.^[8]

The method is clever. It locates the direction inside the model's internal representations that corresponds to refusal, then makes the weight matrices orthogonal to that direction, which strips the model of its ability to say no without touching anything else. Because the tool tunes itself to minimize both refusals and drift from the original, it reports three refusals out of a hundred on Gemma-3-12B at a divergence of 0.16, where competing tools land between 0.45 and 1.04.^[8:1] The uncensored model stays about as smart as it was, which was not a given.

Read what the tool removes, not what its name promises. The refusal direction doesn't tell a political question apart from a request for synthesis instructions, so taking out one takes out the other. This isn't targeted decensoring. It's a model that has lost the ability to refuse anything at all. The repository carries no warning either, no section on use, just technical documentation and results tables. I'll only note that swapping a Chinese publisher's values for no values whatsoever isn't quite the same thing as taking back control.

That said, I know myself well enough to be sure I'll never ask my graphics card how to build a bomb. What I want from an execution model is that it writes the function I described without first explaining that the subject calls for caution, and on that front a model that has lost the habit of refusing would mostly be doing me a favor. So it depends less on the tool than on the hand holding it, and in mine it would strip out little more than refusals that protected nobody.

The fan gets the last word

Last week I listed the screaming fan among the trade-offs I accepted when I bought a laptop. I suspect that trade-off is about to become my real ceiling. My card is rated for 95 to 150 W, and sustained inference is precisely the load that brings a chassis to its knees, since the first minute draws on the thermal mass of the lid before the machine settles down to whatever the fans can actually carry away.^[9] On battery it throttles further still.

One step is worth watching for, because the day a model spills out of video memory into system RAM, throughput collapses by a factor you count in tens. That isn't a slope you negotiate. It's a wall. I'll find out in October which side of it I'm on, and I'll report back with numbers measured on my own machine rather than the desktop-card figures floating around everywhere, which mean nothing here.

One contradiction is left for me to own. I spend a fair amount of time writing that Europe regulates an AI it doesn't build, and I'm about to install a Chinese model to get out from under an American service. From a distance the progress looks thin. It's real all the same, because the weights will sit on my disk, nothing will leave the machine, and no change to anyone's terms of service will take back what I've downloaded one morning. That's the whole difference between using a Chinese model through an interface and owning the file. Sovereignty, at my scale, has less to do with the flag on the model than with who can take it away from me.