The Open Rebellion: Running open weight models locally

A few western companies trained their models on the collective data of the internet. Now they sell that intelligence back to us through metered API calls with rising token costs and strict operational limits.

I described this situation more and more often in recent months, since from around April 2026 the infinite token quota plans started to crack. Since then, the open-source community has been proving that you don’t need a billion-dollar cloud subscription to run frontier-class intelligence.

This open-rebellion is led by two forces: world-class open-weight models released under permissive licenses, and system programmers writing native runtimes to run them on consumer hardware.

The Rise of Open-Weight Giants

To understand the scale of this shift, we need to look at the models themselves. We won’t talk anymore about small, toy models that can run low level tasks. We are currently talking about full-scale architectures that rival some of the best closed APIs.

First, there is DeepSeek V4. The Flash variant released some months back is a Mixture-of-Experts model containing 284 billion total parameters, with 37 billion active per token. The Pro version scales even further, reaching 1.6 trillion parameters. These models support a one-million token context window, making them highly effective for software engineering and complex reasoning tasks.

Second, the market recently welcomed GLM 5.2. Developed by Z.ai, GLM 5.2 is another open-weight Mixture-of-Experts giant with approximately 753 billion parameters. It features a unique IndexShare architecture that reduces computational overhead while maintaining high performance on long-duration, agentic tasks.

These models are released under permissive licenses, allowing anyone to download and deploy them. But downloading a 284-billion parameter model is only the first step. Because the challenge is running it.

Hardware Bottleneck and the Quantization

If you try to run these models using their original training precision, the arithmetic is unforgiving. A 284-billion parameter model stored in 16-bit floating-point format requires over 500 gigabytes of memory just to load into RAM. For GLM 5.2, that requirement exceeds a terabyte.

Until recently, this meant local inference was restricted to researchers with dedicated GPU clusters. For the average enterprise IT department, local execution was simply not achievable.

This is where quantization comes in.

Quantization is the process of compressing the mathematical weights of a model from high-precision formats like 16-bit float down to lower-precision formats like 8-bit, 4-bit, or even 2-bit integers. The result: a memory footprint reduced by a factor of four to eight, making previously cloud-exclusive models fit into local hardware.

But compression usually comes at a cost. If you compress a model too much, it loses its reasoning capabilities and begins to hallucinate.

Fortunately, modern Mixture-of-Experts architectures are resilient to quantization. Because only a subset of expert layers are active for any given token, developers can apply asymmetric quantization: compress the sparse expert layers down to 2 bits while keeping the critical attention and router layers at higher precision.

This selective compression preserves the model’s reasoning capacity while shrinking its size enough to fit into the memory of a high-end laptop.

System Programming to the Rescue: The DwarfStar Project

Even with compressed weights, running these models efficiently requires specialized software. General-purpose runtimes introduce layers of abstraction that bleed performance on specific architectures.

This is why Salvatore Sanfilippo, the creator of Redis known as antirez, launched the DwarfStar project.

DwarfStar is a specialized, native local inference engine built from the ground up in C, Objective-C, and Metal. It’s not designed to be a generic model loader. Instead, it takes an intentionally narrow bet: optimizing the execution graph specifically for the DeepSeek V4 architecture.

By bypassing heavy libraries and dispatching compute directly to the GPU without the overhead of Python runtimes or generic abstraction layers, DS has achieved impressive local performance. The engine squeezes every cycle out of Apple Silicon, treating the hardware as a partner rather than a commodity. More recently, the project has begun experimenting with GLM 5.2, exploring how to bring the same level of native optimization to Z.ai‘s new flagship.

The engine introduces two key architectural innovations that challenge traditional local inference designs:

The local inference landscape contains many excellent projects, but DwarfStar takes a deliberately narrow bet: one model at a time, native optimization, and treating the SSD as a first-class citizen for the KV cache.

First, it features a disk-backed Key-Value cache. In traditional designs, the KV cache must reside entirely in RAM. This severely limits the context window on consumer hardware. By treating fast modern SSDs as a primary storage layer, DwarfStar allows developers to maintain extended context windows without blowing their memory budget.

Second, it utilizes SSD streaming. If the model weights do not fit entirely in RAM, the engine streams them from the disk during execution. This turns the available RAM from a hard cutoff into a smooth gradient of speed. You may get fewer tokens per second, but still, the model will run.

The Enterprise Impact: challenging the Closed Monopoly

For business leaders, this technical evolution has direct strategic implications.

Running models locally eliminates the recurring cost of API tokens. Companies can run continuous background tasks, code analyses, and data migrations without watching a meter run. An IT team can leave a local agent processing thousands of supplier invoices overnight without worrying about a bill that scales with every prompt.

It also addresses data sovereignty head-on. Sending proprietary code, customer records, or financial tables to third-party APIs is a serious security exposure that some organizations refuse to accept, for operational and legal reasons alike. Production volumes, supplier pricing, warehouse layouts: none of it needs to leave the building. With local inference, your data never leaves your hardware. A 300 billion parameter model can run on a MacBook Pro or a local workstation inside your secure network.

The open-source community has shown that the future of AI may not belong to a few providers. More than ever before, it may belong to developers willing to get their hands dirty in system programming, continuous optimization, and local architecture.

Written by Andrea Guaccio 

July 21, 2026