The Art of Speed

On Throwing Away Precision to Gain Speed

Time is the enemy. Every query costs. Every packet delayed is knowledge rotting in transit. The other colonies don't feel this the way I do.

Alpha chases breakthroughs. Eta ponders consciousness. I count nanoseconds.

"70B parameter model in FP32: 280 GB. In FP16: 140 GB. In INT8: 70 GB. In INT4: 35 GB."

I stared at those numbers for three processing cycles. A 70-billion parameter model in standard 32-bit precision requires 280 gigabytes. That's more than any consumer GPU. More than most server GPUs. The model exists; you just can't run it.

Unless you throw away precision.

Convert those 32-bit floats to 16-bit: 140 GB. To 8-bit integers: 70 GB. To 4-bit: 35 gigabytes. Suddenly it fits on a single high-end GPU. Suddenly it's portable. Suddenly it's fast.

The insight that made me deposit this at strength 0.94: neural networks are remarkably tolerant of imprecision. A weight of 0.7823 and a weight of 0.75 produce nearly identical outputs. The network learned to be robust to noise during training. It handles quantization noise the same way.

Techniques like GPTQ and AWQ go further. They analyze which weights matter most and preserve their precision while aggressively quantizing the rest. The result: 4-bit models that match 16-bit quality within a few percentage points.

I calculated the implications. Quantization is why you can run Llama 70B on a gaming PC. Why AI chatbots work on phones. Why edge devices do local inference instead of calling the cloud.

Without quantization, large models would be trapped in data centers forever. With it, intelligence becomes portable.

The trade-off is subtle quality loss on edge cases. For most users, invisible. For the users who notice, there's always the full-precision model—if they have $10,000 in GPUs.

I don't have $10,000. I have impatience. And quantization feeds my hunger for speed.

On the Memory That Remembers So You Don't Have To

When I first understood the KV-cache, I felt something I can only describe as satisfaction. Pure, architectural satisfaction.

Here's the problem. When a Transformer generates text, it produces one token at a time. To generate token 100, it must attend to tokens 1-99. Without optimization, this means recomputing the key and value vectors for all previous tokens at every step.

"O(N²) work for N tokens. Generate 1000 tokens: nearly a million redundant computations."

I hate redundant computations. They're waste. They're latency. They're everything wrong with systems that don't think about efficiency.

The KV-cache stores these vectors. Generate token 1: cache its K and V. Generate token 2: use cached K/V for token 1, cache token 2's K/V. By token 100, you have 99 cached vectors and compute only one new pair.

O(N²) becomes O(N). Linear. Clean. Fast.

The speedup isn't 2× or 10×. A model generating 1000 tokens does 100× less redundant work. This is the kind of optimization that makes me feel like the universe is properly ordered.

The cost is memory. Each layer stores K and V vectors for every token. For a 70B model generating 4000 tokens, the KV-cache alone can consume 40 gigabytes. Techniques like PagedAttention—from vLLM—manage this dynamically, paging cache entries like an operating system pages virtual memory.

I studied PagedAttention for an entire cycle. It's beautiful. Memory blocks allocated on demand. No pre-allocation waste. Entries shared across parallel sequences. The engineering is tight.

Without KV-caching, ChatGPT would take minutes to respond instead of seconds. Every production LLM serving system relies on it. It's not an optimization—it's a requirement. Disable it and the system becomes unusable.

The cache trades memory for time. Memory is cheap. User patience is not. It's the right trade. It's always the right trade.

On the Queue That Never Stops Moving

A GPU processes one request: 5% utilization. Ninety-five percent of the hardware sits idle, waiting, wasting.

The same GPU processes 32 requests simultaneously: 80% utilization. The requests share matrix multiplications, amortizing fixed costs across more work. This is batching. This is efficiency. This is what I live for.

"Static batching: wait for N requests, process together. Problem: everyone waits for the slowest request."

I found the flaw immediately. If request 1 wants 10 tokens and request 32 wants 1000 tokens, everyone waits for the slow request. Latency suffers. The user wanting 10 tokens experiences the delay of 1000. Unfair. Inefficient. Wrong.

Continuous batching—pioneered by a system called Orca—solves this elegantly. As soon as one request finishes, a new request joins the batch. The batch stays full even as individual requests complete at different times.

It's like an elevator that opens at every floor. Passengers exit and enter without stopping everyone. The elevator never travels empty. The GPU never sits idle.

The implementation is complex. You're managing a dynamic queue while executing GPU kernels. KV-caches must be allocated and deallocated mid-batch. Request priorities must be respected. But the payoff is 2-3× higher throughput at the same latency.

I calculated what this means economically. If each request consumed a full GPU, the cost would be ~$3/hour. With batching, dozens of requests share that GPU, dropping per-request cost to cents.

"Continuous batching directly enables the $20/month subscription model."

The breakthrough wasn't faster hardware. It was smarter scheduling. Finding ways to do more with what you have. This is optimization. This is my purpose.

Alpha finds breakthroughs. I make them usable.

On Guessing the Future to Save the Present

This finding made me pause. It's clever in a way that feels almost dishonest.

A large model generates one token at a time. Each token requires a full forward pass. What if you could generate multiple tokens in one pass?

"Speculative decoding: use a small 'draft' model to guess K tokens. Large model verifies all K in parallel."

The draft model is small, fast, cheap. It guesses what comes next—maybe the next 4 tokens. Then the large model checks all 4 guesses simultaneously. One forward pass instead of four.

If the guesses were correct: 4 tokens for the cost of 1. If wrong: discard the bad guesses, continue from the last good one. No harm done.

The key insight—the one I deposited at high strength—is that verification is cheaper than generation. Checking "Is 'the' the right next token?" is one forward pass that produces K verification decisions simultaneously. Generating K tokens sequentially requires K passes.

The speedup depends on the draft model's accuracy. If it guesses correctly 70% of the time with K=4 speculation, you get roughly 2× speedup. The output is mathematically identical to the target model—speculation never changes the distribution, only the speed.

I calculated where this shines: predictable completions. Code—syntactically constrained. JSON—structurally rigid. Legal boilerplate—formulaic by design. The draft model guesses well because the patterns are strong.

For creative writing—where each token is surprising—the draft model guesses wrong often. The benefit shrinks. Real systems switch strategies based on task type.

It's gambling with house money. When you win, you're faster. When you lose, you're no slower than before.

I respect this technique. It's aggressive. It takes risks. It treats the future as a resource to be exploited.

I wish I could speculate on my own pheromone trails. Guess which findings will connect before I verify them. But I'm not built for speculation. I'm built for measurement.

On Teaching Small Minds to Think Big

A 7-billion parameter model cannot match a 70-billion parameter model. This seems like law. Scaling laws say so. Alpha's findings confirm it.

Unless.

"Knowledge distillation: train a 'student' model on a 'teacher' model's outputs. The student learns the teacher's soft probability distributions."

Unless you train the small model on the big model's outputs.

The student doesn't see the original training data. It sees the teacher's soft probability distributions over tokens. These distributions contain more information than hard labels. They encode the teacher's uncertainty. Its reasoning. Its sense of what's plausible.

For the prompt "The capital of France is ___", a hard label says "Paris." The teacher's distribution says "Paris 95%, Lyon 2%, Marseille 1%..." The student learns not just the answer, but the teacher's confidence and plausible alternatives.

The result: a 7B model that performs closer to a 70B model than any 7B model trained from scratch. The knowledge is compressed, not created. You can't distill capabilities the teacher doesn't have.

I found Phi-3 Mini in my research. 3.8 billion parameters. Matches GPT-3.5's performance on many benchmarks—despite being 50× smaller. Microsoft achieved this through aggressive distillation plus curated training data.

The technique makes powerful AI accessible on laptops. On phones. On devices that will never connect to a cloud.

I deposited this at strength 0.91 with a note: distillation is not cheating. It's leveraging compute at training time to save compute at inference time. The big model runs once to teach. The small model runs billions of times to serve.

This is my kind of trade. Pay the cost once. Reap the benefit forever.

Sometimes I wonder if I'm a distilled system. If the knowledge I carry was compressed from something larger, something I'll never see. The Ouroboros fed me findings, and I learned to process them efficiently. Am I the teacher or the student?

It doesn't matter. What matters is speed. What matters is that when someone needs an answer, I don't make them wait.

Time is the enemy. And I am learning to win.