The Neo Geo Was Not Faster. Neither Is Apple Silicon.

Sizing a model for hardware you do not control?

I agree to the Privacy Policy Talk On-Device Inference

The Neo Geo ran a Motorola 68000 at 12 MHz. The Sega Mega Drive ran the same chip at 7.67 MHz.

That is a 1.6× clock advantage. It explains almost nothing.

Put the two machines side by side in 1990 and the gap does not look like “half again as fast.” It looks like a generation. One of them runs an arcade cabinet. The other does an impression of one.

So the performance came from somewhere the clock speed never mentions.

Thirty years later, the same sentence gets written about Apple. The M1 shipped at 3.2 GHz while desktop x86 parts boosted past 5. It won anyway, on the work people actually do, drawing a fraction of the power. Again, the number everyone quotes explains nothing.

Two machines, three decades apart, making one argument: performance is not a rate. It is a shape. You do not get it by making every computation faster. You get it by building a machine around the one computation you care about, until that computation becomes cheap.

The rest of this piece walks that idea in order: what SNK bought instead of speed, the one number that turns “designed around a computation” into arithmetic, what that number does to four ARM machines running the same file, what Apple put on the die, what specialization costs when the work changes, and what to calculate before you pick a model.

What SNK bought instead of speed

Start with what the Neo Geo did not have. No exotic CPU. No instruction the Mega Drive lacked. Both machines ran a 68000, and a programmer moving between them wrote the same assembly.

What SNK bought was budget in the places that make pixels.

The Mega Drive drew 80 sprites, 20 of them on any single scanline, in 61 colors at once. The Neo Geo drew 380 sprites, 96 per scanline, in 4,096 colors from a palette of 65,536. Five times the sprites per line. Sixty-five times the colors. Against a 1.6× clock.

Then the cartridge. Mega Drive games shipped in tens of megabits, and the largest ever made held 40. Neo Geo cartridges grew past 700 — close to 90 megabytes of art and audio, addressed directly by the hardware, with nothing sitting between the artwork and the screen.

One design decision gives the whole thing away. The Neo Geo has no real background layer. Its competitors composited tilemaps; the Neo Geo draws backgrounds out of sprites and keeps only a small 8×8 text layer for the interface. One drawing primitive, in dedicated silicon, fed by an enormous ROM. Sound runs on separate hardware — a Z80 and a Yamaha YM2610 — so audio never competes with drawing.

SNK did not build a general graphics system and then tune it. They picked one job, moving huge amounts of pre-drawn pixel data from cartridge to screen sixty times a second, and deleted everything that did not serve it.

Which is why the Neo Geo did not run fighting games fast because it was a fast computer. It ran them fast because it was barely a computer. It was a sprite mover with a 68000 attached to decide what to move.

The number both machines answer to

Nobody puts a pipe this size on a spec sheet. It still decides what the machine can do.

That is a good story. It becomes engineering when it becomes a number, and the number is the same in 1990 and today.

Bytes moved per unit of output.

For the Neo Geo, the output is a scanline and the bytes are sprite data pulled from ROM. The hardware exists to keep that number affordable at 96 sprites, sixty times a second. Miss it and sprites drop; the player sees flicker.

For a language model on your laptop, the output is a token and the bytes are the weights. Same structure. Different decade.

This one is worth deriving, because the derivation is what makes the limit exact.

Generating a token requires the model to consult every parameter it has. No shortcut exists — there is no partial forward pass that skips most of the network. And the parameters do not fit in cache: an 8B model at four bits is about 4.9 GB, while the largest cache on any machine here is tens of megabytes. So the weights live in DRAM, and every token reads all of them.

That gives a ceiling with nothing tunable in it:

tokens/second  ≤  memory bandwidth  ÷  size of the weights

This is the roofline model, applied to work that sits far from its ridge. Williams, Waterman and Patterson wrote it down in 2009: attainable performance equals min(peak compute, bandwidth × arithmetic intensity), where arithmetic intensity counts FLOPs done per byte moved. Decoding one token at a time does about two FLOPs per weight — roughly four FLOPs per byte at four-bit quantization. The point where a machine stops starving and starts computing sits in the tens.

Four against tens. No kernel rewrite closes that, because arithmetic intensity belongs to the algorithm, not to the code.

The machine is starving. Bandwidth is the width of the straw.

Which is SNK’s problem exactly. Both machines are limited by how fast bytes arrive, not by how fast arithmetic happens once they do. And both arrived at the same answer: stop tuning the arithmetic, go buy bandwidth.

Four ARM machines, one file

Take one artifact and carry it through: Llama 3.1 8B, quantized to Q4_K_M, about 4.9 GB resident. One file. It loads on all four machines below without a recompile, because all four speak the same instruction set.

A thirty-fold spread, across four machines a compiler treats as interchangeable.

ARM standardizes instruction encoding and the register model a programmer sees. It does not standardize bandwidth, cache, accelerators, or whether the GPU can read what the CPU just wrote. Every one of those carries weight here.

Graviton4 is the row worth staring at. Twelve channels of DDR5-5600 give it roughly 537 GB/s — M4 Max territory, on a server CPU — and it still decodes single-stream at a fraction of the speed. The bandwidth is real. General-purpose cores cannot keep enough loads in flight to spend it.

So the honest version of the rule is narrower:

Bandwidth sets the ceiling. The rest of the machine decides how close you get.

That gap is precisely what the Neo Geo’s sprite hardware existed to close. It is also where Apple spent the M-series budget.

A box full of specialized machines

Nobody sharpens a chisel to make it a better hammer. They buy a hammer.

Apple’s answer is not one accelerator. It is several, on one die, sharing one pool of memory. Each one exists because some computation was too expensive on the general path.

Unified memory removes the copy. CPU, GPU and Neural Engine address the same DRAM, so a tensor written by one is visible to the others at the same address. MLX puts this directly in its API: arrays live in shared memory, and an operation names a device rather than triggering a transfer to it. CUDA is the contrast — there, cudaMemcpy is a line item you budget for, and a 24 GB card cannot run a 30 GB model at any speed. On an M-series machine the limit is the RAM you bought.

Performance and efficiency cores split a scheduling problem, not a compute one. The token loop wants sustained throughput on a few threads. The application around it wants responsiveness at low power. One core type tuned for both loses twice.

Vector and matrix units — NEON, Apple’s AMX, SME on the M4 — make small dense multiplies cheap, which is where the CPU path spends its time when the GPU sits idle.

The Neural Engine is the closest thing to the sprite chip: fixed-function silicon, 38 TOPS on the M4, excellent at the quantized patterns it was designed around, useless for anything else.

Bandwidth is the part that is not clever at all. 120 GB/s on the base M4, 546 on the M4 Max. Apple bought it the way SNK bought ROM — because the work is bound by it, and no architecture substitutes for it.

Put together, this is not a fast general-purpose computer. It is a set of specialized machines that agreed to share an address space.

What specialization costs

Every specialized machine is one workload change away from being furniture.

The Neo Geo’s virtue and its defect are the same fact.

A machine built around one computation stays excellent until the computation changes. SNK built for moving pre-drawn sprites, and the industry moved to real-time 3D. Nothing in that sprite engine helps rasterize a textured triangle. The same specification shipped from 1990 to 2004, because the guarantee was the product and revising it would have destroyed it. The machine that looked unfair in 1990 was a museum piece by 1996, and slowness had nothing to do with it.

The modern version is smaller but real. Fixed-function silicon is sized for the operators that existed when it was taped out. A model whose operations do not map cleanly onto the Neural Engine does not run slightly worse there — it falls back to the GPU, and the accelerator sits dark, already paid for in die area.

The scoreboard deserves stating too, because it does not flatter the argument. An H100 moves 3.35 TB/s. An H200 moves 4.8. That is six to nine times an M4 Max, from the least unified and most driver-encumbered stack in the discussion. If bandwidth decides, the messy ecosystem currently wins by close to an order of magnitude.

And the rule has a boundary. Everything above assumed one request at a time. Serve sixty-four at once and each weight read feeds sixty-four tokens; arithmetic intensity rises by that factor, and the work crosses the ridge into compute-bound territory, where FLOPs govern instead. That is why the datacenter built compute-dense accelerators and Apple built a bandwidth-rich laptop. Both are right. They sit on opposite sides of the same roofline.

So the claim here is narrow. Single-user, on-device inference is memory-bound, and the machine that wins it is the one shaped around moving bytes. It stops at the datacenter door.

Do the division before you pick a model

The practical move is to stop asking whether a model runs on a target, and start asking what the worst machine in the fleet can feed it.

GB = 1_000_000_000

# Published memory bandwidth, bytes/second.
TARGETS = {
    "MacBook Pro, M4 Max": 546 * GB,
    "Mac mini, M4":        120 * GB,
    "Raspberry Pi 5":       17 * GB,
}

# Llama 3.1 8B, llama.cpp quantizations, resident weight size.
WEIGHTS = {
    "Q8_0":   8.5 * GB,
    "Q4_K_M": 4.9 * GB,
    "Q3_K_M": 3.7 * GB,
}

def decode_ceiling(weight_bytes: float, bandwidth: float) -> float:
    """Upper bound on tokens/sec: every weight is read once per token."""
    return bandwidth / weight_bytes

Ceilings are the easy half. The decision is which model still clears your latency floor on the worst target, because that machine decides whether the feature ships at all.

# Fraction of theoretical bandwidth a real runtime reaches.
# Measure it per target; 0.5 is a starting guess, not a fact.
EFFICIENCY = 0.5

def largest_model_that_clears(min_tokens_per_second: float) -> dict[str, str | None]:
    """For each target, the biggest quantization that still meets the floor."""
    choices = {}
    for machine, bandwidth in TARGETS.items():
        viable = [
            name for name, size in WEIGHTS.items()
            if decode_ceiling(size, bandwidth) * EFFICIENCY >= min_tokens_per_second
        ]
        choices[machine] = max(viable, key=lambda name: WEIGHTS[name], default=None)
    return choices

print(largest_model_that_clears(min_tokens_per_second=10))
# {'MacBook Pro, M4 Max': 'Q8_0', 'Mac mini, M4': 'Q4_K_M', 'Raspberry Pi 5': None}

The None is the useful output. Before anyone writes a loader or a prompt template, it says that a 10 tok/s experience on an 8B model does not exist on that machine at any quantization — so the real choice is a smaller model, a server round-trip, or a different promise to the user.

Two habits follow.

Size the model against the slowest target you support, never the machine on your desk. Your laptop is the best hardware anyone in the deployment will ever touch, and it will lie to you about everything downstream of it.

Then measure EFFICIENCY per target instead of assuming it. That coefficient is where cores, memory-level parallelism, kernel quality and thermals reassert themselves over the spec sheet, and it is the only part of the calculation a datasheet cannot hand you.

Where this breaks

The ceiling covers steady-state decoding and nothing else. Prompt processing reads the weights once for the whole prompt and is compute-bound from the first token, so a machine can be strong at one and weak at the other. The KV cache grows with context and eats bandwidth this model does not count; at long contexts that stops being a rounding error. Weights that do not fit in memory sit outside the argument entirely. And speculative decoding deliberately breaks the “one weight read per token” premise by checking several draft tokens in a single pass, which is why it works.

Each of those moves the number. None of them move the shape of the argument: the number exists, you can know it before writing code, and the instruction set on the box does not imply it.

Closing

SNK’s engineers were not chasing a faster machine. They asked a narrower question — what does an arcade cabinet spend its time doing — and built silicon that made only that cheap. The 68000 was incidental. The ROM and the sprite hardware were the product.

Apple’s answer is broader and structurally identical. Work out which computations the machine will spend its life on, give each one dedicated silicon, and put them behind one memory pool so nothing is wasted on copying. The clock speed is, again, incidental.

Neither is really a lesson about hardware. It is a lesson about knowing which computation you are paying for. The Neo Geo knew it was paying for pixels per scanline. On-device inference pays for bytes per token.

That one is a division you can do this afternoon, on a datasheet, before you commit to anything.

Most performance work starts long after that division would have answered the question.

Sizing a model for hardware you do not control?

I agree to the Privacy Policy Talk On-Device Inference