Recently, I’ve been coremaxxing on the M5 neural accelerators by running local MLX-community models. But being an engineer, I can’t just enjoy things without trying to take them apart, so I decided to at least run some benchmarks.
First, I ran some benchmarks to see if MLX-conversion actually matters, and how it compares to good old GGUF’s. Benchmarks online report 15-40% average speed improvement – that’s a bold statement, let’s see how if it holds.
So, I coded some simple benchmarking scripts to compare:
bench_mlx.py– MLX proper, as Silicon-native as it gets.bench_llamacpp.pywith is own Metal shaders.bench_torch.py– unfair comparison just for fun, to illustrate how bad it can be if you choose your setup poorly.
The MLX script runs mlx-community/Qwen3-4B-Instruct-2507-4bit, which is converted specifically for MLX. The other two are running unsloth/Qwen3-4B-Instruct-2507-Q4_K_M.gguf.
The results aren’t particularly surprising:
| Metric | MLX | llama.cpp | PyTorch |
|---|---|---|---|
| Decode, mean tok/s | 184.7 | 144.8 | 36.9 |
| Time to first token, short prompts, mean s | 0.077 | 0.037 | 0.036 |
| Time to first token, long prompt, s | 0.109 | 0.065 | 0.061 |
| Model load, s | 1.2 | 1.3 | 20.4 |
Looks like MLX wins over llama.cpp by about 25% on token generation, but loses some on the TTFT (whole 1/20 second).
Obviously, PyTorch benchmark performs very poorly. It converts 4-bit weights into BF16 in order to run, so it’s not really a fair comparison – I don’t know why I included it here, probably not the best idea.
But this doesn’t show the full picture: MLX model deviates from its GGUF counterpart: the latter generated 18% more tokens, so the speedup comes with some costs.
What’s So Special About MLX?
One of the factors is that MLX-converted models stores less data: 4.5 bits/weight against 4.85/weight on Q4_K_M quantization. That means less data being moved, so token throughput is higher. But also means slightly lower precision.
The rest of the difference seems to be coming from how MLX and llama.cpp work with Metal – these two are entirely different implementations, so MLX seems to be better optimized for the M5 neural accelerator.
There are some claims about llama.cpp essentially translating CUDA-style compute patterns into Metal shaders, but it doesn’t seem to be true – judging by the source code, llama.cpp is as Metal-native as it gets.
Source code of my benchmarking scripts is available on GitHub, so you’re free to see for yourself, and judge me if I made a mistake that turns my DIY-benchmarking results upside down 🙂