An LLM inference server for Apple silicon.
3 pull requests merged, 1 open, 4 issues
Pull requests
- oMLX pull request #2161
Merged
Calibrate models larger than RAM: the quantiser streams the model one layer at a time. About 3,000 lines.
- oMLX pull request #2159
Merged
Fix the decoding of MXFP8 block scales in dequantisation.
- oMLX pull request #2140
Merged
Ship the calibration data inside the package.
- oMLX pull request #3504
Open since
Grammar-constrained output that keeps speculative decoding on.
Issues
- oMLX issue #2238
Closed
Mixture-of-experts decoding ran at 34% of the memory roofline, against 94% for a dense model, on an M3 Ultra.
- oMLX issue #3507
Closed
The embeddings endpoint answered 200 with vectors full of nulls when the model returned NaN.
- oMLX issue #3505
Open since
The memory guard sets custom tiers to a soft limit of 0.85 and gives no setting for prefill headroom.
- oMLX issue #3506
Open since
The native 8-bit affine kernel for Qwen3.5 cannot engage during prefill, because of its threshold against the chunk size.