MoEShift - Faster large MoE inference with half the RAM
A new open-source tool called MoEShift allows users to run large AI models locally on Apple Silicon Macs with as little as 10 GB of RAM, achieving high performance.
MoEShift is an alpha release that brings the power of large language models to everyday Mac users. The project is optimized for Apple Silicon, specifically the M-series chips, and can run models with up to 125 billion parameters on a machine with as little as 10 GB of RAM.
This is a significant development for local AI. It allows users to run complex models entirely on their own hardware, without needing a cloud subscription or a dedicated server. The tool is designed to keep the Mac responsive, letting users browse the web, write documents, or code while the AI model works in the background.
How MoEShift works
The core of MoEShift is a Mixture-of-Experts (MoE) architecture. Unlike traditional models where every parameter is used for every single token generated, a MoE model has a massive number of parameters but only uses a small, specialized subset for each task. This makes the model incredibly powerful while being much more memory-efficient.
MoEShift uses a scheduling system called LES (Lookahead Expert Scheduling). This system predicts which parts of the model will be needed next and loads them into memory in advance. It also uses a protected cache to keep frequently used data in RAM, which reduces the need to constantly read from the SSD.
The project is built on top of existing open-source foundations like llama.cpp and ggml. It adds its own Metal execution engine for Apple Silicon, which is optimized for the unique architecture of Apple's chips. This allows the tool to achieve high performance while managing memory usage carefully.
Performance and benchmarks
The tool has been tested on a Mac mini M4 Pro with 24 GB of unified memory. In these tests, MoEShift ran the Qwen 3.8 Flash Next model with 125 billion parameters and an 8,192-token context. It achieved a speed of up to 11 tokens per second, which is a significant improvement over other local MoE inference tools.
A key finding is that MoEShift uses roughly half the peak process memory of Strata, a popular choice for local MoE inference. This means it leaves more room for other applications, preventing the Mac from grinding to a halt under memory pressure. The tool also processes long prompts 2.6 times faster than Strata.
The benchmarks also show that the tool can handle different models and quantizations. For example, it can run the Qwen3.6-35B-A3B model with a UD Q6_K XL quantization on the same hardware. The performance will vary depending on the specific model and the amount of RAM available.
Getting started with MoEShift
To use MoEShift, users need a Mac with Apple Silicon (M-series) and at least 10 GB of RAM. They also need a fast SSD with at least 120 GB of free space for the model files. The installation is straightforward and uses Homebrew, the package manager for macOS.
Once installed, users must download the model files, which can be quite large. For the Qwen 3.8 Flash Next model, the total size is about 76 GB. MoEShift then requires a one-time preparation step to copy the model's specialized parts, called 'experts,' into files it can read directly from the SSD.
After the model is prepared, users can start MoEShift with a simple command. It will create a local server at http://127.0.0.1:8080, which can be connected to any chat or coding application that supports an OpenAI-compatible API. The tool handles one inference request at a time and listens only on localhost.
Next steps and community feedback
The project is currently in its alpha release phase, which means the core runtime is working, but users may encounter 'rough edges.' The developers are actively seeking feedback to shape the next release. They encourage users to report bugs or suggest features through the project's issue tracker.
The team is also asking users to share their experiences running MoEShift on their Macs. They want to know how the model performs on different chips and with different workloads. This information will help them understand where the tool needs to improve and how to make it more useful for a wider range of users.
Related
