Im 17, Built a Swift-native LLM inference engine for Apple Silicon
A new Swift-based language model inference engine called Alpaca has been released for Apple Silicon, showing performance improvements of up to 1.7× over llama.cpp for prefill operations on Apple M5 chips.
Alpaca represents a significant advancement in native language model inference for Apple Silicon, offering a from-scratch Swift implementation with custom Metal kernels. Unlike existing solutions, it doesn't rely on llama.cpp, MLX, Core ML, ONNX, or Python at runtime, providing a more streamlined approach to AI model execution on Apple devices.
The project is currently at version 0.2.0-pre and implements one architecture family (Llama-style decoder with byte-level BPE) along with three weight formats (F16, Q8_0, Q4_0). These implementations have been validated against PyTorch, ensuring compatibility and correctness across different model configurations.
Performance Benchmarks
Testing on an Apple M5 with 16GB RAM showed that Alpaca's prefill operations are 1.3–1.7× faster than llama.cpp when processing 128–4096 tokens using the SmolLM2-135M-Instruct model. The fastest prefill path requires an Apple10-family GPU (M5/A19) and macOS/iOS 26, though it can run on other hardware at roughly 1/3 to 1/2 throughput.
Additional benchmarks revealed that Alpaca's Q4_0 decode at 16-token context reaches 712 tokens per second (compared to llama.cpp's 518), while F16 decode achieves 360 tokens per second (llama.cpp: 319). The engine maintains peak memory usage between 175–194 MiB at a 4400-token context for the 135M-parameter model.
Technical Implementation
Alpaca stands out as an independent implementation with all components—including the tensor engine, CPU reference operations, GGUF parser, BPE tokenizer, Llama forward pass, sampler, and Metal kernels—written from scratch in Swift. Python/PyTorch is only used offline to generate reference fixtures for validation purposes.
The project emphasizes robustness against hostile input, with the GGUF parser treating model files as untrusted. Every length is bounded by the bytes actually present, all arithmetic is overflow-checked, and it survived 20,000 mutation-fuzzed files without a crash during testing.
Usage and Development
Developers can integrate Alpaca using Swift Package Manager with a simple dependency declaration. The library provides an idiomatic API featuring async loading, AsyncSequence token streaming, cancellation support, seeded sampling, per-generation isolated sessions, and a memory-budget check that prevents loading models too large for available memory.
The project includes command-line tools for model fetching, text generation, benchmarking, and profiling, along with a SwiftUI reference application demonstrating model selection, streaming, cancellation, and performance diagnostics. However, the library has only been compiled for the iOS Simulator so far, with no physical iPhone or iPad testing completed.
Related
