My process to get 6-11X speedup running Qwen3.6 on my RTX 4090, full repo.

Lightreason is a local inference lab and interactive studio designed to run Qwen AI models faster on a single RTX 4090 GPU. The project demonstrates significant performance improvements, achieving up to 724.4 tokens per second on certain benchmarks while maintaining output accuracy. The experiments use original BF16 weights for both Qwen2.5-3B-Instruct (3.09B parameters) and the text model of Qwen3.6-35B-A3B (34.66B parameters).

Performance Benchmarks

For the 3B model, Lightreason achieved speeds from 63.0 to 139.6 tokens per second across six prompts, matching all 30 Hugging Face timing runs. It also successfully completed complete Python, JavaScript, and SQL responses in 9 out of 9 test pairs. The separate PARD2 runner reached even higher speeds, from 58.3 to 525.2 tokens per second on a weather response and 63.6 to 724.4 tokens per second on a 128-token Fibonacci slice.

For the larger 35B model, performance increased from 7.93 to 51.08 tokens per second on a capped Python prompt. The 6.44× comparison used identical input across sessions, with output first differing from the Scheduled version at token 116. The packed configuration successfully matched all 20 pairs against its closer fixed eight-position control.

Technical Implementation

Lightreason employs several optimization techniques to achieve these speed improvements. For the smaller model, it fits entirely on the GPU, reusing prepared GPU work and checking multiple tokens simultaneously. For the larger 35B model, whose text weights occupy 64.56 GiB, the system keeps most expert weights in system RAM and only moves the weights Qwen actually uses to the GPU.

The project leverages existing techniques and projects including NVIDIA CUDA Graphs, AMD PARD2, Inco AI DFlash2, and FlashML's FreeToken. Lightreason's implementation includes innovations like LR CUDA graphs, compact key-value caching, and methods for combining BF16 projections and screening vocabulary candidates before scoring.

Studio and Accessibility

Live Studio provides an interactive interface for testing speed and comparing outputs. Users can build and preview web pages, then run the same prompt through both Standard and Lightreason Exact configurations to watch responses stream and compare timings. The project is available on GitHub with detailed setup instructions for Linux systems with an RTX 4090 and CUDA 12.8 compatibility.

The GitHub repository includes source code, pinned download manifests, and benchmark evidence. It does not include checkpoint or adapter weights, which must be downloaded separately. For users interested only in the 3B Studio, a sparse clone option is available to skip the 35B sources and benchmark archives.