RTX 3090’s Token Generation Speed Gets Up To A 4.6x Boost Running A 27B LLM Through Just Inference Optimizations, Opening New Doors For Older GPUs - Wccftech

The bottleneck in AI text generation is not the graphics card's processing power, but the time it takes to fetch the model's weights from memory. This process, known as inference, is the main reason why generating text with large language models (LLMs) can be slow.

A developer, known as u/iamMess on Reddit, tackled this problem by creating a custom inference engine. The goal was to speed up the generation of tokens—the basic units of text in an LLM—without changing the hardware or the model itself.

The developer's custom engine uses a technique called speculative decoding. A fast 'drafter' guesses several upcoming tokens, and the main model checks all of them in a single pass, keeping the correct ones and discarding the rest. This allows the system to process a whole sentence at once instead of a single word.

The improvements were achieved in stages. First, the developer tuned the model's built-in multi-token prediction (MTP), which lets it guess its own next few tokens. This lifted the single-user speed to about 114 tokens per second.

Next, the developer added a second drafting method called 'DFlash2,' which proposes seven tokens at a time. They also added a 'lookup' drafter that scans the text already in the conversation for a match to what was just written and guesses that the same words come next.

The 'lookup' drafter was particularly effective for tasks like quoting a document. It could fill extra slots for free, so the engine now checks 16 tokens per step. This change pushed speeds on a 25,000-token document task from 260 to 382 tokens per second.

The developer also used the approach to boost the model's context window to 138,696 tokens by storing the model's working memory in a more compact format. This is a significant increase, allowing the model to process much longer texts.

The optimizations come with some trade-offs. The longer checking step uses up half the request slots and 8,000 tokens of context, so it is optional. The compact memory format also doubles the time needed to read in a document at the start, so it suits cases where you load a document once and ask many questions.

Outside of text-repeating tasks, this mode runs at about half the speed of plain MTP. For everyday chat, the developer reports around 133 tokens per second, which is still a solid improvement over the previous 82 tokens per second.

The developer notes that accuracy on a math benchmark held steady at 96.5 percent. However, since the information has not been independently verified, it is best to treat the results with a grain of salt for now.