Hyper64 – Fast AVX-512 Base64 engine in Rust (10-25 GiB/s)
A new Rust library called hyper64 has been released, claiming to be up to 340 percent faster than the current standard.
The new library, hyper64, is designed to solve performance bottlenecks in the standard base64 encoding process. It is an industrial-grade engine built from the hardware up to handle real-world data processing.
Hyper64 achieves sustained throughput exceeding 10 GiB/s while guaranteeing zero heap allocations. It maintains 100 percent API drop-in compatibility with the base64 0.22 crate.
Hardware Acceleration
The engine leverages multi-tier SIMD vectorization to maximize speed across different hardware architectures. It supports AVX-512BW, AVX2, SSSE3, AArch64 NEON, and WebAssembly SIMD128.
This wide-vector support allows the library to process data blocks in parallel, significantly reducing the time required for encoding and decoding operations.
Zero-Allocation Decoding
Hyper64 includes a formally proven in-place zero-copy decoding feature. This allows data to be decoded directly inside an existing mutable slice, such as a packet buffer or a pre-allocated arena.
The library also features zero-double-write memory initialization. It writes directly into spare vector capacity before calling set_len, eliminating zero-filling overhead without reading uninitialized memory.
Security Features
The engine includes a constant-time cryptographic mode to eliminate cache-timing vulnerabilities. This is essential for handling private keys, cryptographic nonces, or tokens securely.
It employs explicit core::ptr::read_unaligned and _mm256_loadu_si256 instructions to avoid strict-aliasing violations. The library also includes a hardware CPUID gating mechanism to validate that the OS kernel has enabled YMM register saving.
Formal Verification
Hyper64 is built under a strict zero-undefined-behavior mandate. The developers have provided a formal proof of in-place non-clobbering, ensuring that memory corruption is mathematically impossible.
The library is audited on every commit by an exhaustive verification matrix. This includes Miri for continuous memory model validation, differential oracle testing, and libFuzzer with AddressSanitizer.
Real-World Use Cases
The library supports asynchronous streaming with Tokio, allowing developers to stream multi-gigabyte files or network sockets without loading the entire payload into RAM.
It also includes features for parsing PEM certificates and MIME whitespace directly in-place. This allows for the decoding of real-world certificates, SSH keys, and emails without memory copies.
Installation and Availability
Developers can add hyper64 to their Cargo.toml file by using the version "0.2". The library is licensed under either the Apache License 2.0 or the MIT license.
The project is hosted on GitHub and validated on multiple architectures, including Linux x86_64, macOS Apple Silicon, Windows MSVC, and Big-Endian s390x.
Related
