Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

Google's Project Zero has conducted a comprehensive comparison between two automated vulnerability detection systems used to find bugs in Chrome's JavaScript engine V8. The study compared Fuzzilli, a traditional generative JavaScript engine fuzzer, with BigSleep, an LLM-based bug finding system. The results show that BigSleep discovered more actionable vulnerabilities than Fuzzilli over a six-month period from April to September 2026.

Production Statistics

When analyzing raw bug reports from Chromium's bugtracker, BigSleep initially reported significantly more issues (441) compared to Fuzzilli (316). However, when filtering specifically for security vulnerabilities, the numbers shifted dramatically with Fuzzilli reporting 178 vulnerabilities to BigSleep's 95. After accounting for duplicates and unactionable reports, the final comparison showed a near tie with BigSleep having a slight advantage at 45 actionable vulnerabilities to Fuzzilli's 37.

The researchers noted that while BigSleep initially found many configuration and integration issues in V8, these had less real-world security impact. The comparison focused on 'actionable' bugs - those not marked as duplicates or 'working as intended' (WAI) and that received at least one fix commit referencing them.

Overlap Analysis

A key question in the study was how much overlap exists between the two approaches in terms of vulnerabilities discovered. Through bugtracker analysis and deduplication efforts, researchers found that approximately 25% of actionable vulnerabilities were discovered by both systems, with a slight majority of these found first by Fuzzilli and later by BigSleep.

Complementary Approaches

Despite the overlap, the study concluded that fuzzing and LLM-based approaches are still complementary methods for vulnerability detection. The researchers performed additional experiments to test theoretical overlap in vulnerabilities reported since 2023, estimating overlap at roughly 60%. This suggests that each method discovers unique vulnerabilities that the other might miss.

The researchers emphasized that while it's extremely promising that LLMs have become able to find so many vulnerabilities so quickly, they don't believe these approaches replace traditional fuzzing. Instead, they recommend that vendors perform both fuzzing and agentic analysis of their code to maximize the number of vulnerabilities that get reported before they impact users.

The study focused specifically on V8 as a target, a high-performance JavaScript and WebAssembly engine with a sophisticated testing setup. The researchers noted that the complicated nature of bugs in these engines makes for an interesting 'playground' to compare the two approaches' ability to find deep and complex bugs in well-tested software.