ShadowDiff – Shadow testing and SQL diffing for AI-migrated back ends
A new open-source tool called ShadowDiff addresses the 5% risk that prevents enterprises from deploying AI-modernized legacy code to production by providing dual-traffic shadow testing.
Enterprises are increasingly modernizing legacy backends using AI code migration engines, but the remaining 5% of hidden edge cases make production deployment an unacceptable risk. ShadowDiff, a new open-source project, aims to eliminate this risk by providing a dual-traffic shadow testing engine for AI-refactored legacy code.
The Problem
While AI generates approximately 95% of refactored code accurately, subtle discrepancies can cause significant issues. These include serialization differences like Jackson omitting empty lists versus Go serializing null, implicit framework quirks such as legacy servlet filter behaviors, and database state regressions including silent differences in transaction boundaries or N+1 query loops.
The Solution
ShadowDiff functions as a high-performance network proxy that duplicates real HTTP traffic, sending identical payloads to both legacy and AI-refactored services simultaneously. It integrates directly with Envoy Proxy and Istio Service Mesh using request_mirror_policies, cloning traffic asynchronously at line rate with zero application latency penalty.
The tool employs a 3-way differential noise cancellation model that automatically filters out non-deterministic runtime noise like timestamps and UUIDs. It also captures and correlates SQL statements executed by both services, detecting missing writes, transaction leaks, and query explosions.
Automated Testing
Every isolated edge case is analyzed by Google Gemini (gemini-2.0-flash), which generates turnkey integration tests and diagnoses code fixes. ShadowDiff also features an autonomous closed-loop GitHub PR bot that automatically branches repositories, commits synthesized regression tests, and creates detailed pull requests with analysis tables.
Eliminates manual triaging by directly integrating with the GitHub API
Deployment Safety
The tool includes an ephemeral copy-on-write (CoW) database sandboxing system that allows safe shadow mirroring of mutating HTTP methods with zero database pollution. It uses transactional savepoint rollbacks and ephemeral schema sandboxing to ensure isolation. Additionally, ShadowDiff provides a CI/CD 'Shadow Gating' CLI that evaluates traffic quality during pre-release tests, only failing on genuine regressions after noise cancellation.
For production Kubernetes clusters running Istio or native Envoy, the tool can be deployed via included EnvoyFilter to mirror traffic dynamically. The project is licensed under the MIT License and includes comprehensive documentation for architectural specifications and quickstart guides with Docker Compose and GitHub Actions.
Related
