AI penetration testing cannot replace independent cybersecurity assurance - Federal News Network

Defense contractors face renewed scrutiny over protecting controlled unclassified information (CUI) as the government re-evaluates measurement methods. The suspension of Cybersecurity Maturity Model Certification (CMMC) Phase 2 third-party assessment requirements has sparked discussion about whether AI-based penetration testing could offer a faster, cheaper alternative to current independent assessments.

While continuous automated testing seems attractive compared to periodic reviews every three years, the two approaches answer fundamentally different questions. Substituting one for another would merely replace one flawed security measure with another, according to cybersecurity experts.

Testing limitations

Automated penetration testing can help address timing gaps by repeatedly checking assets and attempting exploitation within allowed bounds. For some technical controls, evidence of failed attacks may be more convincing than policy statements about controls being in place.

However, penetration testing alone cannot provide complete analysis of CUI protection. Many breaches result from stolen credentials, compromised third parties, or misuse of legitimate access—issues that network perimeter testing won't detect.

A tool that's probing the network perimeter is simply not going to see the sensitive data being copied to an unauthorized collaboration site, sent via a normal email or left on an unmanaged device.

Independence concerns

Independent assessments examine documented controls, interview personnel, and review implementation evidence, though they can only capture a moment in time. Their value lies in evaluating security governance, people practices, external service providers, incident response, and actual CUI handling—areas that cannot be reduced to automated attacks.

A critical distinction is independence. Contractors with sophisticated test platforms still control which systems are tested, how the platform is configured, and what results are reported. Greater automation doesn't guarantee independent evidence, as demonstrated by cases where self-reported cybersecurity scores conflicted with government evaluations.

The gap between self-reported scores and government assessments has been a factor in False Claims Act enforcement, as seen in a case where a contractor reported perfect scores that contradicted later evaluations by the Defense Industrial Base Cybersecurity Assessment Center.

Layered approach

Experts recommend a layered assurance strategy: first, independently verify required controls and CUI boundaries; second, continuously test exposed technical controls; third, conduct detection and response exercises. These layers should inform each other, with findings modifying assessment scope and remediation driving retesting.

Cost considerations are significant. While AI tools may seem inexpensive per call, sustained testing campaigns require substantial work including reconnaissance, exploit attempts, evidence analysis, reporting, and validation. Cape Endeavors' analysis of 500 hosts and 20 web apps showed a single test cycle might cost around $395 in compute costs, while high-frequency continuous testing could reach $4,200 monthly—excluding vendor margins, infrastructure, human review, and remediation costs.

For small and midsize defense contractors, continuous testing would transform one-time assessment costs into perpetual operating expenses. There should be no assumption of a universal Level 2 AI testing mandate, as policy must consider ongoing costs to the contractor base.

Contractors should focus on maintaining accurate CUI inventories, updated system security plans, minimized exposure, and documented changes and remediation. Better scoping reduces both security risks and testing requirements. For policymakers, the principle is clear: define assurance outcomes first, then select appropriate tools.

AI can accelerate technical validation but cannot independently prove safeguards are properly implemented, CUI is correctly located, or organizations will handle compromise appropriately. The solution isn't choosing between human assessment and machine testing, but recognizing neither provides complete security answers alone.