Why I’m building my hacking agent out of tiny, boring tools
Security researchers are finding that AI hacking agents perform better when built around small, deterministic tools rather than relying on large language models for all tasks.
In 1978, computer scientist Doug McIlroy articulated a philosophy that would become the foundation of Unix design: "make each program do one thing well" and "expect the output of every program to become the input to another, as yet unknown, program." Nearly 50 years later, this approach is finding new relevance in AI development, particularly for security applications.
Modern AI agents, when used for security testing, often produce unreliable results. Studies show that models like GPT-4 change their vulnerability assessments 17% of the time when code is merely cosmetically altered. Detectify reports that approximately 80% of findings from AI-driven static analysis tools turn out to be false positives.
Small Tools Beat Big Models
Language models function as "confidence machines" that provide authoritative-sounding answers regardless of accuracy. For deterministic tasks like security testing, specialized tools outperform AI. For example, executing an out-of-band payload and monitoring for a DNS callback should be handled by a dedicated tool, while deciding where to inject the payload might require AI's more dynamic intelligence.
Let the tools do the tool things and let the model do the thinking things.
Efficiency Through Design
Using AI for tasks that deterministic tools can perform wastes computational resources. Anthropic, the AI company behind Claude, discovered that optimizing tools consumed more effort than prompt engineering when building their SWE-bench agent. They found that concise tool responses used about 72 tokens compared to 206 for verbose versions, demonstrating efficiency gains from minimal output.
The Model Context Protocol (MCP) provides a standardized way to describe tools to AI models, eliminating the need to repeatedly explain tool functionality in prompts. This reduces context window usage from potentially 150,000 tokens to around 2,000 in some implementations.
Building Effective Tools
Effective AI agent tools follow specific principles: they should avoid wrapping every CLI flag instead focusing on complete workflows; use proper typing for inputs to prevent garbage data; maintain minimal output without unnecessary formatting; and use namespacing to help distinguish between multiple tools. Crucially, tools should maintain their own logs of actions performed, separate from the AI's analysis.
A practical example involves creating a simple MCP server in Python that wraps the subfinder tool for subdomain enumeration. This approach allows an AI agent to efficiently use the tool without repeatedly searching for basic techniques, significantly reducing token usage and improving reliability.
Related
