Webcrawl, an open-source web crawling API and MCP server for agents

Developed by brainfish-ai, the tool allows users to set up web crawling capabilities with a single command: 'npx skills add brainfish-ai/webcrawl-anything'. This installs eight Webcrawl Agent Skills into popular coding assistants like Claude Code, Cursor, Codex, Gemini, and Windsurf. The tool works without API keys by using a built-in TLS-impersonated fetcher and document parsers.

Users can target specific skills by name, such as 'webcrawl-deal-watch' for Black Friday price monitoring or 'webcrawl-protected-sites' for accessing sites with anti-bot measures. The tool also includes an MCP server registration option that enables agents to call crawl, search, and monitor tools directly.

Comprehensive Crawling Capabilities

Webcrawl-anything can process various content types including websites, single-page applications, PDFs, YouTube videos, GitHub repositories, ArXiv papers, product pages, and CVE security reports. It offers both CLI usage for direct commands and integration with AI agents through its skill system.

The tool's architecture includes a multi-tier fetcher that handles static pages without launching Chrome, while JavaScript pages use a CDP-driven local Chrome session with timeout and fallback options. Its extractors support structured data from various verticals including shopping, real estate, and security vulnerabilities.

Local and Cloud Options

The tool runs locally with zero API keys using its built-in engine, but users can upgrade to the webcrawl.app cloud service with one command when managed browsers or faster global cache hits are needed. The local engine and cloud service use the exact same TypeScript codebase, ensuring consistent functionality across both platforms.

The MCP server can be configured with a compact schema profile to minimize token overhead in AI agents' context windows while maintaining full server-side validation. The tool's architecture is designed as a shared engine flywheel, where improvements to target plugins, vertical extractors, or document parsers benefit both local and cloud users immediately.

Community and Development

The project welcomes contributions for new target plugins or extractors, with a three-step guide available in the documentation. Contributors are credited in release notes, and users are encouraged to share their builds through community discussions. The tool is released under an MIT license, allowing for broad adoption and modification.

The development team has structured the project with separate directories for the core engine, CLI interface, MCP server, and client SDK, enabling modular improvements. Each component contributes to a unified system that powers both local command-line usage and cloud API services.