Web Crawler MCP Server

Example Use Cases

• 1

  Fetch and normalize HTML content from a URL with configurable timeouts and delays to avoid overloading targets.

• 2

  Crawl a site to a limited depth to discover internal links and build a small index or link graph for downstream processing.

• 3

  Integrate a web crawler as an MCP service for distributed workflows, automated data collection, or periodic monitoring of web pages.

Description

This repository contains a Node.js-based web crawler packaged to run as an MCP server. It exposes a 'crawl' tool that accepts a URL and depth, and supports configurable behavior via environment variables (crawl links, max depth, request delay, timeout, max concurrent requests). The project is intended to be integrated into an MCP deployment so agents can request on-demand crawls as part of automated workflows. Benefits include easy MCP integration, simple configuration via .env or MCP env settings, and control over crawl depth and concurrency to limit scope and load.