Web Crawler MCP Server

Example Use Cases

• 1

  Fetch and extract readable main text from arbitrary public web pages to use as context when answering queries.

• 2

  Render JavaScript-heavy sites and bypass basic anti-bot protections using Puppeteer with a stealth plugin.

• 3

  Normalized, whitespace-cleaned content suitable for LLM input and downstream parsing or summarization.

Description

This server launches a real browser (Puppeteer with stealth) to load and render public web pages, extracts the main article/content using Cheerio, and returns whitespace-normalized plain text optimized for LLM consumption. It's designed to integrate as an MCP-compatible tool for clients like Claude Desktop and Cursor, exposing a single web-crawler tool that accepts a URL and returns cleaned text. The project emphasizes compatibility and ease of integration and includes development scripts for building and debugging.