Open Crawler MCP Server

Example Use Cases

• 1

  Extract clean text or structured content from web pages.

• 2

  Respect robots.txt rules and rate limits during crawling.

• 3

  Multiple output formats including markdown, XML, and JSON.

Description

Open Crawler MCP Server enables extraction of web page content in text, markdown, XML, or JSON formats with support for CSS selectors. It ensures compliance with robots.txt, enforces rate limiting and page size limits, and provides structured content extraction including headings, links, images, and lists. The server offers comprehensive error handling for various failure scenarios, making it reliable for automated web crawling tasks.