Documentation Crawler MCP

Example Use Cases

• 1

  Search and retrieve relevant sections from crawled website documentation using semantic similarity.

• 2

  Load and browse heading-structured documentation from local Markdown files for context-aware responses.

• 3

  A local, cached embedding backend to reduce startup time and avoid repeated remote embedding calls.

Description

This project provides a web crawler that turns site content into consolidated Markdown files and an MCP server that loads, chunks, and embeds those files for semantic search. It includes caching of chunks and embeddings to speed subsequent startups and automatic cache invalidation when source files change. The MCP server exposes tools (list documents, get headings, semantic search) suitable for integration with agents like Cursor/Claude via stdio. Benefits include local, privacy-friendly search over documentation and easier agent access to site content without external services.