Crawler NestJS MCP Server

Example Use Cases

• 1

  Check if a webpage is already indexed in the database.

• 2

  Crawl webpages and retrieve structured data via HTTP.

• 3

  Store and retrieve full webpage content for summarization or further processing.

Description

This MCP server provides a bridge for external LLMs and agents to interact with a modular web crawler and REST API built with NestJS. It supports crawling webpages, managing webpage data in a PostgreSQL database, and retrieving full HTML content, all accessible through HTTP endpoints. The server facilitates automated workflows for checking, crawling, storing, and summarizing web content, enhancing agent capabilities without direct database access.