LLMsTxt Architect

Example Use Cases

• 1

  Structured summaries of website contents to guide information retrieval.

• 2

  Generate or update llms.txt files automatically from URLs or existing files.

• 3

  Integrate website content summaries with code tools or MCP servers for enhanced LLM workflows.

Description

LLMsTxt Architect is a Python package and MCP server that builds llms.txt files, a markdown standard listing URLs and their descriptions to guide LLMs in fetching and reading web pages. It supports crawling websites to configurable depths, updating existing llms.txt files, and uses various LLM providers including hosted and local models. The tool offers checkpointing for large sites, URL blacklisting, and customizable prompts and extractors, enabling efficient and structured summarization of website content for downstream LLM tasks.