SimpleDocs Search Engine

Example Use Cases

• 1

  Crawl and index documentation websites and generate embeddings for semantic search.

• 2

  Answer user queries by searching across multiple documentation sources using vector similarity.

• 3

  Monitor long-running crawling and indexing jobs with real-time progress updates and a dashboard.

Description

SimpleDocs crawls documentation sites, uses trafilatura to extract clean content, splits large documents into chunks, generates OpenAI embeddings, and stores vectors in Supabase with pgvector for fast semantic search. It includes rate limiting, batching and retry logic for embeddings, and supports multi-source indexing. A WebSocket-backed dashboard provides real-time progress updates for crawling and indexing. The system is intended to help agents quickly find relevant information across documentation sites.