Agentic Web Scraper

Example Use Cases

• 1

  Perform interactive web searches via CLI.

• 2

  Scrape content from URLs programmatically.

• 3

  Fetch and aggregate news from various sources.

Description

This project provides a scraper-only AI agent that integrates LangGraph and OpenRouter to enable interactive command-line web searching, URL scraping, and news fetching. It supports multiple APIs including Tavily for web search, Firecrawl for scraping, and NewsAPI for news retrieval, requiring API keys for operation. The tool is designed for Python 3.12 and offers flexible environment setup options including traditional venv and the faster uv package manager. It benefits users by providing a streamlined, interactive scraping experience with modular API support.