Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
github.com/opendatalab/MinerU
What each tool is actually for, described plainly. No ratings, no scores, no invented benchmarks.
Web scraping and data: Getting text out of the web and out of documents, in the shape a model can read.
Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
github.com/opendatalab/MinerU
#1 PDF Application on GitHub that lets you edit PDFs on any device anywhere
github.com/Stirling-Tools/Stirling-PDF
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
github.com/unclecode/crawl4ai
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
github.com/apify/crawlee
The context API to search, scrape, and interact with the web at scale. 🔥
github.com/firecrawl/firecrawl
Convert PDF to markdown + JSON quickly with high accuracy
github.com/datalab-to/marker
Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
github.com/microsoft/playwright
A pure-python PDF library capable of splitting, merging, cropping, and transforming the pages of PDF files
github.com/py-pdf/pypdf
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
github.com/jina-ai/reader
Scrapy, a fast high-level web crawling & scraping framework for Python.
github.com/scrapy/scrapy
🔥 Open Source Browser API for AI Agents & Apps. Steel Browser is a batteries-included browser sandbox that lets you automate the web without worrying about infrastructure.
github.com/steel-dev/steel-browser
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
github.com/Unstructured-IO/unstructured