Collie

Collie is a web scraping tool that automatically visits URLs and extracts content, media, and files to build a searchable index. It handles multiple file types including PDFs, images, videos, audio, HTML, and text documents. Once scraped, all assets are stored in Collie's search index, which you can query to find specific information across your collected content. This makes it useful for building knowledge bases, conducting research, or creating private search functionality across websites and documents you own or have permission to access. The tool is available on a freemium model, so you can start indexing content without upfront cost.