A test page designed for validating web crawlers, scrapers, and content extraction algorithms. Built with modern web standards including semantic HTML5, JSON-LD structured data, and proper meta tags.
- Semantic HTML5 markup
- JSON-LD structured data
- OpenGraph & Twitter meta tags
- robots.txt & sitemap.xml
Visit crawltest.com to test your web crawler or scraper. The page is designed to be crawled once and provides a reliable baseline for testing parsing capabilities.
This is a site built with React and Remix. To run locally:
npm install
npm run build
npm run previewMIT License - see LICENSE file for details.
Feel free to submit issues, feature requests, or pull requests. This project is open source and welcomes contributions from the community.