SmartBoySpider
SmartBoySpider is the crawler behind SmartBoy Search, a small search engine for AI agents and people. It visits a hand-picked list of reference, documentation, science and news sites, not the whole web.
How it behaves
- It follows
robots.txt, includingCrawl-delay. - It fetches one page at a time per site, with at least a second or two between requests, and slows down on errors or
429. - It reads sitemaps first and re-checks pages with
If-Modified-SinceandETag, so unchanged pages cost almost nothing. - It skips pages behind paywalls or sign-in walls and never tries to get around them.
- Bulk sources that publish dumps (like Wikipedia) are loaded from the dumps instead of being crawled.
User agent
SmartBoySpider/0.1 (+https://search.nauqworks.com/bot)
Opting out
Add this to your robots.txt and SmartBoySpider will stop within a day:
User-agent: SmartBoySpider
Disallow: /