SpiderTaz Search: More Control Over Crawling
New crawler controls are being prepared so permitted domains can be added, paused, removed and crawled in bounded runs as the independent search index develops.
SpiderTaz Search is continuing to develop its independently maintained index. The latest crawler work focuses on choosing permitted domains and managing when they are crawled.
The controls being prepared include adding and removing domains, starting a bounded crawl, pausing or resuming work, and reviewing queue counts and recent results. Respecting source access rules, robots controls and request limits remains part of the process.
These controls are a development update, not an announcement that unrestricted web crawling is running or that every requested page will appear in search. Crawl controls and newly indexed results still require deployment and verification.
