Sugikit crawler
Last updated
What it is
The Sugikit crawler is used by sugikit check --site. The Sugikit CLI is run by a site owner on their own machine or in CI. It checks pages of a site for polish, search and AI answer issues.
User agent
Mozilla/5.0 (compatible; sugikit/<version>; +https://sugikit.com/bot)
How it behaves
It follows robots.txt rules for User-agent: sugikit and User-agent: *, including Crawl-delay.
It makes at most 3 requests at a time and pauses between requests. It slows down when a server answers 429 or 503.
How to block it
Add these lines to your robots.txt:
User-agent: sugikit
Disallow: /Contact
Write to hello@sugikit.com.