TechShuBot
TechShuBot is a web crawler run for TechShu, a digital agency. It visits the public websites of UK digital-marketing, web-development and software companies to find their business contact details, so TechShu’s sales team can get in touch about working together.
It is separate from SEOAIAuditsBot, the crawler behind SEO AI Audits’ free website audits. The two have different names and each follows its own rules, so you can block one without blocking the other.
How it identifies itself
Every request carries this user-agent:
Mozilla/5.0 (compatible; TechShuBot/1.0; +https://seoaiaudits.com/techshubot)
It runs from the same server as SEOAIAuditsBot, whose address resolves to crawler.seoaiaudits.com.
What it reads
At most 15 public pages per website: the home page and pages such as contact, about, team, privacy and terms. Nothing behind a login, and it never fills in or submits forms.
What it keeps
- The company’s name, registered number and public business contact details: email addresses, phone numbers, the contact-page address and company social media pages.
- The names and job titles of senior people (founders, directors, heads of department) where the website lists them.
It doesn’t keep the text of your pages. It also looks the company up on the public Companies House register.
How it behaves
- It reads your robots.txt first and follows it, including Crawl-delay.
- It makes one request at a time, at least 5 seconds apart.
- If your server answers 429 or 503, it waits as long as Retry-After asks, or stops.
- If your site refuses it (403) or shows a bot check such as a CAPTCHA, it stops and leaves your site alone. It never tries to get round a block.
- Email addresses your site hides from bots are never decoded.
How to block it
Add this to your robots.txt. It checks robots.txt before every visit, so the block applies from its next visit.
User-agent: TechShuBot Disallow: /
Blocking SEOAIAuditsBot, the audit crawler, is separate: see SEOAIAuditsBot.
Remove your details
Email [email protected] with your website or email address. TechShu will delete what it holds and add you to a permanent do-not-contact list, so you won’t be contacted, or have your details collected, again.
How TechShu uses these details, and your rights: TechShuBot privacy notice.