AgentDataBot

AgentDataBot is the web crawler used by AgentData to build our company intelligence database. It visits publicly accessible web pages to detect technology stacks, extract company information, and identify team members.

Crawl behavior

• Identifies as: AgentDataBot/1.0 (+https://agentdata.run/bot)
• Respects robots.txt directives
• Crawls at a moderate rate (max 1 request per second per domain)
• Only visits publicly accessible pages — never attempts to bypass login or authentication
• Fetches HTML only — does not download images, videos, or large files

Opt out

To block AgentDataBot, add to your robots.txt:

User-agent: AgentDataBot
Disallow: /

To remove existing data about your company, visit our data removal page.

Contact

Questions about our crawler? Email support@agentdata.run.