OUR CRAWLER
PipelineSimpleBot
Pipeline Simple helps companies see how they show up in search and in AI answers. PipelineSimpleBot reads public web pages for those reports: a customer’s own website, sites a customer asks us to research, and pages that AI answers cite about a customer’s brand.
How to recognize it
Every request names the bot in its user agent and links to this page. It only reads public pages and never logs in, submits forms or clicks ads.
Mozilla/5.0 (compatible; PipelineSimpleBot/1.0; +https://pipelinesimple.com/bot)
Pages that AI answers cite
- It reads the site’s robots.txt first and follows the group for PipelineSimpleBot, or the group for all bots when there is none. It checks again before following each redirect.
- If robots.txt cannot be read because the server errors or does not answer, it fetches nothing else from the site. A missing robots.txt allows everything, as the standard says.
- It skips sites whose terms forbid automated access, such as LinkedIn, Facebook, Instagram, X, TikTok, Glassdoor, Indeed, Yelp, Amazon and Google.
- It waits about a second between requests to the same site and keeps robots.txt for at most an hour, so changes apply within an hour.
Customer sites
When a customer audits their own site, the bot follows robots.txt to choose which pages to crawl. When a customer gives us one specific link, such as their homepage or a company they research, it reads that page once, when they ask, the way a browser would.
How to block it
Add this to your robots.txt to keep it off your whole site, or replace the slash with the paths you want to keep private.
User-agent: PipelineSimpleBot Disallow: /
Questions
Email support@pipelinesimple.com with the address of your site and we will look into any visit you are unsure about.