Back to home

CareerCornerBot

Last updated: October 24, 2026

CareerCornerBot is CareerCorner's web crawler. It reads public job listings on employers' careers sites so job seekers can find open roles in one place. This page is for site owners: how to recognise it, how it behaves, and how to limit or stop it.

How to recognise it

Crawling requests carry this user agent:

CareerCornerBot/1.0 (+https://careercorner.ai/bot)

Crawling requests are signed with HTTP Message Signatures (Web Bot Auth). The signature agent is https://careercorner.ai, and our public keys are at https://careercorner.ai/.well-known/http-message-signatures-directory.

What it reads

Public pages and feeds that list open jobs: job titles, locations, descriptions and links to apply. It does not sign in, submit forms, solve challenges, or collect information about your employees.

How it behaves

  • It reads robots.txt before anything else and follows the rules for User-agent: CareerCornerBot, or for User-agent: * when there is no CareerCornerBot group.
  • It sends at most one request per second to a host, and waits longer when your robots.txt sets a Crawl-delay.
  • When your site answers 403 or 429 or shows a challenge page, it stops reading your site for the rest of the day. After repeated refusals it waits days or weeks before it tries again.
  • Most reading happens once a day, at night in the United States.

Single-page fetches

Some requests fetch one known page instead of crawling: a job posting a CareerCorner user pasted or opened, a check that a posting is still open, or a company's logo. They carry the user agent CareerCornerFetch/1.0 (+https://careercorner.ai/bot), follow no links, and do not read robots.txt.

How to limit or stop it

Add this to your robots.txt to stop the crawler everywhere on your site:

User-agent: CareerCornerBot
Disallow: /

To slow it down, add a Crawl-delay line to the same group. Changes take effect at the next daily read.

The robots.txt rule does not cover single-page fetches. To stop everything, e-mail support@careercorner.ai from an address at your company's domain. We stop reading your careers site within two business days and confirm by e-mail.

Contact

Questions or problems: support@careercorner.ai.