# IMERGEA robots.txt
# SEO and AEO first, globally crawlable, with only technical exclusions.
#
# NOTE: robots.txt is a public file. It must never name private paths
# (data room, admin, login, logs), because doing so hands an attacker a map.
# Those areas are protected by authentication and by returning 404 for
# private paths in .htaccess, which is what actually keeps them out of an
# index. A Disallow rule does not prevent indexing of a URL that is linked
# elsewhere.
#
# AI assistants and answer engines (GPTBot, OAI-SearchBot, ChatGPT-User,
# ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended,
# Applebot-Extended, Bingbot) are deliberately welcome and are covered by the
# group below. They are NOT named individually on purpose: naming a crawler
# creates a separate group that it reads to the exclusion of all others, which
# would silently strip the technical exclusions below from exactly the
# crawlers that matter most. Absence of a rule already means allowed.

User-agent: *
Allow: /
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /_notes/
Disallow: /*.log$
Disallow: /*.jsonl$

# Commercial SEO scrapers: bandwidth cost, no benefit to IMERGEA.
# AhrefsBot is deliberately left allowed while IMERGEA is building its
# backlink profile, because blocking it removes IMERGEA's own ability to
# audit and monitor inbound links. Re-block it once link building is done
# by adding the two commented lines back.

# User-agent: AhrefsBot
# Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

Sitemap: https://imergea.com/sitemap.xml

# LLM guidance file (convention, not a robots.txt directive):
# https://imergea.com/llms.txt