# SOPHIA XT - Robots.txt
# Every search and AI crawler is welcome on the same terms.
#
# One thing about the format is worth knowing before editing this file: a
# crawler obeys exactly one group, the most specific one that names it, and
# ignores every other group including the wildcard. This file used to give
# Googlebot, Bingbot, GPTBot and twenty-odd others their own group containing
# nothing but "Allow: /", which meant the sign-in and admin rules below applied
# to every crawler except the ones anybody cares about. The named agents are now
# listed against the same rules rather than against a group of their own.

# One group, no blank lines inside it.
#
# A blank line ends a record in robots.txt, so the empty line that used to sit
# between "Allow: /" and the disallow list below split this into a group that
# allowed everything and a set of orphan rules belonging to no user-agent. It
# happened to fail open, which is why nothing broke and nobody noticed, but a
# strict parser is entitled to make its own decision about a malformed file and
# two assistants have now reported being unable to read the site. Comments do
# not end a record; blank lines do. Keep them out of this block.
User-agent: *
# Deny before allow, deliberately.
#
# The specific rules used to sit under "Allow: /", which is correct for Google,
# because it resolves conflicts by longest match. It is not correct everywhere:
# the parser in the Python standard library, which a fair number of fetchers
# use, returns the FIRST rule that matches, and "Allow: /" matches everything.
# In that reading the sign-in and admin paths were crawlable. Ordering the
# narrow rules first is right under both interpretations and costs nothing.
Disallow: /admin/
Disallow: /dashboard
Disallow: /auth
Disallow: /login
Disallow: /signup
Disallow: /tickets
Disallow: /sophiabuilds/invite
# Everything else, which is the entire website.
Allow: /

# Sitemaps. The first is an index of the other three and is the only one worth
# submitting; the children are listed too because some crawlers read robots.txt
# and never fetch the index.
# The engineering feed, for anyone who wants to subscribe rather than visit.
Sitemap: https://sophiaxt.com/rss.xml
Sitemap: https://sophiaxt.com/sitemap.xml
Sitemap: https://sophiaxt.com/sitemap-pages.xml
Sitemap: https://sophiaxt.com/sitemap-research.xml
Sitemap: https://sophiaxt.com/sitemap-papers.xml

# Named crawlers, welcome, and held to the same rules. Repeating the list is
# what keeps the group above from being overridden into permissiveness.
#
# Search engines
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Google-InspectionTool
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Bingbot
User-agent: msnbot
User-agent: Slurp
User-agent: DuckDuckBot
User-agent: Baiduspider
User-agent: YandexBot
User-agent: Applebot
# AI crawlers and assistants
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: ClaudeBot
# Claude-User is the fetcher that runs when a person asks Claude to open a
# page, and Claude-SearchBot is the one behind its search. Neither was listed,
# which left both on the wildcard group. That group allows them, so nothing was
# ever actually blocked, but an assistant that could not read the site reported
# it as a robots refusal and the ambiguity is not worth keeping.
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: xAI-Grok
User-agent: GrokBot
User-agent: FacebookBot
User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: PerplexityBot
User-agent: cohere-ai
User-agent: YouBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Diffbot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Bingbot-Extended
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Google-CloudVertexBot
# Social preview and archival
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: LinkedInBot
User-agent: Pinterest
User-agent: ia_archiver
User-agent: archive.org_bot
Disallow: /admin/
Disallow: /dashboard
Disallow: /auth
Disallow: /login
Disallow: /signup
Disallow: /tickets
Disallow: /sophiabuilds/invite
Allow: /

# === Disabled crawlers (if any) ===
# None. SOPHIA XT welcomes all AI and search crawlers.

# === What this site is ===
# SOPHIA XT (also written SophiaXT) is a software company in St. Louis,
# Missouri, founded by Thomas Garren in 2024, and the engineering arm of
# 3 Point Forward. It builds the operational systems companies run on: ERP and
# enterprise business suites, point of sale and inventory, dispatch and field
# service, the integrations between them, and AI grounded in the customer's own
# records. It publishes the research underneath its own tooling.
#
# Pages worth reading first, all of them live:
# - /llms.txt - structured company summary for LLM systems
# - /about - the company, and its relationship to 3 Point Forward
# - /what-is-sophia-xt - what the name means and what the company is not
# - /work - systems built and deployed, with the verification state of each
# - /built - the engineering index
# - /services/ai-implementation-assurance - workflow-first implementation assurance
# - /services/automation-systems - automation systems, API wrappers, routing, analytics
# - /services/custom-model-design-training - model design, fine-tuning, evaluation
# - /solutions - what the company takes on
# - /models/cassandra - Cassandra model-family architecture, demo and benchmarks
# - /research - research index
# - /research/anchor-token-masking - anchor-token masking, DOI 10.5281/zenodo.20496783
# - /whitepaper - technical documentation
# - /atlas - every page on the site
# - /contact - direct contact and booking