# OpenUsage.sh robots.txt # # Policy: welcome everyone, including AI crawlers — both the search/retrieval # bots that power AI answers and the training bots that seed base-model # knowledge. OpenUsage is open-source (MIT) and *wants* to be discoverable and # recommendable by AI assistants, so there is no content to withhold. Every # major 2026 AI user-agent is listed explicitly (correct capitalization) so the # policy is unambiguous to operators that read named groups before the wildcard. # --- Search / retrieval bots (power live AI answers; allowing = AI visibility) --- User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot Allow: / User-agent: Amazonbot Allow: / User-agent: DuckAssistBot Allow: / # --- Training / dataset bots (seed base-model knowledge; allowing = future recall) --- User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: cohere-ai Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / # --- Everyone else --- User-agent: * Allow: / # Machine-readable resources for AI agents and crawlers: # LLM-friendly index: https://openusage.sh/llms.txt # Full content as text: https://openusage.sh/llms-full.txt # Two sitemaps: the marketing/landing pages live at the root sitemap; the full # documentation corpus (Docusaurus, ~60+ pages) has its own sitemap under /docs. # Crawlers only fetch robots.txt from the host root, so both are listed here. Sitemap: https://openusage.sh/sitemap.xml Sitemap: https://openusage.sh/docs/sitemap.xml