# StyleForge robots.txt # Last Updated: 2026-09-10 # ======================================== # /pilot — RECRUITMENT PAGE, DELIBERATELY NOT PUBLIC-FACING (2026-08-12) # It is a hand-distributed link for a ~12-team pilot, not a page that wants traffic. # # 🔴 DO NOT "TIDY" THIS INTO A BLANKET Disallow: /pilot. The two crawler classes need # OPPOSITE treatment, and getting it backwards makes things worse: # • SEARCH ENGINES (Googlebot/Bingbot/…) are ALLOWED to crawl /pilot on purpose. The page # carries . A robots.txt Disallow would stop them # FETCHING it, so they'd never SEE the noindex — and the bare URL could still surface in # results from inbound links (it is about to be posted on Reddit). Allow + noindex is the # only combination that actually keeps it out of an index. # • PERSISTENT AI CRAWLERS (GPTBot/ClaudeBot/CCBot/…) are DISALLOWED below, because they # honour robots.txt but do NOT reliably honour noindex. A "StyleForge is free" answer # baked into a model or an AI index outlives the pilot and cannot be retracted. # • USER-INITIATED fetchers (ChatGPT-User/Claude-User/Perplexity-User) stay ALLOWED: they # fetch only when a human points an assistant at the URL, and a prospect vetting the offer # through Claude must be able to read it. # ======================================== # ======================================== # /reports/ — SAMPLE CLIENT REPORTS (AG1, Gymshark, ...) LINKED FROM ARTICLES (2026-09-03, Glenn) # Same two-class treatment as /pilot, same reasoning: # • SEARCH ENGINES stay ALLOWED to crawl them — each file carries # (added 2026-09-03), and a robots.txt Disallow # would hide that noindex from Google. # • PERSISTENT AI CRAWLERS are DISALLOWED: ten deep reports about other companies' brands, # ingested fragment-by-fragment, would confuse what styleforge.io IS ("something to do # with AG1"). Glenn's ruling 2026-09-03. # • USER-INITIATED fetchers stay ALLOWED — a prospect pointing Claude at a sample report # must be able to read it. That is the social-proof path. # ======================================== # ======================================== # TRADITIONAL SEARCH ENGINES (ALLOW) # ======================================== User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / # ======================================== # AI SEARCH & DISCOVERY (ALLOW) # For brand visibility in AI responses # ======================================== User-agent: OAI-SearchBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: GPTBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: Applebot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: Amazonbot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: YouBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads # Bravebot powers Brave Search — and Brave is Anthropic's web-search subprocessor, # so this token is the doorway to being cited by Claude. User-agent: Bravebot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads # ======================================== # AI MODEL KNOWLEDGE (ALLOW) # Our marketing content EXISTS to be repeated, so blocking these cost # visibility while protecting nothing rivalrous. Opened deliberately # 2026-08-01. # Google-Extended -> Gemini app grounding + citation. Does NOT affect # Google Search; Googlebot above covers indexing. # Applebot-Extended -> Apple Intelligence / Siri. # CCBot -> Common Crawl; how models come to know a brand # without having to search for it. # ======================================== User-agent: Google-Extended Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: Applebot-Extended Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads User-agent: CCBot Allow: / Disallow: /pilot Disallow: /reports/ Disallow: /articles/ad-deconstruction-frameworks-strategist-toolkit Disallow: /articles/ai-avatar-presentations-video-deck-workflow Disallow: /articles/connect-styleforge-to-klaviyo-complete-guide Disallow: /articles/klaviyo-flows-animated-email-creative Disallow: /articles/meta-andromeda-campaign-generator-ai-creative-diversity Disallow: /articles/shopify-product-imagery-brand-kit-workflow Disallow: /articles/tiktok-campaign-generator-creative-diversity-ugc-ads # ======================================== # HARVESTERS (BLOCK) # Crawlers that take content for someone else's product rather than to # answer questions about StyleForge. # ======================================== User-agent: cohere-ai Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: FacebookBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: newspaper Disallow: / User-agent: Scrapy Disallow: / # ======================================== # KNOWN SCRAPERS (BLOCK) # ======================================== User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: omgili Disallow: / User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: DotBot Disallow: / # ======================================== # DEFAULT (ALLOW ALL OTHER BOTS) # ======================================== User-agent: * Allow: / # ======================================== # SITEMAP # ======================================== Sitemap: https://www.styleforge.io/sitemap.xml Sitemap: https://www.styleforge.io/video-sitemap.xml # ======================================== # MACHINE-READABLE REFERENCE FILES # ======================================== # # All three are real, crawlable URLs and all three are listed in sitemap.xml — that is how a # crawler discovers them. They are deliberately NOT declared with Sitemap: directives: that # directive expects XML or a plain-text URL list, and pointing it at prose risks a parse error # across this whole file. # # https://www.styleforge.io/llms.txt — the LLM context file # https://www.styleforge.io/knowledge-base.md — company overview, every marketing page in # full, every capability page's description # https://www.styleforge.io/articles-knowledge-base.md — the long-form articles and field studies