Skip to content
StudioCMS Docs
Main Site

integrations/robots/schema

Type Aliases

Section titled “Type Aliases”

PolicyOptions

Section titled “PolicyOptions”
type PolicyOptions = {
  allow?: string | string[];
  cleanParam?: string | string[];
  crawlDelay?: number;
  disallow?: string | string[];
  userAgent?:   | "*"
     | "360Spider"
     | "360Spider-Image"
     | "360Spider-Video"
     | "HaoSouSpider"
     | "AdsBot-Google"
     | "AdsBot-Google-Mobile"
     | "AdsBot-Google-Mobile-Apps"
     | "Googlebot"
     | "Googlebot-Image"
     | "Googlebot-Mobile"
     | "Googlebot-News"
     | "Googlebot-Video"
     | "Mediapartners-Google"
     | "adidxbot"
     | "bingbot"
     | "BingPreview"
     | "MicrosoftPreview"
     | "msnbot"
     | "msnbot-media"
     | "Applebot"
     | "AppleNewsBot"
     | "Baiduspider"
     | "Baiduspider-image"
     | "Baiduspider-mobile"
     | "Baiduspider-news"
     | "Baiduspider-video"
     | "coccoc"
     | "coccocbot-image"
     | "coccocbot-web"
     | "DuckDuckBot"
     | "DuckDuckGo-Favicons-Bot"
     | "facebookcatalog"
     | "facebookexternalhit"
     | "Facebot"
     | "gooblog"
     | "ichiro"
     | "Sogou blog"
     | "Sogou inst spider"
     | "Sogou News Spider"
     | "Sogou Orion spider"
     | "Sogou spider2"
     | "Sogou web spider"
     | "Yandex"
     | "YandexMobileBot"
     | "Algolia Crawler"
     | "BublupBot"
     | "CCBot"
     | "Cliqzbot"
     | "Daumoa"
     | "DeuSu"
     | "EuripBot"
     | "Exploratodo"
     | "Feedly"
     | "Findxbot"
     | "istellabot"
     | "JikeSpider"
     | "Lycos"
     | "Mail.Ru"
     | "MojeekBot"
     | "OrangeBot"
     | "Pinterest"
     | "Plukkie"
     | "Qwantify"
     | "Rambler"
     | "SemanticScholarBot"
     | "SeznamBot"
     | "Sosospider"
     | "Slurp"
     | "Twitterbot"
     | "WhatsApp"
     | "yacybot"
     | "YepBot"
     | "Yeti"
     | "YioopBot"
     | "yoozBot"
     | "YoudaoBot"
     | (
     | "*"
     | "360Spider"
     | "360Spider-Image"
     | "360Spider-Video"
     | "HaoSouSpider"
     | "AdsBot-Google"
     | "AdsBot-Google-Mobile"
     | "AdsBot-Google-Mobile-Apps"
     | "Googlebot"
     | "Googlebot-Image"
     | "Googlebot-Mobile"
     | "Googlebot-News"
     | "Googlebot-Video"
     | "Mediapartners-Google"
     | "adidxbot"
     | "bingbot"
     | "BingPreview"
     | "MicrosoftPreview"
     | "msnbot"
     | "msnbot-media"
     | "Applebot"
     | "AppleNewsBot"
     | "Baiduspider"
     | "Baiduspider-image"
     | "Baiduspider-mobile"
     | "Baiduspider-news"
     | "Baiduspider-video"
     | "coccoc"
     | "coccocbot-image"
     | "coccocbot-web"
     | "DuckDuckBot"
     | "DuckDuckGo-Favicons-Bot"
     | "facebookcatalog"
     | "facebookexternalhit"
     | "Facebot"
     | "gooblog"
     | "ichiro"
     | "Sogou blog"
     | "Sogou inst spider"
     | "Sogou News Spider"
     | "Sogou Orion spider"
     | "Sogou spider2"
     | "Sogou web spider"
     | "Yandex"
     | "YandexMobileBot"
     | "Algolia Crawler"
     | "BublupBot"
     | "CCBot"
     | "Cliqzbot"
     | "Daumoa"
     | "DeuSu"
     | "EuripBot"
     | "Exploratodo"
     | "Feedly"
     | "Findxbot"
     | "istellabot"
     | "JikeSpider"
     | "Lycos"
     | "Mail.Ru"
     | "MojeekBot"
     | "OrangeBot"
     | "Pinterest"
     | "Plukkie"
     | "Qwantify"
     | "Rambler"
     | "SemanticScholarBot"
     | "SeznamBot"
     | "Sosospider"
     | "Slurp"
     | "Twitterbot"
     | "WhatsApp"
     | "yacybot"
     | "YepBot"
     | "Yeti"
     | "YioopBot"
     | "yoozBot"
     | "YoudaoBot")[];
};

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:265^

Type Declaration

Section titled “Type Declaration”

allow?
Section titled “allow?”
optional allow: string | string[];
Description
Section titled “Description”

[ At least one or more allow or disallow entries per rule ] Allows indexing site sections or individual pages.

Example
Section titled “Example”
policy:[{allow:["/"]}]

Path-based URL matching, refer to SYNTAX^ via Google.

cleanParam?
Section titled “cleanParam?”
optional cleanParam: string | string[];
Description
Section titled “Description”

[ Optional ] Indicates to the robot that the page URL contains parameters (like UTM tags) that should be ignored when indexing it.

Example
Section titled “Example”
Terminal window
# for URLs like:
www.example2.com/index.php?page=1&sid=2564126ebdec301c607e5df
www.example2.com/index.php?page=1&sid=974017dcd170d6c4a5d76ae
policy:[
 {
   cleanParam: [
     "sid /index.php",
   ]
 }
]

For additional examples, please consult Yandex’s SYNTAX^ guide.

crawlDelay?
Section titled “crawlDelay?”
optional crawlDelay: number;
Description
Section titled “Description”

[ Optional ] Specifies the minimum interval (in seconds) for the search robot to wait after loading one page, before starting to load another.

Example
Section titled “Example”
policy:[{crawlDelay:5}]

About the Crawl-delay^ directive.

disallow?
Section titled “disallow?”
optional disallow: string | string[];
Description
Section titled “Description”

[ At least one or more disallow or allow entries per rule ] Prohibits indexing site sections or individual pages.

Example
Section titled “Example”
policy:[
 {
   disallow:[
     "/admin",
     "/uploads/1989-08-21/*.jpg$"
   ]
 }
]

Path-based URL matching, refer to SYNTAX^ via Google.

userAgent?
Section titled “userAgent?”
optional userAgent:
  | "*"
  | "360Spider"
  | "360Spider-Image"
  | "360Spider-Video"
  | "HaoSouSpider"
  | "AdsBot-Google"
  | "AdsBot-Google-Mobile"
  | "AdsBot-Google-Mobile-Apps"
  | "Googlebot"
  | "Googlebot-Image"
  | "Googlebot-Mobile"
  | "Googlebot-News"
  | "Googlebot-Video"
  | "Mediapartners-Google"
  | "adidxbot"
  | "bingbot"
  | "BingPreview"
  | "MicrosoftPreview"
  | "msnbot"
  | "msnbot-media"
  | "Applebot"
  | "AppleNewsBot"
  | "Baiduspider"
  | "Baiduspider-image"
  | "Baiduspider-mobile"
  | "Baiduspider-news"
  | "Baiduspider-video"
  | "coccoc"
  | "coccocbot-image"
  | "coccocbot-web"
  | "DuckDuckBot"
  | "DuckDuckGo-Favicons-Bot"
  | "facebookcatalog"
  | "facebookexternalhit"
  | "Facebot"
  | "gooblog"
  | "ichiro"
  | "Sogou blog"
  | "Sogou inst spider"
  | "Sogou News Spider"
  | "Sogou Orion spider"
  | "Sogou spider2"
  | "Sogou web spider"
  | "Yandex"
  | "YandexMobileBot"
  | "Algolia Crawler"
  | "BublupBot"
  | "CCBot"
  | "Cliqzbot"
  | "Daumoa"
  | "DeuSu"
  | "EuripBot"
  | "Exploratodo"
  | "Feedly"
  | "Findxbot"
  | "istellabot"
  | "JikeSpider"
  | "Lycos"
  | "Mail.Ru"
  | "MojeekBot"
  | "OrangeBot"
  | "Pinterest"
  | "Plukkie"
  | "Qwantify"
  | "Rambler"
  | "SemanticScholarBot"
  | "SeznamBot"
  | "Sosospider"
  | "Slurp"
  | "Twitterbot"
  | "WhatsApp"
  | "yacybot"
  | "YepBot"
  | "Yeti"
  | "YioopBot"
  | "yoozBot"
  | "YoudaoBot"
  | (
  | "*"
  | "360Spider"
  | "360Spider-Image"
  | "360Spider-Video"
  | "HaoSouSpider"
  | "AdsBot-Google"
  | "AdsBot-Google-Mobile"
  | "AdsBot-Google-Mobile-Apps"
  | "Googlebot"
  | "Googlebot-Image"
  | "Googlebot-Mobile"
  | "Googlebot-News"
  | "Googlebot-Video"
  | "Mediapartners-Google"
  | "adidxbot"
  | "bingbot"
  | "BingPreview"
  | "MicrosoftPreview"
  | "msnbot"
  | "msnbot-media"
  | "Applebot"
  | "AppleNewsBot"
  | "Baiduspider"
  | "Baiduspider-image"
  | "Baiduspider-mobile"
  | "Baiduspider-news"
  | "Baiduspider-video"
  | "coccoc"
  | "coccocbot-image"
  | "coccocbot-web"
  | "DuckDuckBot"
  | "DuckDuckGo-Favicons-Bot"
  | "facebookcatalog"
  | "facebookexternalhit"
  | "Facebot"
  | "gooblog"
  | "ichiro"
  | "Sogou blog"
  | "Sogou inst spider"
  | "Sogou News Spider"
  | "Sogou Orion spider"
  | "Sogou spider2"
  | "Sogou web spider"
  | "Yandex"
  | "YandexMobileBot"
  | "Algolia Crawler"
  | "BublupBot"
  | "CCBot"
  | "Cliqzbot"
  | "Daumoa"
  | "DeuSu"
  | "EuripBot"
  | "Exploratodo"
  | "Feedly"
  | "Findxbot"
  | "istellabot"
  | "JikeSpider"
  | "Lycos"
  | "Mail.Ru"
  | "MojeekBot"
  | "OrangeBot"
  | "Pinterest"
  | "Plukkie"
  | "Qwantify"
  | "Rambler"
  | "SemanticScholarBot"
  | "SeznamBot"
  | "Sosospider"
  | "Slurp"
  | "Twitterbot"
  | "WhatsApp"
  | "yacybot"
  | "YepBot"
  | "Yeti"
  | "YioopBot"
  | "yoozBot"
  | "YoudaoBot")[];
Description
Section titled “Description”

[ Required ] Indicates the robot to which the rules listed in robots.txt apply.

Example
Section titled “Example”
policy:[
 {
   userAgent: [
     'Googlebot',
     'Applebot',
     'Baiduspider',
     'bingbot'
   ],
   // crawling rule(s) for above bots
 }
]

Verified bots, refer to DITIG^ or Cloudflare Radar^.

RobotsConfig

Section titled “RobotsConfig”
type RobotsConfig = {
  host?: string | boolean;
  policy?: {
     allow?: string | string[];
     cleanParam?: string | string[];
     crawlDelay?: number;
     disallow?: string | string[];
     userAgent?:   | "*"
        | "360Spider"
        | "360Spider-Image"
        | "360Spider-Video"
        | "HaoSouSpider"
        | "AdsBot-Google"
        | "AdsBot-Google-Mobile"
        | "AdsBot-Google-Mobile-Apps"
        | "Googlebot"
        | "Googlebot-Image"
        | "Googlebot-Mobile"
        | "Googlebot-News"
        | "Googlebot-Video"
        | "Mediapartners-Google"
        | "adidxbot"
        | "bingbot"
        | "BingPreview"
        | "MicrosoftPreview"
        | "msnbot"
        | "msnbot-media"
        | "Applebot"
        | "AppleNewsBot"
        | "Baiduspider"
        | "Baiduspider-image"
        | "Baiduspider-mobile"
        | "Baiduspider-news"
        | "Baiduspider-video"
        | "coccoc"
        | "coccocbot-image"
        | "coccocbot-web"
        | "DuckDuckBot"
        | "DuckDuckGo-Favicons-Bot"
        | "facebookcatalog"
        | "facebookexternalhit"
        | "Facebot"
        | "gooblog"
        | "ichiro"
        | "Sogou blog"
        | "Sogou inst spider"
        | "Sogou News Spider"
        | "Sogou Orion spider"
        | "Sogou spider2"
        | "Sogou web spider"
        | "Yandex"
        | "YandexMobileBot"
        | "Algolia Crawler"
        | "BublupBot"
        | "CCBot"
        | "Cliqzbot"
        | "Daumoa"
        | "DeuSu"
        | "EuripBot"
        | "Exploratodo"
        | "Feedly"
        | "Findxbot"
        | "istellabot"
        | "JikeSpider"
        | "Lycos"
        | "Mail.Ru"
        | "MojeekBot"
        | "OrangeBot"
        | "Pinterest"
        | "Plukkie"
        | "Qwantify"
        | "Rambler"
        | "SemanticScholarBot"
        | "SeznamBot"
        | "Sosospider"
        | "Slurp"
        | "Twitterbot"
        | "WhatsApp"
        | "yacybot"
        | "YepBot"
        | "Yeti"
        | "YioopBot"
        | "yoozBot"
        | "YoudaoBot"
        | (
        | "*"
        | "360Spider"
        | "360Spider-Image"
        | "360Spider-Video"
        | "HaoSouSpider"
        | "AdsBot-Google"
        | "AdsBot-Google-Mobile"
        | "AdsBot-Google-Mobile-Apps"
        | "Googlebot"
        | "Googlebot-Image"
        | "Googlebot-Mobile"
        | "Googlebot-News"
        | "Googlebot-Video"
        | "Mediapartners-Google"
        | "adidxbot"
        | "bingbot"
        | "BingPreview"
        | "MicrosoftPreview"
        | "msnbot"
        | "msnbot-media"
        | "Applebot"
        | "AppleNewsBot"
        | "Baiduspider"
        | "Baiduspider-image"
        | "Baiduspider-mobile"
        | "Baiduspider-news"
        | "Baiduspider-video"
        | "coccoc"
        | "coccocbot-image"
        | "coccocbot-web"
        | "DuckDuckBot"
        | "DuckDuckGo-Favicons-Bot"
        | "facebookcatalog"
        | "facebookexternalhit"
        | "Facebot"
        | "gooblog"
        | "ichiro"
        | "Sogou blog"
        | "Sogou inst spider"
        | "Sogou News Spider"
        | "Sogou Orion spider"
        | "Sogou spider2"
        | "Sogou web spider"
        | "Yandex"
        | "YandexMobileBot"
        | "Algolia Crawler"
        | "BublupBot"
        | "CCBot"
        | "Cliqzbot"
        | "Daumoa"
        | "DeuSu"
        | "EuripBot"
        | "Exploratodo"
        | "Feedly"
        | "Findxbot"
        | "istellabot"
        | "JikeSpider"
        | "Lycos"
        | "Mail.Ru"
        | "MojeekBot"
        | "OrangeBot"
        | "Pinterest"
        | "Plukkie"
        | "Qwantify"
        | "Rambler"
        | "SemanticScholarBot"
        | "SeznamBot"
        | "Sosospider"
        | "Slurp"
        | "Twitterbot"
        | "WhatsApp"
        | "yacybot"
        | "YepBot"
        | "Yeti"
        | "YioopBot"
        | "yoozBot"
        | "YoudaoBot")[];
  }[];
  sitemap?: string | boolean | string[];
};

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:266^

Type Declaration

Section titled “Type Declaration”

host?
Section titled “host?”
optional host: string | boolean;
Default
Section titled “Default”
false
Description
Section titled “Description”

[ Optional ] Some crawlers(Yandex) support and only accept domain names.

Example
Section titled “Example”
integrations:[
 robots({
   host: siteUrl.replace(/^https?://|:\d+/g, "")
 })
]

policy?
Section titled “policy?”
optional policy: {
  allow?: string | string[];
  cleanParam?: string | string[];
  crawlDelay?: number;
  disallow?: string | string[];
  userAgent?:   | "*"
     | "360Spider"
     | "360Spider-Image"
     | "360Spider-Video"
     | "HaoSouSpider"
     | "AdsBot-Google"
     | "AdsBot-Google-Mobile"
     | "AdsBot-Google-Mobile-Apps"
     | "Googlebot"
     | "Googlebot-Image"
     | "Googlebot-Mobile"
     | "Googlebot-News"
     | "Googlebot-Video"
     | "Mediapartners-Google"
     | "adidxbot"
     | "bingbot"
     | "BingPreview"
     | "MicrosoftPreview"
     | "msnbot"
     | "msnbot-media"
     | "Applebot"
     | "AppleNewsBot"
     | "Baiduspider"
     | "Baiduspider-image"
     | "Baiduspider-mobile"
     | "Baiduspider-news"
     | "Baiduspider-video"
     | "coccoc"
     | "coccocbot-image"
     | "coccocbot-web"
     | "DuckDuckBot"
     | "DuckDuckGo-Favicons-Bot"
     | "facebookcatalog"
     | "facebookexternalhit"
     | "Facebot"
     | "gooblog"
     | "ichiro"
     | "Sogou blog"
     | "Sogou inst spider"
     | "Sogou News Spider"
     | "Sogou Orion spider"
     | "Sogou spider2"
     | "Sogou web spider"
     | "Yandex"
     | "YandexMobileBot"
     | "Algolia Crawler"
     | "BublupBot"
     | "CCBot"
     | "Cliqzbot"
     | "Daumoa"
     | "DeuSu"
     | "EuripBot"
     | "Exploratodo"
     | "Feedly"
     | "Findxbot"
     | "istellabot"
     | "JikeSpider"
     | "Lycos"
     | "Mail.Ru"
     | "MojeekBot"
     | "OrangeBot"
     | "Pinterest"
     | "Plukkie"
     | "Qwantify"
     | "Rambler"
     | "SemanticScholarBot"
     | "SeznamBot"
     | "Sosospider"
     | "Slurp"
     | "Twitterbot"
     | "WhatsApp"
     | "yacybot"
     | "YepBot"
     | "Yeti"
     | "YioopBot"
     | "yoozBot"
     | "YoudaoBot"
     | (
     | "*"
     | "360Spider"
     | "360Spider-Image"
     | "360Spider-Video"
     | "HaoSouSpider"
     | "AdsBot-Google"
     | "AdsBot-Google-Mobile"
     | "AdsBot-Google-Mobile-Apps"
     | "Googlebot"
     | "Googlebot-Image"
     | "Googlebot-Mobile"
     | "Googlebot-News"
     | "Googlebot-Video"
     | "Mediapartners-Google"
     | "adidxbot"
     | "bingbot"
     | "BingPreview"
     | "MicrosoftPreview"
     | "msnbot"
     | "msnbot-media"
     | "Applebot"
     | "AppleNewsBot"
     | "Baiduspider"
     | "Baiduspider-image"
     | "Baiduspider-mobile"
     | "Baiduspider-news"
     | "Baiduspider-video"
     | "coccoc"
     | "coccocbot-image"
     | "coccocbot-web"
     | "DuckDuckBot"
     | "DuckDuckGo-Favicons-Bot"
     | "facebookcatalog"
     | "facebookexternalhit"
     | "Facebot"
     | "gooblog"
     | "ichiro"
     | "Sogou blog"
     | "Sogou inst spider"
     | "Sogou News Spider"
     | "Sogou Orion spider"
     | "Sogou spider2"
     | "Sogou web spider"
     | "Yandex"
     | "YandexMobileBot"
     | "Algolia Crawler"
     | "BublupBot"
     | "CCBot"
     | "Cliqzbot"
     | "Daumoa"
     | "DeuSu"
     | "EuripBot"
     | "Exploratodo"
     | "Feedly"
     | "Findxbot"
     | "istellabot"
     | "JikeSpider"
     | "Lycos"
     | "Mail.Ru"
     | "MojeekBot"
     | "OrangeBot"
     | "Pinterest"
     | "Plukkie"
     | "Qwantify"
     | "Rambler"
     | "SemanticScholarBot"
     | "SeznamBot"
     | "Sosospider"
     | "Slurp"
     | "Twitterbot"
     | "WhatsApp"
     | "yacybot"
     | "YepBot"
     | "Yeti"
     | "YioopBot"
     | "yoozBot"
     | "YoudaoBot")[];
}[];
Description
Section titled “Description”

[ Optional ] List of policy rules.

Default
Section titled “Default”
policy:[
 {
   userAgent: "*",
   allow: "/"
 }
]

For more help, refer to SYNTAX^ by Yandex.

sitemap?
Section titled “sitemap?”
optional sitemap: string | boolean | string[];
Description
Section titled “Description”

[ Optional, zero or more per file ] The location of a sitemap for this website.

Example
Section titled “Example”
sitemap: [
 "https://example.com/sitemap.xml",
 "https://www.example.com/sitemap.xml"
]

The value of the SITEMAP^ field is case-sensitive.

SearchEngine

Section titled “SearchEngine”
type SearchEngine =
  | "360Spider"
  | "360Spider-Image"
  | "360Spider-Video"
  | "HaoSouSpider"
  | "AdsBot-Google"
  | "AdsBot-Google-Mobile"
  | "AdsBot-Google-Mobile-Apps"
  | "Googlebot"
  | "Googlebot-Image"
  | "Googlebot-Mobile"
  | "Googlebot-News"
  | "Googlebot-Video"
  | "Mediapartners-Google"
  | "adidxbot"
  | "bingbot"
  | "BingPreview"
  | "MicrosoftPreview"
  | "msnbot"
  | "msnbot-media"
  | "Applebot"
  | "AppleNewsBot"
  | "Baiduspider"
  | "Baiduspider-image"
  | "Baiduspider-mobile"
  | "Baiduspider-news"
  | "Baiduspider-video"
  | "coccoc"
  | "coccocbot-image"
  | "coccocbot-web"
  | "DuckDuckBot"
  | "DuckDuckGo-Favicons-Bot"
  | "facebookcatalog"
  | "facebookexternalhit"
  | "Facebot"
  | "gooblog"
  | "ichiro"
  | "Sogou blog"
  | "Sogou inst spider"
  | "Sogou News Spider"
  | "Sogou Orion spider"
  | "Sogou spider2"
  | "Sogou web spider"
  | "Yandex"
  | "YandexMobileBot"
  | "Algolia Crawler"
  | "BublupBot"
  | "CCBot"
  | "Cliqzbot"
  | "Daumoa"
  | "DeuSu"
  | "EuripBot"
  | "Exploratodo"
  | "Feedly"
  | "Findxbot"
  | "istellabot"
  | "JikeSpider"
  | "Lycos"
  | "Mail.Ru"
  | "MojeekBot"
  | "OrangeBot"
  | "Pinterest"
  | "Plukkie"
  | "Qwantify"
  | "Rambler"
  | "SemanticScholarBot"
  | "SeznamBot"
  | "Sosospider"
  | "Slurp"
  | "Twitterbot"
  | "WhatsApp"
  | "yacybot"
  | "YepBot"
  | "Yeti"
  | "YioopBot"
  | "yoozBot"
  | "YoudaoBot";

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:263^

UserAgent

Section titled “UserAgent”
type UserAgent =
  | "*"
  | "360Spider"
  | "360Spider-Image"
  | "360Spider-Video"
  | "HaoSouSpider"
  | "AdsBot-Google"
  | "AdsBot-Google-Mobile"
  | "AdsBot-Google-Mobile-Apps"
  | "Googlebot"
  | "Googlebot-Image"
  | "Googlebot-Mobile"
  | "Googlebot-News"
  | "Googlebot-Video"
  | "Mediapartners-Google"
  | "adidxbot"
  | "bingbot"
  | "BingPreview"
  | "MicrosoftPreview"
  | "msnbot"
  | "msnbot-media"
  | "Applebot"
  | "AppleNewsBot"
  | "Baiduspider"
  | "Baiduspider-image"
  | "Baiduspider-mobile"
  | "Baiduspider-news"
  | "Baiduspider-video"
  | "coccoc"
  | "coccocbot-image"
  | "coccocbot-web"
  | "DuckDuckBot"
  | "DuckDuckGo-Favicons-Bot"
  | "facebookcatalog"
  | "facebookexternalhit"
  | "Facebot"
  | "gooblog"
  | "ichiro"
  | "Sogou blog"
  | "Sogou inst spider"
  | "Sogou News Spider"
  | "Sogou Orion spider"
  | "Sogou spider2"
  | "Sogou web spider"
  | "Yandex"
  | "YandexMobileBot"
  | "Algolia Crawler"
  | "BublupBot"
  | "CCBot"
  | "Cliqzbot"
  | "Daumoa"
  | "DeuSu"
  | "EuripBot"
  | "Exploratodo"
  | "Feedly"
  | "Findxbot"
  | "istellabot"
  | "JikeSpider"
  | "Lycos"
  | "Mail.Ru"
  | "MojeekBot"
  | "OrangeBot"
  | "Pinterest"
  | "Plukkie"
  | "Qwantify"
  | "Rambler"
  | "SemanticScholarBot"
  | "SeznamBot"
  | "Sosospider"
  | "Slurp"
  | "Twitterbot"
  | "WhatsApp"
  | "yacybot"
  | "YepBot"
  | "Yeti"
  | "YioopBot"
  | "yoozBot"
  | "YoudaoBot";

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:264^

Schema for validating user agent strings in robots.txt rules.

Accepts either the wildcard ’*’ (representing all user agents) or a value from the SearchEngines schema.

See

Section titled “See”

SearchEngines for the list of supported search engine user agents.

Variables

Section titled “Variables”

PolicyOptionsSchema

Section titled “PolicyOptionsSchema”
const PolicyOptionsSchema: ZodObject<PolicyOptions>;

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:134^

RobotsTXTConfigSchema

Section titled “RobotsTXTConfigSchema”
const RobotsTXTConfigSchema: ZodObject<RobotsConfig>;

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:218^

UserAgentSchema

Section titled “UserAgentSchema”
const UserAgentSchema: ZodUnion<UserAgent>;

Defined in: studiocms/packages/studiocms/src/integrations/robots/schema.ts:132^

Schema for validating user agent strings in robots.txt rules.

Accepts either the wildcard ’*’ (representing all user agents) or a value from the SearchEngines schema.

See

Section titled “See”

SearchEngines for the list of supported search engine user agents.