MediaWiki:Robots.txt: Unterschied zwischen den Versionen
Aus AuroraWiki
Formatierung |
Ergänzung |
||
| (Eine dazwischenliegende Version desselben Benutzers wird nicht angezeigt) | |||
| Zeile 1: | Zeile 1: | ||
<pre> | <pre> | ||
robots.txt for https://www.aurorawiki.de/ | robots.txt for https://www.aurorawiki.de/ | ||
User-agent: nbot | |||
Disallow: / | |||
User-agent: GPTBot | User-agent: GPTBot | ||
Aktuelle Version vom 9. Oktober 2024, 08:43 Uhr
robots.txt for https://www.aurorawiki.de/ User-agent: nbot Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: fast Disallow: / User-agent: wget Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: * Allow: /w/api.php?action=mobileview& Allow: /w/load.php? Allow: /api/rest_v1/?doc Disallow: /w/ Disallow: /api/ Disallow: /trap/ Disallow: /wiki/Vorlage: Disallow: /wiki/Special: Disallow: /wiki/Spezial: Disallow: /wiki/Spesial: Disallow: /wiki/Special%3A Disallow: /wiki/Spezial%3A Disallow: /wiki/Spesial%3A Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch #
