MediaWiki:Robots.txt: Unterschied zwischen den Versionen
Aus AuroraWiki
Robots.txt erstellt |
Formatierung |
||
| Zeile 1: | Zeile 1: | ||
<pre> | |||
robots.txt for https://www.aurorawiki.de/ | |||
User-agent: GPTBot | User-agent: GPTBot | ||
| Zeile 115: | Zeile 114: | ||
Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search | Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search | ||
Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch | Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch | ||
# | #</pre> | ||
Version vom 3. Oktober 2024, 12:16 Uhr
robots.txt for https://www.aurorawiki.de/ User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: fast Disallow: / User-agent: wget Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: * Allow: /w/api.php?action=mobileview& Allow: /w/load.php? Allow: /api/rest_v1/?doc Disallow: /w/ Disallow: /api/ Disallow: /trap/ Disallow: /wiki/Vorlage: Disallow: /wiki/Special: Disallow: /wiki/Spezial: Disallow: /wiki/Spesial: Disallow: /wiki/Special%3A Disallow: /wiki/Spezial%3A Disallow: /wiki/Spesial%3A Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch #
