Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
N
Nutzungsvorbehalt
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Analyze
Contributor analytics
Help
Help
Support
GitLab documentation
Compare GitLab plans
GitLab community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Modul 12 (SEO)
Nutzungsvorbehalt
Commits
68d48c8e
Commit
68d48c8e
authored
1 month ago
by
Jan-Philipp Igla
Browse files
Options
Downloads
Patches
Plain Diff
add README and add robots.txt
parents
No related branches found
No related tags found
No related merge requests found
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
README.md
+136
-0
136 additions, 0 deletions
README.md
robots.txt
+120
-0
120 additions, 0 deletions
robots.txt
with
256 additions
and
0 deletions
README.md
0 → 100644
+
136
−
0
View file @
68d48c8e
# Nutzungsvorbehalt der ARD
**Stand: 13.05.2025**
-
[ ] Füge bitte folgenden Block (bitte nicht verändern!) Deiner
**robots.txt**
hinzu (Bsp. https://www.domain.de/robots.txt)
-
[ ] Prüfe nach Veröffentlichung, ob Deine robots.txt die neusten Änderungen enthält
-
[ ] Nutze das Tool https://technicalseo.com/tools/robots-txt/ zur Validierung der neuen robots.txt
**Nach erstmaliger Anpassung Deiner robots.txt**
: Gibt bitte eine Rückmeldung samt URL (https://www.domain.de/robots.txt) an philipp.igla@hr.de
Bei Rückfragen oder Unklarheiten bitte jederzeit bei philipp.igla@hr.de (Modul SEO ARD) melden.
---
```
txt
############
# Stand: 13.05.2025
# Liste an Crawler (Modul SEO ARD)
# Amazon
User-agent: Amazonbot
Disallow: /
# Anthropic
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: claude-web
Disallow: /
# Apple
User-agent: Applebot-Extended
Disallow: /
# ByteDance
User-agent: Bytespider
Disallow: /
User-agent: TikTokSpider
Disallow: /
# Cohere
User-agent: cohere-ai
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /
# Common Crawl
User-agent: CCBot
Disallow: /
# Diffbot
User-agent: DiffBot
Disallow: /
# DuckDuckGo
User-agent: DuckAssistBot
Disallow: /
# Google
User-agent: Google-Extended
Disallow: /
# Huawei
User-agent: PetalBot
Disallow: /
User-agent: PanguBot
Disallow: /
# Meta
User-agent: meta-externalagent
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: FacebookBot
Disallow: /
# Mistral
User-agent: MistralAI-User
Disallow: /
# OpenAI
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User/2.0
Disallow: /
# Perplexity
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
# Webz.io
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
# You.com
User-agent: YouBot
Disallow: /
# Zyte
User-agent: Scrapy
Disallow: /
############
```
This diff is collapsed.
Click to expand it.
robots.txt
0 → 100644
+
120
−
0
View file @
68d48c8e
############
# Stand: 13.05.2025
# Liste an Crawler (Modul SEO ARD)
# Amazon
User-agent: Amazonbot
Disallow: /
# Anthropic
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: claude-web
Disallow: /
# Apple
User-agent: Applebot-Extended
Disallow: /
# ByteDance
User-agent: Bytespider
Disallow: /
User-agent: TikTokSpider
Disallow: /
# Cohere
User-agent: cohere-ai
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /
# Common Crawl
User-agent: CCBot
Disallow: /
# Diffbot
User-agent: DiffBot
Disallow: /
# DuckDuckGo
User-agent: DuckAssistBot
Disallow: /
# Google
User-agent: Google-Extended
Disallow: /
# Huawei
User-agent: PetalBot
Disallow: /
User-agent: PanguBot
Disallow: /
# Meta
User-agent: meta-externalagent
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: FacebookBot
Disallow: /
# Mistral
User-agent: MistralAI-User
Disallow: /
# OpenAI
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User/2.0
Disallow: /
# Perplexity
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
# Webz.io
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
# You.com
User-agent: YouBot
Disallow: /
# Zyte
User-agent: Scrapy
Disallow: /
############
\ No newline at end of file
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment