Initial commit
This commit is contained in:
commit
97e799fd52
61 changed files with 2252 additions and 0 deletions
17
yksa_web/static/yksa/robots.txt
Normal file
17
yksa_web/static/yksa/robots.txt
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# robots.txt for yktaero-web
|
||||
|
||||
# Allow Google
|
||||
User-agent: Googlebot
|
||||
Allow: /
|
||||
Crawl-delay: 5
|
||||
Disallow: /admin/
|
||||
|
||||
# Allow Yandex
|
||||
User-agent: Yandex
|
||||
Allow: /
|
||||
Crawl-delay: 5
|
||||
Disallow: /admin/
|
||||
|
||||
# Disallow all other robots
|
||||
User-agent: *
|
||||
Disallow: /
|
||||
Loading…
Add table
Add a link
Reference in a new issue