robots.txt 2020: для рукописного сайта Codeinteger
Всем привет.
Перелопатил кучу статей с комментами, кто-то пишет открывайте полностью:
User-agent: *
Disallow:
Host: https://oneyearinthailand.com
Sitemap: https://oneyearinthailand.com/sitemap.xml
, кто-то пишет скрывайте админ-зону и другие технические (ненужные) директории:
User-agent: *
Disallow: /admin*
Disallow: /search
Disallow: /*/index
Disallow: /index
Disallow: /profile
Disallow: /user
Disallow: /cache
Disallow: /modules
Disallow: /archive
Disallow: /fonts
Disallow: /img
Disallow: /js/admin
Disallow: /ci
Disallow: /assets
Disallow: /uploads
Disallow: /vacancy
Disallow: /questions
Disallow: /soglashenie
Disallow: /donates
Disallow: /archive
Disallow: /app
Disallow: /oneyearinthailand.com/categories/contest
Disallow: /oneyearinthailand.com/categories/tovari
Disallow: /oneyearinthailand.com/categories/culture
Disallow: /oneyearinthailand.com/categories/citation
Disallow: /oneyearinthailand.com/articles/84-the-places-we-visited
Disallow: /oneyearinthailand.com/categories/service
Disallow: /oneyearinthailand.com/categories/country/thailand/phuket
Disallow: /oneyearinthailand.com/categories/motherhood/planning
Disallow: /oneyearinthailand.com/categories/country/thailand/phuket/beaches
Disallow: /oneyearinthailand.com/categories/motherhood
Disallow: /oneyearinthailand.com/categories/service/excursions
Disallow: /oneyearinthailand.com/categories/country/thailand/phuket/frukti
Disallow: /oneyearinthailand.com/categories/spending
Disallow: /oneyearinthailand.com/categories/culture/holidays
Disallow: /oneyearinthailand.com/categories/country/russia/religion
Disallow: /oneyearinthailand.com/categories/journal
Disallow: /oneyearinthailand.com/categories/country/russia
Disallow: /oneyearinthailand.com/categories/spending/price-for-childbirth-in-thailand
Disallow: /oneyearinthailand.com/categories/spending/zatraty-za-vs-vremya
Disallow: /oneyearinthailand.com/categories/country/thailand/tasks
Disallow: /oneyearinthailand.com/categories/country/thailand/videoblog
Disallow: /oneyearinthailand.com/categories/country/thailand/phuket/sights
Disallow: /oneyearinthailand.com/categories/country/thailand/ways
Disallow: /oneyearinthailand.com/agreement
Disallow: /oneyearinthailand.com/categories/country/thailand/excursions-1
Disallow: /oneyearinthailand.com/categories/country/thailand/dokumenti/vizi
Disallow: /oneyearinthailand.com/categories/country/thailand/dokumenti
Disallow: /oneyearinthailand.com/categories/country/thailand
Disallow: /oneyearinthailand.com/categories/country/thailand/phuket/finance
Disallow: /oneyearinthailand.com/categories/country
Disallow: /oneyearinthailand.com/uploads/posts/125/d019e-new-years-eve-preview-new-0-190-110.jpeg
Disallow: /oneyearinthailand.com/categories/tsitaty/authors
Disallow: /oneyearinthailand.com/articles/157-igor-dvorovs-videoblog-meet
Disallow: /oneyearinthailand.com/articles/170-igor-dvorovs-videoblog-how-to-prepare-for-pregnancy-mans-opinion
Disallow: /oneyearinthailand.com/articles/158-igor-dvorovs-videoblog-beginning
В итоге есть какое-то однозначное, правильное решение? - подскажите, пожалуйста, кто в теме
|