Quantcast
Channel: How to assign specific sitemaps for specific crawler-bots in robots.txt? - Stack Overflow
Viewing all articles
Browse latest Browse all 2

How to assign specific sitemaps for specific crawler-bots in robots.txt?

$
0
0

Since some crawlers don't like the sitemap versions made for Google, I made different sitemaps.

And there is an option to put

Sitemap: http://example.com/sitemap.xml

to robots.txt. But is it possible to put it kinda like this:

User-agent: *Sitemap: http://example.com/sitemap.xmlUser-agent: googlebotSitemap: http://example.com/sitemap-for-google.xml

I couldn't find any resource for this topic and robots.txt is not something I want to joke around with.


Viewing all articles
Browse latest Browse all 2

Latest Images

Trending Articles





Latest Images