Making Sitemaps Easier to Manage and Scale
Date : 2008 02 27 Category : Search Engines Yahoo! Search BlogToday, Yahoo!, along with Google and Microsoft, is announcing Cross-host submission for Sitemaps, which will make it easier for webmasters to manage their Sitemap submissions to the major search engines. With this announcement, webmasters can now submit Sitemaps that correspond to several differently-hosted websites using a single mechanism.
For background, a Sitemap file contains the URLs for the pages on a site along with meta-data, such as priority, last crawled date and change frequency for the content. To ensure validity of this metadata, Sitemaps have previously been required to be on the same host and path as the URLs they contain. This requirement forced the Sitemaps files to be hosted on the same servers as the actual site content.
With today's announcement, a Sitemap can now be hosted on a different host and path than the URLs it contains. For example, say you have a Sitemap (sitemap-www.xml) for the URLs on http://www.example.com but you want to put that Sitemap on http://sitemaps.example.com. That is now possible. To make the Sitemap valid and preserve data security you need to refer to it from the robots.txt file on the site where the URLs it contains are located. For example, add the following line to http://www.example.com/robots.txt:
Sitemap: http://sitemaps.example.com/sitemap-www.xml
Our collaboration with Google and Microsoft began back in November 2006 when we announced joint support for the Sitemaps protocol. Since then, we've learned a lot about how webmasters and site owners manage their sites and feeds. We know that segregating user facing content from feeds, like Sitemaps, is important. We've also learned that managing feeds for large websites or websites using third party feed publishing services is critical. We hope this enhancement helps address those needs.
We'll continue to work on addressing the needs of our webmasters through new standards and protocols. If you have other thoughts about how we can collaborate with other search engines on standards such as robots.txt, we'd love to hear from you. Leave us a comment below, or give us your feedback here.
Details about the Sitemaps protocol, including our recent addition, are available on the protocol website. Or, if you're at SMXW this week, bring your questions to our panelists and speakers.
Priyank Garg
Director, Product Management, Yahoo! Search
Sean Suchter
Vice President, Engineering, Yahoo! Search