Sitemap Submission Made Simple

Thursday, December 18, 2008 at 7:58 AM

Submitting a Sitemap to Google just became even easier. No longer do you have to specify the Sitemap file type—we'll determine the type of data you're submitting automatically. Let's take a quick look at the kinds of Sitemap files we support as well as the ways they can be submitted to us.

Sitemap file formats supported by Google
Part of what makes the web so interesting is that there are so many different kinds of content out there. Do you use videos on your website? If so, send us a Video Sitemap file so that we can send you visitors to those videos! Do you host source-code samples? Submit a Code Search Sitemap! Here are the various kinds of Sitemap files that Google supports at the moment:

XML Sitemap files for web pages - Use these files to submit all of your web pages (this is the preferred format for web pages). While not all search engines may support the Sitemap types listed below, the XML Sitemap for web pages is supported by all search engines of sitemaps.org.

RSS 2.0 and Atom 1.0 feeds for web pages - Many blogs create these automatically.

Text files with web page URLs - If you can't automatically create one of the above formats, you can create a text file with your URLs in it.

XML Sitemap files for Video Search - Videos on your website can be indexed and made available for Google Video Search.

Media-RSS feeds for Video Search - mRSS feeds are used by various other systems, we can use these for Google Video Search as well.

XML Sitemap files for Google Code Search - If you make programming samples or code available to your users, you can submit these for Google Code Search.

XML Sitemap files for mobile web pages - Using this kind of format allows us to recognize content that has been optimized for mobile devices (please note that there was recently a small change in the format).

XML Sitemap files for geo-data - If you have geographic data on your website in the form of KML or GeoRSS files, please let us know about these files.

XML Sitemap files for News - News websites can submit their news content in this special Sitemap format (please note that you must first register with Google News before these files are processed).

If you have multiple Sitemap files that you wish to submit to Google, you can include up to 1,000 of these in an XML Sitemap Index file. If you have more than 1,000 Sitemap files, you can just submit multiple Sitemap Index files - we'd love to take them all!

Submitting your Sitemap files to Google
Once you have your Sitemap files ready and available on your server, all that's left is making sure that the search engines can find them. Google supports three simple ways to submit Sitemap files:

Using Google Webmaster Tools
Submitting your Sitemap files through Google Webmaster Tools is the preferred way of letting us know about them. The main advantage of doing it this way is that you'll always have direct feedback about how your Sitemap files were downloaded (were we able to reach your server?), how they were recognized (were they in the right format?) and what happened to the web pages listed in them (how many were indexed?). To submit your Sitemap files, make sure that your website is verified in Webmaster Tools, then go to "Sitemaps" in Webmaster Tools and enter the file name of your Sitemap(s).

Sometimes it makes sense to keep your Sitemap file on a different server / domain name. To submit Sitemap files like that, you must verify ownership of both sites in Webmaster Tools and submit the Sitemap on the appropriate site. For instance, if your Sitemap file for http://www.example.com is kept on http://sitemap-files.example.com/ then you need to verify ownership of both sites and then submit the Sitemap file under http://sitemap-files.example.com (even though the URLs listed in it are for http://www.example.com). For more information, please see our Help Center topic on submitting Sitemap files for multiple sites.

Listing Sitemap files in the robots.txt file
Another way of submitting a Sitemap file is to specify the URL in your robots.txt file. If you use this method of submitting a Sitemap file, it will be found by all search engines that support the Sitemaps protocol (although not all of them support the extensions listed above). Since you can specify the full URL of your Sitemap file in the robots.txt file, this method also allows you to store your Sitemap file on a different domain. Keep in mind that while Sitemap files submitted this way are processed on our side, they will not be automatically listed in your Webmaster Tools account. In order to receive feedback on your files, we recommend adding them manually to your account as well.

Using an HTTP "ping"
If your Sitemap files are generated automatically, a convenient way to submit (and re-submit) them is to access the "ping" URL for Google Sitemaps. This URL includes the URL of your Sitemap file. For more information on the "ping" URL for your website, please see the Help Center article on Updating a Sitemap. Feel free to "ping" this URL whenever you update your Sitemap file - we'll know to pick it up and process it again. If you also have your Sitemap file registered in Webmaster Tools, we'll update the status there as well. This method is also valid if your Sitemap file is kept on a different server, but you must still verify both sites in Webmaster Tools as previously mentioned.

Search engines that are a members of sitemaps.org support a similar way of submitting general web Sitemap files.

We hope these simplifications make it even easier for you to send us your Sitemap files!

Posted by John Mueller, Webmaster Trends Analyst, Google Z├╝rich


Using ASP to Code an RSS Feed

Guest Author - Lisa Shea

RSS feeds let you easily syndicate your content to an end user or another website. ASP can help you easily create your own RSS feed for your website. I know what I'm about to describe will sound complicated at first :). The best way to learn this system is simply to give it a try, and things will make sense when you see them in action.

First off, what IS an RSS feed? Think of it as a news alert. If you have an RSS reader, it actively polls a website to see if anything new is added. It typically runs as a taskbar icon on your desktop. Let's say you are polling this ASP site. Every time I add a new article, the RSS feed will update for my site. Your reader will see that new article and put a little alert on your desktop. You click on your reader, see a brief description of my article, and can read it if you want.

To get a reader, go to RSSReader.com. It's free and easy. Now you can test how RSS works.

So now that you have a reader, you need to point the reader at an RSS feed, so it gets the updates of what is going on at the site. In essence, you point the reader at a XXXXXXX.RSS file. The format of the RSS feed is XML. XML is a tag language, just like HTML. Instead of having commands like HTML and BODY, in the RSS file you end up with tags such as COPYRIGHT and PUBDATE. The tags in the RSS file help the RSS reader describe each "news flash" it is displaying.

Go into your RSS reader and create a new feed, using this RSS feed source:


That is the feed for this ASP site. See how your RSS reader now shows the latest 3 articles from this site? Whenever I add a new article to this ASP site, the reader will alert you and let you see what it's about. This is a GREAT way to stay in touch with your website visitors and to let them know what is fresh and new.

So now that you understand what RSS is all about, and how it works, it's time to look into how to code a RSS file for your own website, to alert your visitors to changes you've made.


Blogs,RSS News feeders, and ATOM [ Part One ]

In this article you will be shown RSS standard, history and versions, also you will get familiar with the new next generation standard which is ATOM. Also you will get a brief history of Blogs, News Readers and its evolution, and the strong relationship between blogs and RSS. Most, if not all, of the news sites and agencies support a news feed link to the latest news or breaking news. Actually I consider the website that's not showing the news feed magic icon a poor one, you will find in CodeProject a feed for the latest articles and site news, so everybody should be familiar with the concept of news feeds, both developers and novice users!

Each and every morning, I visit BBC website and check the latest added news, then I move to MSDN just published articles, then I jump to CodeProject and check the latest articles, so ain't you find this process so tedious? Actually I suffer from this sometimes, I get blue, I guess you do the same, so what if you have some way to get all the latest content of one website or thousands of websites using some utility, then you can go offline but you have all the cream of one site or more! I think it would be a great utility. This tool is as important to web site content as Outlook Express to mail server, you grab all your mails and then go offline, you can check them out anytime.

What is to be shown ?
This article is divided into two parts; the first one shows you the history and evolution of RSS standard and the next standard that emerged from RSS which is ATOM, and what a news reader is. The first part will be clear for everybody, even for non-developers. Second part shows you how to make your own news feed and how to consume the others' feeds and show the cream or the latest contents of one site into your own website.

Weblogs and bloggers
Blog is a shortcut for WebBlog. It’s the process of "logging" into a website. Blog is a medium in which an author writes a journal-style web site and the readers can respond to the material or topics provided. Bloggers are the individuals who are authoring or responding to topics provided in a blog. Blogs serve a wide spectrum of interests ranging from politics to chemical issues!. You can check out some blogs at MSDN.

Get your blog now free
Actually, nobody now has no blog, you can now have your blog and register and record your thoughts, upload your weekend pictures, write your technical notes, write quotes you like, just get your blog and have a corner to express your own thoughts. Today, you have many free blog engines. You can check out Blogger, Blogging. You can view my blog here.

.NET Most Known Blogging Engines
dasBlog : das blog stores data inside XML files.
.Text : stores data using SQL Server. It's open source project so that you can download it and view code. You can install and configure it. You can view the workspace of .Text on gotdotnet website. .Text is the most powerful blogging engine as it is the driver of ASP.NET blogs.

What is RSS ?
Imagine that you post an entry to a blog of some type and you wait for a response to the entry you have been posted, you may get response after 1 min, or 1 hour, if even worse after 1 or 3 days; so you are asked to visit the blog again and more again to check if you’ve got some answer to your question!! So how Poor and Boring!!? What do you think of some type of technology to provide you with all the content of some weblog or any other website in a standard, agreed on, well-known format. It’s RSS, it’s the Buzzword! You can get the cream of one site or blog using a program which falls under the umbrella of News Aggregator applications. Websites including weblogs provide a feed to its content (or top useful content) by providing a dynamically generated RSS document available at a URL, RSS document is nothing but an XML file ( WoW !!, XML again). It seems that each new technology or standard is based heavily on XML.

RSS document can contain any of the following contents and more:

Weather Conditions
Stock Market
Latest Post in Forums
Publishers’ Books
Online Bidding
Articles List
Latest News
Suppliers’ Products
Blogs Entries
Technology Related Issues
RSS Aggregators and News Readers

RSS Aggregator is the same as News Reader, so what is the News Reader? It's an application that reads RSS documents, and lists items of the document in a readable polished view. You can subscribe in an RSS feed by simply entering the URL of the output RSS document into your RSS Aggregator and you’re done now .. ! You have the most valuable contents of that site into your fingertips, you can filter content, read brief about some launched software programs or products, if you like the brief, you can go directly to the full content. Moreover You can search old items. There are many RSS Aggregators in the market as NewzCrawler, SharpReader, SlashDot. SharpReader is free but NewzCrawler is much more stronger and usable than SharpReader, but it is not for free. Furthermore, you can get an online News Aggregator, it’s nice when you need to check out your feeds from anywhere, but the downside is that you need to login to check your feeds, you don't have the offline reading feature of desktop news reader. You can check out NewsGator. You can view a list of almost all the available news aggregators here, all these aggregators support RSS 2.0 ( more about versions later ).

RSS Readers Common Features
Most RSS Aggregators Applications make you feel that you are just reading your e-mail, just as you use MS Outlook, or any other Mail Program, most of them has a tree on the left listing all the feeds (Channels) and to the right the items of the channel.

RSS Aggregator has an embedded browser, so you follow the full story right from the same application; no need to copy and paste the URL and open it in an external window, it uses IE infrastructure behind the scenes.

Now I can read all the technical articles I need right from my desktop, organized and just to the point, no trivial issues, all you need to do is to get the URL of the RSS document and just add new channel to my RSS aggregator application and I am done.

Most Sites provide or syndicate content using RSS, for example, News agencies such as Wired, Cnet, Yahoo, NPR News, Reuters, CNN, BBC, BBCArabic. You can read all these content right from your desktop, you just need to pick them all at once, then disconnect and you have all that great content of those giant sites. How Great and How Amazing ?! It saves a great deal of time and tedious revisiting of your most visited sites.

You need to check out syndic8 or blogdigger to get more RSS Links.

RSS Search Engines
Feedster is the most known Search Engine to search RSS documents. Feedster Spiders all RSS documents and blogs and returns you with a full list including all the search results returned, you will be amazed when you find its interface is so similar to Google! You can also view Blogdigger, it's one of the best weblogs search engine.

What RSS stands for ?!! , It’s not an Easy Question !!
Netscape created the original format of RSS 0.90. It was the first version and it stands for “RDF Site Summary” or “Rich Site Summary”, NetScape provided RSS 0.90 for use in their portal activities, but others saw more effective uses. Userland Software adopted the idea and was the first company to use RSS commercially in Weblog products.

RSS 0.90 was based on RDF “Resource Description Framework” provided by W3C, RDF is pretty complex, so a new version was launched by Userland and they called it RSS 0.91. It was RDF-Free. Userland launched many new versions including 0.92, 0.93, and 0.94, and to prove their trend of simplicity, they made RSS stands for “ Really Simple Syndication “

A group of developers appeared and re-adopted RDF version (RSS 0.90), because they saw that it’s more flexible than RSS Userland versions, After some hard work they published RSS 1.0 which stands for “RDF Site Summary”, and back to versions and shortcuts MESS !!

Userland saw that RSS 1.0 was about to displace RSS 0.94 so they published a new version RSS 2.0 which stands for “Really Simple Syndication” as well. That is where we stand today !

So we have two threads; one that’s RDF based and the other of Userland which is not based on RDF. So you have two different technologies sharing the same name with different version numbers. I was first confused because I believed that RSS 2.0 is the sequel of RSS 1.0. What a mess !!

ATOM , a new syndication age
A new group of developers didn’t like that mess so they are working in a project called Atom. It’s a new syndication technology . Atom solves almost all the RSS problems including versions mess, extensible by anybody ( just like XML is ), Atom technology is similar to both RSS 1.0 and RSS 2.0, it promises to provide a standard Archiving format, and some APIs which is called ( Atom API ). Some websites are now emitting their syndications in Atom format. Most RSS Aggregators support all RSS versions, there are a few ones supporting Atom.

BlogRoll , OPML Language
What do you think of the importance of Playlist to Multimedia files (items), ain’t it great? BlogRoll is simply a collection of blog feeds gathered together, most bloggers provide blogrolls for their visitors. This provides a great deal of links to the visitor who’s interested in a specific topic. I see it as a playlist. Outline Processor Markup Language is an XML ( again ) based language to write blogRolls. Most of Aggregators today can read BlogRolls and extracts the contained feeds easily. How great!

As you have seen RSS and Blogs are state of art, every website should provide a link to RSS or ATOM feed. You should get a news reader, install it, add channels to your news reader, be updated. In part two of this article I will show you have to consume the others' feeds and show items in your website, and you will be able to create your own site feed, we will also analyze the schema of RSS versions and Atom.

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves.

About the Author: Kareem Shaker

Kareem Shaker is a .NET Architect , He's been working with VC++ and VB since version 4.0 ; Kareem has been working on design and development of many business applications , And he's now spending most of his time working with .NET Framework 1.x and 2.0 using both VB.NET and C# , Kareem has been giving some technical sessions targeted to .NET technologies, One of the technolgoies that I have passion to is EAI, I spend most of my times working on BizTalk Server 2006, I like to share knowledge and to interact with geeks around, Blogging is one of my hobbies, my blog is http://CairoCafe.Blogspot.com

Occupation: Architect
Location: United Arab Emirates


Google News gets RSS and Atom feeds

By Graeme Wearden
Special to CNET News.com

Google has introduced RSS and Atom feeds for its popular Google News aggregation service.

RSS, or Really Simple Syndication, and Atom are XML-based document formats that alert Internet users to the latest articles or postings on their favorite Web sites via a single feed reader, which can be integrated into an e-mail application or Web browser.

Google News users can now subscribe to get an RSS or Atom feed from any of seven key subject areas designated by Google. They can also create customized RSS news feeds or see the results of any Google News alerts they have set up.

Google News, which aggregates links to the latest news stories on thousands of Web sites, is available in 22 versions for different audiences around the world. The RSS and Atom feeds are being initially offered on just six versions--U.S., Canada, Australia, New Zealand, U.K. and India.

"We've launched this service because we've had a lot of requests from Google users to support RSS and Atom," a Google UK spokeswoman explained.

In April 2004, U.K. programmer Julian Bond said he received a cease-and-desist notice from Google after creating his own feed that scraped headlines off Google News. These headlines were then displayed on another Web site, called Ecademy.

Google UK declined to comment on this matter. (Google representatives in the U.S. have instituted a policy of not talking with CNET News.com reporters until July 2006 in response to privacy issues raised by a previous story.)

But Bond said Tuesday that the company took action because the results of his RSS feed from Google were being fed into online RSS aggregators and then republished on the Web.

"I was really irritated about Google not offering RSS feeds from (Google) News search. So I wrote a scraper that did the search and generated an RSS feed from it," explained Bond, who added that he welcomed the news that Google was now offering RSS and Atom feeds.

"It will mean that I can retire my scraper and stop having to maintain it every time Google changes their page layout," Bond said.

Graeme Wearden of ZDNet UK reported from London.