Search the world

.. and find the haystack with needle in it

As we all know searching does not always mean finding. It seems easy to find a needle in a haystack once you have found the haystack. The problem is finding the haystack.

Google is nowadays the leader in search on the Internet. Although there many more. Search around the ever increasing blogosphere has only recently been addressed by Google through the beta service Blog Search.. But there are others, see below.

• Technorati
• DayPop
• BlogDex
• PopDex

Now there are even more ways to get completely lost in cyberspace.

Google Sitemaps

.. observations [2]

In my previous post I was already enthusiastic about my sitemap being pickup by Google twice within the first day. I can now say that that is not because I submitted it twice.

This morning I noticed that it was again picked up last night without me saying that it should do so. And again this morning at 9 am!. To me this proves observations by others that the Google Sitemaps get picked up twice daily. So this part of the system is working the way I hoped it would.

Now for the real purpose of the Sitemaps: getting your pages crawled. The Crawlers haven't yet crawled my site using the Sitemap. This can take some time before it happens. The predominant belief is that a fresh crawl of the Internet is done every couple of weeks or month. (BTW do you believe in Google?)

So, since I only uploaded my sitemap two days ago, it could take up to a month before my hidden pages are discovered.

Meanwhile, I sit and wait, check my logs and get on with my work.

Google Sitemaps

.. observations

In my previous post I was already enthousiastic about my sitemap being pickup by Google within a couple of hours. I can now say that it gets better than that.

Much better, even.

I just noticed that it has been picked up again this morning sometime between 8 and 9 am. So it's my guess that the sitemap is downloaded by Google on a very frequent basis.

Or maybe it is because I submitted it twice. Anyhow, it is on the move.

Google Sitemaps

... the start of an experiment

I have been reading up on the Google Sitemaps over the past few week. A Google sitemap is an XML file with all the URL to all the pages of your website and you can add some extra data to it like: last update date, change frequency. In theory it should make Google crawl the site more efficiently. Although it is still in Beta I have decided to give a go.

To see if the Google Sitemaps offer some benefit I have devised a little test project. I have used a free tool to generate a sitemap and have added to that a URL to a page that is not linked to from any of my pages. On that page is a link to a further page that is not linked to.

If I can see that Google spiders these two loose pages then I know that there is some benefit from the Google Sitemaps.

I have uploaded the sitemap.xml on my site and have signaled Google around 7 pm last night. This morning I saw that the sitemap was downloaded by Google about one and a half, two hours later!

As an added bonus I got a list of pages that Google tried to crawl on my site but could no longer find. Yep, I had deleted those. So, I can now figure out what to do about that. Probably just ignore it. Read more about these handy sitemap statistics.

For now I am already seeing some interesting effects.

Can't wait to see more results.

The Golden Day has come

... and moved quietly into oblivion

The much anticipated magical day of September 13 arrived yesterday and we did not have a celebration or a brilliant fireworks display. No corks of Formula One size champagne bottles flying through the office and the contents ending down the neck of the management.

No, nothing like that.

Our project has not yet completed and we are still working on it. Getting close though, but still it is not clear if we have built all required functionality as no one dares to take the responsibility to speak on that. So it still a wait and see experience.

And we still struggle on.