Implementing a Google Mini

.. if only everything in life was this simple

The Google Mini implementation together with a redesign (or rather re-alignment) of our website will be put in production on the 24th October. It took longer than this post suggests because of other higher priority projects that were implemented in the past few weeks. But hey! It is coming at last.

As briefly mentioned in my previous post I am working on the incorporation of a Google Mini appliance in the Univé website. I will go into this a little bit more detailed.

Side stepping

I am a sort of a DIY-person. I like to do things myself. That means most of the time that things take longer before it is finished, but when it is finished the feeling of pride and self esteem pays for that. Also along the way of any of these DIY projects I get to stages that I have to do something new with some new tools or materials. Most of these have words on the packages like "can be assembled in 5 minutes" and over time I have learned that that is the moment to be aware, to be very aware of pit falls. Usually I have picked something to use it in a slightly different way or want to use it inside a small space so it is impossible to use a screw driver to fasten it.

I think that these problems are directly related to my lack of skills, because usually (after some tears and sweat from me and cursing the people who ever designed the stuff) I can use it for the purpose that I had intended. I think of myself as a reasonably well DIY person.

The same applies in my IT skills. I sometimes wander about and pick the wrong route at first, but usually I get stuff working, again after some tears and sweat from me and cursing the people who ever designed the stuff.

But these marketing words: "up and running in NO time" still put me in a special mode. When something can not fail I know it will.

Stepping back

So, when we had wheeled in the Google Mini appliance I was ready for weeks of administering and tweaking to finally get some search results on my screen that vaguely would like what we would want to see.

I know I have been very enthusiastic about the stuff coming from Google in the past months, years and maybe it seems like I am a blind follower of the company at Mountain View, but I am not. But I am of the opinion that have created some very good stuff in recent years. And when they haven't created it themselves they give it a twist by presenting it for free, like Google Analytics.

Back to the topic

After these lengthy opening paragraphs I am now returning to the subject of this post: implementing a Google Mini.

During the build of our new website we had dropped search as one of the deliverables for Go-Live. It was postponed to the Point One Release. You can have a nice long discussion about that, but don't bother we did as well and management decided to drop it and so we did.

We had acquired a Google Mini to facilitate the site search and had it hanging in a server rack for a couple of months gathering dust, but once we were ready for launch I was given time to explore its potential. I must admit, that I was a bit afraid to have a stab at it. To me it was a black box (although it's happily blue) and I did not see an opening. And misplacing the manuals was another hurdle we had to take and after some Googling on Google we found a way to download the stuff and have a go at it. Then a colleague opened his drawer and said that he had secured the manuals. Sigh!

Initial setup

The first things you have to do is some basic setup like giving a IP address and an administrator password. This was done very simple using a laptop connected directly to the box. Then using the web interface it was really easy to configure the crawling and after the first crawl see the first results through the familiar Googlish interface. Tweaking a bit of XSLT brought out the company colors and logo and I was ready to be enthusiastic again. This machine would provide an easy site search for our website. And so we moved on.

Integration into the site

Well, we did not want to give visitors direct access to the Google Mini, but we wanted to really integrate search into the site context. We have added a search box to our menu bar and created a new search container or template. Communication with the Google Mini is managed through a Web Service.

Since, the Google Mini already came with an API it was relatively simple to set this all up. Cutting some corners we decided to stay with a plain search screen and leave out the advanced bits for the moment. This meant that we needed a simple Web Service method that would only take a string as the search query. Added to that we wanted have navigation from one page of search results to the other and we another method was needed for accepting the (more complex) query with page indication and the like. This second method was also used for the spelling suggestions and the keymatch results (aka Sponsored Links).

Communication with the Google Mini

The Google Mini API is very simple in nature. All one needs to do is build a long URL with query parameters and you are provided with XML string that holds the outcome of the search request.

The XML string is then returned to the site and there it is loaded into an XmlDocument and then parsed using XPath queries and presented to the visitor in simple XHTML.

Presentation to the user

First of all, we did not want to reinvent yet another better wheel. We feel that Google does a great job of presenting search results. But being a web developer I also know that the code used to produces the list of results is not exactly what we call modern or semantic web design.

So, we liked what it looks like on the outside and decided to change the inside.

Show in the screen shot is part of what the page will look like. The similarity with Google search result pages should be obvious. We even have space for AdSense ;-)

The search results are presented in a <dl>, with the linked title in a <dt> and the snippet from the page in a <dd>. This seems to me a reasonable microformat for this purpose. Keeping the pages lightweight was and is one of our primary goals.

Looking back

When I now look back at this project (just before it is moved into the final test cycle and put online) I think that we did a great job. And the Google Mini is a great way of providing site search. Implementation went smooth and the only challenges were again in getting the .NET Framework work as we want it to work.

The Google Mini once configured does not need any real maintenance. If the machine should lose it's complete settings and database then setting it up fresh and let it crawl our site can be done within an hour. The Google Mini really lives up to its expectations. It was great fun to do.

Looking forward

After putting it in production we have a very strong tool for our content writers to see how a search engines sees our site and how Google presents the results. I plan to work with them to make the site even better than it is already.

In that way the Google Mini will work as a benefit by providing search to the visitors, but also as a tool to improve the search engine performance of our site. Not bad for a little blue black box.

Tags:

• Google mini
• corporate seach
• site search
• xhtml
• xml
• webservice
• microformat

A new Standard in screen resolution?

.. smaller or bigger?

A sequel to this article has been made. In that I discuss the current standard screen width you could design for. Based on data from the first months of 2007.

In a post today Tony Patton over on TechRepublic discusses Determining a standard screen resolution for your application. He has some point to keep in mind when designing a web site.

• There is a general trend towards bigger monitors with higher resolutions. Multiple screens are close to becoming a common household item as well.
• Never ever use your own screen resolution as the standard.
• Nor the monitors of your colleagues. Both will be probably more state of the art than the old CRT or cheapo flat screen from Joe Average.
• Screen resolution does not equal the viewable area or the browser window size. Most users will not have their browsers maximized on screen.
• It is good to look at visitor statistics, but be sure to sample the right audience. Taking the numbers from a site largely visited by developers will not do when the site you are designing is aiming at pensioners.
• Also keep accessibility issues in mind. The disabled users can use screen readers with or low resolution monitors with odd metrics. You should test, test, test.
• You can use JavaScript to establish the viewport size and serve the client a fitting design.

All in all, a not all too shocking article, but a good summing up of things to keep in mind.

Monitors are becoming smaller, not larger!

In a comment TechRepublic member Wombat Ron then adds the increasing use of handheld and smartphones that have much smaller screens. He predicts that the 640*480 resolution could be coming back into the charts. So, it is difficult to say whether we are moving towards bigger or smaller screens.

Separating content and styling

I think Ron is right, although I think these could be handled by using the mediatype'handheld' to serve special styling through a dedicated CSS file. This is where separation comes back in its full power. With a good design of the structure of your site (the html) you will keep that lean and you can then just as easily serve that to handhelds and adjust the styling so that it will fit on a small screen.

What is wrong in the article?

One thing struck me as quite odd in the article was the example code that was in the article. This code looks at the size of the window to determine what to present to the customer. See code.

<SCRIPT language="JavaScript">
<!--
if ((screen.width >= 1024) && (screen.height >= 768)) {
window.location="high_resolution_version.html";
} else {
window.location="low_resolution_version.html";
}
//-->
</SCRIPT>

Here the principle of separation is missed completely. The users is redirected to another page and I think that it would be more then enough to serve another CSS file.

Separation of structure and styling gives the versatile option of changing the style while still leaving the structure intact. At our new website we have a 'normal' version and an 'impaired' version. The only difference is that other stylesheets are delivered to the visitor. The html stays the same. (Mind you: this does not make the site accessible, there are still plenty of issues that we will tackle soon.)

Right now, I am working on changing the layout of the site so that it strechted from an 800 pixel wide design to a 1024 pixel wide design. I this is all relatively painful as I only need to make changes to a couple of CSS files and stretch a few background images.

Afterburner

What is the most odd really in the 'resolution business' is that apparently designs are made for 1024*768 or 800*600. I hardly ever design the height of a web page. The width is a valid parameter but the height of the page is usually determined by the content and as a designer you have hardly any influence on that unless it is your own site.

I usually design for a certain screen width. Also I have a preference for fluid or liquid designs.

Google Sitemaps: overview of benefits

.. looking back with pleasure

Sometime in September 2005 Google put out the Google Sitemaps product. To help website owners all over the world to inform Google of the pages of their websites. Being a website owner, designer and developer I immediately jumped on the wagon and tried it out. I started an experiment to try the effectiveness out and see if it did really do what I hoped it would do: make my new content appear in the Google index more quickly than before.

My experiment

In a couple of posts ( 1, 2 & 3 ) I outlined the experiment I did with Google Sitemaps on my site.

What was the setup?

I created a sitemap of my website. In this XML file all the interlinked pages appear. I added two pages to my website:

1. a page (A) that I added to my sitemap.xml but that was never linked from any the other pages on my website
2. another page (B) that I did not add to my sitemap.xml, but it was linked to from page A.

Both pages had legitimate content and also link back to other pages of my website. Actually, both were to be part of my website but I merely have omitted putting links on the other pages pointing to these new pages. They are in essence orphans. But could also be seen as landing pages.

The experiment was to see if the Google crawler would visit these two pages. The first directly from the sitemap.xml and the second through crawling the links on the first page.

So what happened?

After having submitted the sitemap.xml I was filled with joy when I discovered that Google scanned the file within 24 hours. Moreover it appeared to come by twice a day. That did not mean that the pages appeared in the Google search results immediately, but still it gave me hope that they would after the next crawl.

Image via Wikipedia

So, I kept checking my website logs to see if any of the secret pages (A or B) would pop up in my requested pages list. And checked again and checked again.

Now for the real purpose of the Sitemaps: getting your pages crawled. The Crawl process can take some time before it happens. The predominant belief is that a fresh crawl of the Web is done every couple of weeks or once a month.

So, after uploading my sitemap I expected it could take up to a month before my hidden pages were discovered. But after a couple of days the first secret page A was crawled. Bang! The sitemap.xml really showed its purpose. The file was indeed used by the Google crawler to crawl pages.

The second page did not show up immediately. So apparently, the links on the page were not yet crawled. They were probably queued to be crawled in a next crawl. Later they indeed did show up and so that part of the process worked as well.

The experiment was successful and has convinced me that Google Sitemaps really add something to a website.

Added value

A bonus of the Google Sitemap system is that a website master also has the possibility to view reports on the last crawl results. This can be achieved by placing an html file with a name provided by Google to verify that you can manage the site. Google then provides you with a list of failed pages. Pages that no longer exist or have another error.

These statistics have been extended over the last few months and now show also the most used search strings and also the search queries with the most clicks. There is more like a robot.txt checker and error reports. All very valuable for a web site owner. Read more about these handy sitemap statistics.

Conclusion of the experiment

By adding a Google Sitemaps file to your website (and keeping it up to date!) you can ensure that new pages are crawled at the next scheduled crawl. If you have a popular CMS there is probably already a plugin available to create and maintain a sitemap. I can recommend it to any site owner. It is worth the effort. For me it is a little bit of work because I still use static html pages on my site.

More goodies

Some time ago after the experiment more goodies were added. What extras are hidden in the Google Sitemaps console?

• Crawl statistics: Pages successfully crawled, pages blocked by robots.txt, pages that generated HTTP errors or were unreachable.
• The PageRank distribution within your site.
• Various indexing stats (pages indexed, etc.)

If you, for a completely obscure reason, do not wish to create a Google Sitemap, you can rest assured. You do not need to have Google Sitemap to be able to use this functionality. All you have to do is create an empty HTML file and upload to your site to verify that you own the site. And then you start leveraging the benefits. For free.

Google Sitemaps turns itself into an absolutely awesome troubleshooting tool for all webmasters.

Experimentation is fun, but real life implementation is better

This interesting experiment shows that Google Sitemaps is doing what it is supposed to do: making pages noted by the Google crawlers.

As said we launched our new website on Tuesday 30 May and implemented Google Sitemaps with it. I had registered the sitemap with Google a couple of days before the launch so it was trying to download the then still not existing sitemap. On launch day around 6 AM our time the sitemap was downloaded by Google. That was the first possible moment less than two hours after the site went live.

Success #1: new pages are already in the index

Two days later we see already many pages in the Google Index. When I did a search on the first 100 pages changed in the last three months on Google I found that of these only 12 are still pointing to our old site. My guess was that these are there only because we had some problems setting the rewrite rules for the old asp files. That was not entirely correct as a week later we still see old pages, but those are different ones.

The Google index now already returns new pages only days after going live. Without the Google Sitemap this would taken much, much longer.

Success #2: we have a check on what goes wrong

Thanks to the statistics that we receive from Google we can now track the old, no longer existing pages that are being crawled. We can see errors popping up and if necessary we can further tweak our web application or web server to gracefully handle these.

Conclusion

Implementing Google Sitemaps has given us already huge advantages in the first week of going live. By having the Google Sitemaps ready at go-live we have been able to have Google find our new pages and content.

I would strongly recommend implementing Google Sitemaps with any web project so the pages will appear as quickly as possible in the index. Even though we had to implement the Google Sitemaps ourselves within our site framework it was not the greatest of challenges we had.

Many of the (open source) content management systems nowadays provide Google Sitemaps out of the box or through a plugin. So, there is hardly any reason not to implement them.

Update (4th September 2008)

The Google Sitemaps protocol has in the meantime been adopted by more search engines and is now als an integral part of the Robots.txt. So, it's nowadays even more important to use the protocol.

Better still: It's now an open standard.

Tags:

• google sitemaps
• google
• sitemaps

Related articles by Zemanta

• Automated Google Sitemaps Generators

Google Sitemaps rule!

.. yoohoo!

Yep, we are definitely doing things right over here!

As said we launched our new website on Tuesday and implemented Google Sitemaps in it. We have registered the sitemap and Tuesday morning around 6 o'clock our time the sitemap was downloaded by Google.

Now on Thursday some two days later we see already many pages in the Google Index. When I do a search on the first 100 pages changed in the last three months on Google I find that of these only 12 are still pointing to our old site. My guess is that these are there only because we had some problems setting the rewrite rules for the old asp files.

This rocks!

Tags:

• google
• google sitemap

Separation is the way to go

.. in modern web design

Maybe I should again apologize for the ''Building websites is fun post I did yesterday. I was maybe cheering to hard for a goal I scored as part of a great team. But once again, I could not have done without the rest of the team.

And it is just such a good feeling that things that you have put together prove to be working in real life. I guess that's why I became a developer in the first place: creating things. That's probably also why I keep up this blog.

Col. John "Hannibal" Smith (George Peppard)

I love it when a plan comes together.

What's all the fuss about?

So, what's all this fuss about then? What did I contribute to the project that works like it should? Why cheering for just doing the job I was hired for? Shouldn't things always work when I create them?

Uhm? Yeah, well, right. Ok, yes, sorry again. I just made another impression that successful development work is something to cheer about loudly and long just because it so rarely happens. Apologies to all colleagues around the globe.

OK, now stop apologizing and tell the handful readers what it was that I did.

What did I do?

In the process of building a new website for an insurance company I was assigned to make sure that the XHTML and CSS coming out of the site was as optimal as possible. More or less self assigned as I thought to have gathered some knowledge in that area. The team was already underway with several Proofs of Concepts of early phases of the site framework and I joined in time to take up my task King of XHTML and CSS. And happily so.

Just before last summer the graphic designs came to us and they were delivered mainly as Photoshop files and some early versions of cut up html. I then started to work on tweaking these to be lighter in markup by stripping out unnecessary bits and making sure stuff showed like they should in the two major browsers of that moment: Internet Explorer 6 and Firefox. I also checked to see if things were reasonable in Opera and on a Mac. But that was only on an occasional basis.

Plan of attack

The old website was a highly successful site (in its market) and was developed some years ago. Since then many tweaks and enhancements were added. The site ran in frames and the layout was largely table based. Good old fashioned Old School web design. In essence nothing wrong with that: it produced very good revenue.

For the new web site we opted to become more up to date. We moved from the classic ASP to ASP.NET and on the client side we opted for XHTML. As configuration was set high over old fashion programming we did not go with the standard .aspx files (in fact we have only one .aspx file). A complete framework was built that has several types of containers for the different types of pages. In the CMS one can pick the right container (template) for the page and the information will then be displayed correctly. There are quite a few webservices running in the background that glued all things together.

My role in this was taking the different page designs and translate these into XHTML and CSS which the other developers could then implement in containers and components of the framework. My most important focus was separating the structure of the page and the styling. The third aspect of a web page: behavior was already being handled by external JavaScript files.

Using the principle of separation of structure (semantics), style and behavior we were able to keep the site very lightweight for our visitors. The users don't for all the clever stuff we did in the background on our servers and load balancers. They just want to get the information quickly.

Structure & semantics

As far as was possible within the framework and the design the html was stripped to the minimum so it was still semantically correct and enough structure to allow the styling to make up the design. This was all implemented in various containers and components that show different types of content and controls. That gave a few restrictions and added a bit of extra fat, but the overhead was small and made for easier development of the server-side framework.

Styling

At first we had a huge CSS file. Later that was cut up into several as not all containers use all styling. We added a bit of intelligence to the containers and let them load extra stylesheets when needed.

We also added a print stylesheet that simply hides all unnecessary stuff like menus and also adds the URL to links through the content() styling. This proofed to be quite a simple approach and gave satisfying results really quick.

"Bad vision" styling

Then there was a little bit more work. The site needed also to be viewable by people with some kind of disability. We added a button for "bad vision" that would link in other stylesheets. The "normal" stylesheets had pixel based fonts and as you know that does not really resize very well in Internet Explorer and also the used background images and all measurements were not relative to font size. Resizing the font in your browser will lead to a broken page layout pretty quickly.

So everything needed to be em based. Then the layout was changed from fixed width to fluid so the browser window would be used to the full at any resolution and not just 800 by 600 the site was designed for originally.

We also dropped a lot of the background images to make it easier to read and added a "skip links" link. (This "skip links" link is hidden in the normal vision site which the more I think about that was maybe not the best decision.)

This special impaired version of the stylesheet did involve some work, but it was done without any changes to the html. The only change is that containers load an extra set of alternative stylesheets.

Benefits

The benefits of separating structure, style and behavior have become really apparent during the development of the different styles. It was very easy to update and also very flexible. I also did some research on bandwidth savings and a saving from between 10 to 20 % is made. That is taking into account the images that are part of the content. During the research I found that these images are nowhere near optimized for the web. SO if we could improve on that I guess that the savings could go up to around 30%.

Conclusion

By separating the different aspects of the web pages we have reduced on time spent making changes and will also save on bandwidth usage. So overall it is the right thing to do, not because it is hot or trendy, but because it saves both time and money.

Go ahead and have look at the new website.

Tags:

• separation
• structure
• semantics
• styling
• behavior

Building websites is fun

.. when you do it the right way

"Disclaimer"

I know this blog entry is close to self-indulgence, but it is just a self inflicted boost for my self-confidence. Heck, every now and then you have to tickle yourselves.

OWI Van Lieshout

If you want to laugh, you have to tickle yourselves.

A new website

The website that we launched yesterday is a big cooperative effort from the entire department and each one of us has put in their respective special skills into action. This has resulted in a completely fresh site that is ready for the next couple of years.

We have moved away from an all ASP site with frames and table-based layout to fully fledged .NET site with our own framework that spits out clean XHTML and CSS.

This last bit has been my main focus in the project. I must say that I am proud of what has come out of it. There is room for improvement, as always. On top of that one learns new things every day and these new things have pointed me to little bits of code that can be improved.

Let's make things better

Now we have finally finished our deliverable we are fixing the inevitable small issues and are already looking at improvements. During the build we compiled lists of these and now we can at last start making them.

In my area of XHTML and CSS I have started working changing the layout. The site was designed with 800 by 600 pixels as an already overdue requirement. So, right now I am tweaking the site's CSS to make it a 1024 by 768 pixel design. Much to my enjoyment I have already one of the templates working in Firefox and I only have hack a bit to make things work in Internet Explorer.

A couple of things make feel very good today:

• The speed at which I have made the changes
• I have also made some changes to the implementation (background colors and images) are now on more logical elements.
• By changing the implementation I have reduced the number of background images and made some considerably smaller.
• To top things of I have not changed one bit of the XHTML. Meaning that I can drop the changed CSS files and background images on the server and implement the new size!

That's why I am pretty proud of myself.

Tags:

• website
• fun
• separation
• structure
• style
• behavior

Yes we went live

.. and without any major glitches

Yesterday was the the biggest milestone in the project that had this weird end. The website that we have been building for the last year and in the time before that exploring in several Proofs of Concept. It had been a struggle at times with new technology, getting specifications sorted out, having a deadline before we had any specifications, struggles within our department, new people, struggling with a not very performing CMS and going through burning hoops to get it to perform etc. etc. You know, the usual stuff that happens in any major project. For some reasons they always happen.

Yesterday we had breakfast with champagne to celebrate the event and after that we were allowed to "enjoy the achievement a bit", which meant just staying in and doing as little as we did over the last couple of weeks.

Only the usual little problems popped up that kept some of us from really "enjoying the achievement a bit".

I think we pulled off quite an impressive job, but also think that we have just set a first step on a long road ahead.

Tags:

• web site
• milestone
• live