Making Your Spider Outperform Google

| No Comments

Even the best search engine can fail. But when it falls short of finding the most commonly requested information, whether on an external web site or an intranet, that's a real problem.

That's why I found Rich Wiggins' Internet Librarian presentation so interesting. He discussed in detail how he enhanced the search engine at Michigan State University in a way that no doubt improves the user experience.

First, he analyzed the search logs to identify the most common search terms, creating what he calls an "accidental thesaurus" from those keywords and phrases. He then stored them in a database, mapping them to the URLs that were most likely to provide the information being sought.

When a search is run on the MSU site, the "best bets" database is queried first, then the standard search engine, in this case, Google. The best bets, if any, appear at the top of the page followed by the standard search engine results. It is similar way that sponsored links are displayed on a commercial search engine. Rich can add search terms and/or URLs at any time to tweak the database for better search results.

He also created a browsable directory from the accidental thesaurus, providing another easy to use finding tool.

According to Rich, the best bets database typically works well for common searches, while Google's relevancy ranking works well for uncommon searches. So the searcher gets the best of both worlds.

You can try it out for yourself at the University web site.

Now why didn't I think of that?

Leave a comment

About this Entry

This page contains a single entry by Cindy published on November 24, 2003 11:51 AM.

Amazon Can Do It. So Can Westlaw was the previous entry in this blog.

Elearning as a KM Application is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Subscribe

Enter your Email


Powered by FeedBlitz


Google Reader or Homepage
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Add to Technorati Favorites!
Add to netvibes