For my Search Engine Analysis I will be analyzing WebCrawler, a metasearch engine.
WebCrawler® brings users the top search results from Google, Yahoo!, Bing, Ask and other popular search engines. WebCrawler also provides multimedia results, including images, video, news, and local information. WebCrawler is a registered trademark of InfoSpace, Inc.
WebCrawler.com utilizes metasearch technology to compile results from many of the Web's major search properties, delivering more relevant and comprehensive results every time you search. By accessing multiple search engines for each query, WebCrawler.com provides you with a richer and more relevant spectrum of results than you would from using any single search engine.
Metasearch technology takes results from the leading search engines, eliminates the duplicates and delivers you the most comprehensive set of results. You benefit by obtaining a quicker, more accurate set of results to your query.
The search results you receive are a combination of the top commercial (sponsored advertising) and non-commercial (algorithmic) results from the most popular search engines on the Web. WebCrawler’s metasearch technology combines the top ranking search results from each of the separate search engines based on your specific query. The blend of sponsored and non-sponsored results for a given search term depends on the nature of the term, but sponsored results are always clearly identified with a "Sponsored" or similar designation like this:
WebCrawler.com web search is designed to identify the intent of a user's search and results are generally listed in order of relevance as they are received from the various search engine providers. The specific ranking and mixture of sponsored and non-sponsored results will depend on the nature of your search. For example, if you are searching for information about the prices of digital cameras and enter the term "digital camera prices," the results will generally include more sponsored results and include commercial Web pages containing information on the prices businesses are offering cameras for sale. If, on the other hand, you are searching for academic or general research purposes and enter the term "digital camera technology," the results will be weighted more toward articles, information and other non-commercial results about the technology behind digital cameras.
On this page I will describe how to use WebCrawler.
Preferences
Let us start with looking at how this site allows us to customize our search according to our preferences. On the home page, next to the search textbox, click on the Preferences link.
Once you reach the preferences oage you have the option of customizing your search in 3 ways, as shown in the screenshot below:
- SEARCH FILTER allows you to filter potentially explicit content from search results. It has 3 options: no filter, moderate filter and heavy filter. The Moderate filter removes as much potential explicit content from Multimedia Search Results as possible. Web search results are not screened.
- RECENT SEARCHES allows you to turn on/off recent search history.
- LANGUAGE SELECTION allows you to filter results based on language.
Frequently Asked Questions
You can find Frequently Asked Questions by clicking on the Help link below the Preferences link.
Simple Web Search
For our web search, lets start with a simple one. Since this wiki is about the Asset Management Industry, I will type Asset Management into the search box.
When you enter a search, metasearch technology automatically organizes results into categories based on words and phrases contained in the results. As you can see on the right side of the page under "Are you looking for?" the search provides different categories of related searches.
Also, WebCrawler.com provides a history of your 15 most recent searches. On the right side of the page, below the "Are you looking for" list, these is a "Recent Searches" list that gives you a history of your most recent searches.
Image Search
For our image search, I will search for rolling stones. The result page is displayed as follows:
Along with the "Are you looking for?" and "Recent Searches" lists on the right side of the page, on the the top bar there is also a "View Results by" option. As can be seen in the following snapshot, the images can be listed according to Relevance or Search Engine.
Video Search
For the video search example, I will use rolling stones again.
The Video Search is a simple search tool that just provides basic search results. It does not provide any other options by which we could further filter our search.
This pages describes my evaluation of this search engine.
Generating Results
- This search engine displays 20 searches per page (an option which can be changed - view the Demonstration Tab on this page to understand how).
- According to me there is no particular format to the search results. If you see the results you will see that the sponsored Ads are all over the page and not in a specific location. This I think is mainly because, being a metasearch engine deriving its search results from other other big search engines, it does not format the way the search results are displayed.
- Finally, I don't think this page has any paid results as it takes its results from other search engines. Thus, in case of a paid search results on the search engines from which WebCrawler derives its searches, it provides a statement next to the URL of the search result such as [Found on Ads by Google ].
On average, this search engine provides around 50 to 60 search results. Rarely does it ever generate search results of more than 100. According to me this is because being a metasearch engine, WebCrawler tries to filter out the repetitive searches from the search engines that it derives its results from.
For example:
- when I searched for Warren Buffet, it gave me only 58 results.
- when i searched for Swine Flu. it gave me only 78 results.
For each search result, it also provides the search engine from which it got the result.
Also, like other search engines, WebCrawler provides a number of similar searches on the right corner of the site which is quite helpful. Also, below that it gives you a list of your most recent searches which is another plus point considering a time when you want to go back and forth in your search results.
Exploring results
Relevance Vs Retrieved
Each result has a one line summary on what the site contains, similar to any other search engine. After exploring the various search results, I found that the search results were quite relevant. The only search results that I thought were not so relevant were some of the results that were Ads on other search engines (For instance: the [Found on Ads by Google ] found next to the URL of the search result).
Going back to the examples of:
- Warren Buffet, I found that 17 searches our of the 20 on the first page were relevant.
- Swine Flu. I found that 19 searches out of the 20 on the first page were relevant.
Special Query Terms
Since this is a metasearch engine that takes search results from popular searches and filters them based on relevance, all the different Query Terms from different search engines work on this site. Thus terms like AND, OR, " ", -, intitle:, inurl:, etc. all work on this search engine.
For instance:
- If I search for "warren buffet" AND biography, I get 46 results out of which 13 searches out of the 20 on the first page are relevant.
- On the other hand, if I search for "warren buffet" OR biography, I get a surprising total of 44 search results (surprising because this search should have had more than the previous number of search results of 46). This anomaly may be because of the way these search engines retrieve their results from other search engines. According to me, it basically takes what it thinks are the most relevant searches from other search engines when this specific query would be in that search engine. This, I don't think this search runs on actual logic.
- If I type intitle:"warren buffet", it gives me a total of 61 search results, of which 15 out of the 20 on the first page are relevant.
Search Target
Images
For image searches, on average WebCrawler displays only 40 results. Though the results are so few, again they are quite relevant.
For instance:
- While search for images of Warren Buffet, WebCrawler displayed 40 search results. Out of the 18 search results on the first page, all results were relevant.
- Similarly, while searching for images of Barack Obama, WebCrawler displayed 40 search results. Out of the 18 search results on the first page, all results were relevant.
- Changing the type of search a little, while searching for images of the sity of Mumbai, WebCrawler displayed 40 search results again. Out of the 18 search results on the first page, all results were relevant.
Unfortunately, WebCrawler does not allow you to customize your Image Search much. The only thing that can be changed is the way by which the results are sorted. This option is at the top of the search results page and allows you to sort results on the basis of:
- Relevance
- Search Engine
Videos
For Video Searches, WebCrawler provides the main sources of its search at the top of the page and also provides the source of each video below the short summary of the video. For most video results, the video icon plays the video without sound which is a nice interactive feature.
On average, the video search displays 60 search results. The results don't seem to be great. Could probably get much better results through other search engines.
For instance:
- While searching for videos on Warren Buffet, WebCrawler displayed 60 search results. Out of the 10 search results on the first page, only 4 search results were relevant.
- While searching for videos on Mumbai, WebCrawler displayed 57 search results. Out of the 10 search results on the first page, all were relevant and mostly related to news.
- While searching for videos on Rolling Stones, WebCrawler displayed 60 search results. Out of the 10 search results on the first page, 8 search results were relevant and mostly related to their music videos.
News
The WebCrawler News search provides 30 results on average. The framework of this search is nothing great. You can't customize your news search on the basis of time like other news search engines. For instance, you can't search for news over the last week or month or year; it just provides the most recent news that it gathers from other search engine sources. The sources of the search are provided on the top right hand corner and also after the summary of each search result.
For instance:
- While searching for news results on Warren Buffet, WebCrawler displayed 30 search results. Out of the 20 search results on the first page, 14 search results were relevant.
- While searching for news results on Mumbai, WebCrawler displayed 30 search results. Out of the 20 search results on the first page, all search results were relevant.
- While searching for news results on ING Investment Management, WebCrawler displayed 30 search results. Out of the 20 search results on the first page, only 9 search results were relevant.
Yellow Pages
The WebCrawler Yellow Pages is quite a cool tool. It allows you to search by phone number or by using an advanced search in which you can look for something on the basis of Category or Keyword, Business Name, Address and City, State or Zip. Once you search for something. it provides you with a map as well.
This tool is quite useful as it allows you to customize your search to a large extent extent and also has many resources that can make your search easier.
- Customizing Search (options located at the top of the search results page)
- It allows you to sort on the basis of relevance, A-Z, Distance and Ratings.
- It allows you to decide over what range you want to search for in terms of distance; for instance all areas, 10 miles, 15miles, 20 miles, etc..
- And it allows you to change the number of results per page
- Resources (located on the left of the search results page)
- Best Match: Gives you good matches to your search query, making it a little more specific.
- Products: Provides a list of products that go with your search query
- Services: Provides a list of services that go with your search query
- Brands: Provides a list of brands that go with your search query
For instance:
- When I searched for Restaurants in the city of Ann Arbor, the search results provided me with a list of 426 restaurants, their locations on the map, a list of specific types of restaurants under the Best Match resource and so on.
- Similarly, when i searched for Salons in Ann Arbor, the search results provided me with a list of 134 salons, their locations on the map and so on.
What I realized about the Yellow Pages search was that it is a separate search engine on its own. It is not completely a part of WebCrawler.com. This is becuase the URL of this page does not begin with webcrawler.com.
Opacity
WebCrawler.com is quite transparent with its search results. This is because being a metasearch engine WebCrawler gets its search results from other search engines. Thus, in their search resuts, WebCrawler highlights which search engine it got its results form.
Frequency of updates
Again, being a metasearch engine WebCrawler does n ot really update its search results itself. According to my experience, every time the search engines from which WebCrawler retrieves its search results updates their search results, WebCrawler's search results automatiocally get updated too. Thus, WebCrawler.com does not really have to put in much effort to make their search engine perform as they are merely taking their search results from other search engines who put in all the hard work.
Since WebCrawler is a metasearch engine that is an integrated search for multiple search engines, I will compare and contrast it with other metasearch engines that are integrated searches for multiple search engines.
I will be comparing the different search engines in the following ways:
- I will be comparing the search results of the other search engines listed below on the following search queries:
- I will also be comparing the different facilities that these sites offer in comparison with WebCrawler.
Here are the metasearch engines with which I will be comparing WebCrawler.
Info.com (http://www.info.com)
Info.com is a search engine that provides great results. As you can see from a metasearch experiment that Professor Scott Moore has conducted, Info.com provides better search results than Google.
When searching for:
- ing investment management, Info provided 90 search results. From the first 10 search results all were relevant. On the other hand, while using WebCrawler, from the first 10 search results, ing investment management returned only 7 results were relevant.
- bny mellon asset management, Info provided 90 search results. From the first 10 search results all were relevant. While using WebCrawler, from the first 10 search results, bny mellon asset management returned all relevant results.
While looking at the different search options that WebCrawler and Info have to offer, Info has a much better selection. Not only does it cover all the search options such as image, yellow page and video searches that WebCrawler provides, it also provides research, shopping, job, flights, hotels and more as categories to search from. Also, Info provides a search wizard from which you can perform custom searches.
Infospace.com (http://www.infospace.com)
Infospace is also one of the good metasearch engines around. As you would have seen on the About tab of this *search-engine-analysis, WebCrawler is a registered trademark of As you can see from the [*http://www.howcanifindit.com/ex09:20-metasearch metasearch experiment that Professor Scott Moore did, Infospace did quite well in their search results as compared to Google.
When searching for:
- ing investment management, Info provided 60 rsearch esults. From the first 10 search results all were relevant. On the other hand, while using WebCrawler, from the first 10 search results, ing investment management returned only 7 results were relevant.
- bny mellon asset management, Info provides 60 search results. From the first 10 search results 9 were relevant. While using WebCrawler, from the first 10 search results, bny mellon asset management returned all relevant results.
Being the owner of WebCrawler, Infospace has better search options.
- It allows you to look at search results from particular website by selecting the search website that you want to look at from the top menu bar.
- It also has a great advanced search option that you can use which is one of the major setbacks of WebCrawler
URL.com (http://www.url.com)
URL.com uses a "bionic search engine" that lets them collaborate and improve upon a foundation of search results gathered from the best web search engines, including Yahoo!, Bing™, and Google. They are also quite a good metasearch engine site. As you can see from the metasearch experiment that Professor Scott Moore conducted, URL.com also did quite well (like Infospace) in their search results as compared to Google.
When searching for:
- ing investment management, Info provides 21 search results. From the first 10 search results all were relevant. On the other hand, while using WebCrawler, from the first 10 search results, ing investment management returned only 7 results were relevant.
- bny mellon asset management, Info provides 23 search results. From the first 10 search results again all were relevant. While using WebCrawler, from the first 10 search results, bny mellon asset management returned all relevant results.
Thus URL.com provides much better results than [http://www.webcrawler.com]. Also, they have better search options as well:
- Along with the Web, they allow you to search for Lyrics, Games and **Layouts as well.
- They allow you to rate the search results which is a good way of tracking which results are more relevant to people.
- They give you rankings of the various search results on the basis of the search engine from which they are taken from. (look at the left side of the screen; you can see the rankings under Google, Yahoo and Bing).
You should have an overview/executive summary that summarizes your thoughts on this search engine and your recommendation for using it (or not).
Even though WebCrawler seems like a simple search engine that provides quite relevant search results on average, it does not have much to offer.
- The results are too few and thus does not provide enough variety for the user.
- WebCrawler speaks about its Advanced Search Option and how it helps in searches on their Help page. They say:
You can also refine your search within each category by using the Advanced Search option. For example, when searching for an image, you can filter for specific image color, size, or file format. Video searches can be filtered for playtime or file format. And news searches can be filtered by news category, location, date or source.
Though they mention this, these options do not actually exist. Thus they promise something that they don't have which made me dislike the search engine.
- Also, the search engine is quite slow. It takes quite a lot of time for it to load the results page.
- Finally, WebCrawler does not have many options by which a user can customize his/her search except for while searching in the Yellow Pages. Thus it does not make the search experience anything special. It does not help the user to specifically search for something.
My Recommendation
My recommendation would be to use other specialized search engines. For instance, use Google for web search and news search or Bing for image search. This is because:
- These search engines provide a large variety of search results from which the user can choose
- They allow users to customize their searches as much as possible thus personalizing the search experience.
- They are much faster than WebCrawler
- The lack of an advanced search option is one of the major problems as well
If you really want to explore metasearch engines I would advise you to use Info, Infospace and URL.com. If you want to know why, visit the Comparison to Other Search Engines Tab.
The only search that I will recommend people to try on WebCrawler would be the Yellow Pages search.
