Monday, September 6, 2010

"You mean there are spiders crawling inside my computer?!"

That was my first thought as I learned about how search engines work. Yes, a little naive but you must understand that I never wondered about how web information was sorted or accumulated.  All I ever knew was that I had to type something in and then take the time to sort out the best response.  Truth is that "spiders" are used by search engines to compile databases.  These figurative spiders search the web and index pages which are then stored in a database.  The spiders regularly return to the sites that they have indexed and make updates if needed. Therefore, a search engines is made up of only the sites that have been indexed and not the entire web.  Search Engines will vary in speed, size, pages indexed, and their ranking criteria which will in turn affect your searches.  A search engine will always have results for your searches but that may present a problem.  For example, if the search criteria or key words are not specific you may be faced with thousands of responses, many of which are not relevant or reliable.  Examples of search engines are: Google (Individual) and Gigablast (partnered with subject directories).

Here is a website that helped me visualize what "spiders" do in a search engine:

http://www.webconfs.com/search-engine-spider-simulator.php

http://www.webconfs.com/spider-view-article-9.php

No comments:

Post a Comment