0 Spider

A search engine spider is a computer program that goes through websites, page by page, gathering information about what's on each page. That information goes into the search engine's index, to help improve search results.
Spiders discover new web pages via links from one page to another. That's true whether a link on a web page goes to another page on the same site, or a page on a different site.
A spider is also called a crawler or robot. The word "spider" refers, of course, to the fact that websites exist within the World Wide Web. The word "robot" refers to the fact that the crawling activity is automated.