Glossary: Crawling a Website
Glossary: Crawling a Website Crawling a Website is what a search engine (such as Google)does when it reads an index of a website, usually with a small program called a spider or a bot. Spiders (or Bots) are the software that the search engine uses to crawl your site.