Googlebot

Googlebot is the name of the automated software — a crawler or "spider" — that Google uses to discover and read pages across the web. It is the first step in how your site shows up in Google: before a page can be indexed or ranked, Googlebot has to find and fetch it.

Googlebot works by following links. It visits pages it already knows about, reads the links on them, and adds any new URLs it discovers to a queue of pages to crawl next. There are versions for desktop and mobile, and today Google primarily uses the mobile crawler (mobile-first indexing), which is why a good mobile experience matters so much.

Your goal is to make Googlebot's job easy and to avoid accidentally getting in its way. The most common self-inflicted disaster is blocking Googlebot by mistake — a leftover Disallow: / in robots.txt or a stray noindex tag from a site build can hide your entire site from Google.

To help Googlebot find and read everything that matters:

• Link to your important pages from your navigation and homepage — orphaned pages may never be found.
• Submit an XML sitemap so Google has a clean list of your URLs.
• Do not block resources Googlebot needs to render your page properly.
• Keep your site fast and mobile-friendly so the crawler can process it efficiently.

Googlebot only handles the finding and fetching. What happens next — indexing and ranking — depends on the quality of what it finds. The full picture is in our how Google Search works guide.