Search engines, like Google, Bing, Yahoo, etc. exist to discover, understand, and organize the internet’s content. Their main goal is to offer the most relevant results to searchers for whatever they’re searching for.
If you want to show up when people search for what you’re offering, your content has to be visible to search engines—sounds simple, right? Getting found is the most important piece of the SEO pie. If search engines can’t find your site, there’s no hope for you to show up in the search engine results page.
Search engines have three main functions: crawling, indexing, and ranking. We’ll go over all three, right now.
Search Engine Crawling
Search engine crawling is the discovery phase of the entire process. The search engine sends out a team of robots called crawlers or spiders (gross, we know) to find new and updated content. The content they find can be in a wide variety of formats, like a webpage, an image, a video, or a PDF. No matter what the format is, this is how content is found.
A Google crawler will start by grabbing a few web pages, and then following the links on those pages to find new URLs. They’ll travel along this path of links to find new content that they can add to their index, called Caffeine. Caffeine is a huge database of discovered URLs—the information inside is later retrieved whenever a searcher is looking for the information within that content.
Search Engine Index
This phase is fairly self-explanatory. The search engine processes and stores the information they find while crawling—as long as they deem it good enough and relevant enough to give to searchers.
Search Engine Ranking
When someone searches for something, search engines go through their index for the most relevant content—relevancy is how they decide how early you’ll show up in the results. They order the content in the hopes of answering the searcher’s question, or query. This ordering of search results by relevance is called ranking.
Generally, the higher a website is ranked, the more relevant the search engine thinks that site is to the question at hand. It is possible to block search engines from your website—or parts of it—or to tell search engines to avoid storing certain pages in their index. There can be reasons for doing this, but if you want your content to be found by searchers, you need to make sure the crawlers can crawl it. Otherwise, it’s essentially invisible.
Why Google? What About the Other Search Engines?
Most people already know that Google has the largest market share—but what about optimizing for Bing, Yahoo, and other search engines? Why are we only talking about Google? Even though there are over 30 major web search engines, the SEO community really only cares about Google. Here’s why:
Google is where the vast majority of people search the web—and we mean vast majority. If we include Google Images, Google Maps, and YouTube (owned by Google), more than 90% of web searches happen on Google. That’s 20 times more than Bing and Yahoo put together. So yeah, Google is pretty important.
How Do You Know if Google Can Find Your Pages?
If you already have a website, your first step should be seeing how many of your pages are in Google’s index. This will give you some great insight into whether or not Google is finding the pages you want it to—and ignoring the ones you don’t.
Head to Google right now (but come back!) and type in “site:yourdomain.com” into the search bar. This will give you results that Google has in its index for your specific website. Pay attention to the top of the page, where it says, “About ___ results.” That should give you an approximate estimation of which pages are indexed on your site, and how they’re showing up in search results.
You can also monitor and use the Index Coverage Report in Google Search Console—you can sign up for a free account if you don’t already have one. Here, you can submit sitemaps and monitor how many submitted pages have actually been added to Google’s index, among other things.
If you tragically find that you aren’t showing up anywhere in the search results, here are a few common reasons why:
- Your site is brand spankin’ new, so it hasn’t been crawled yet—these things take time!
- Your site isn’t linked to any external websites—Google likes that a lot.
- Your site’s navigation makes it hard for a robot to crawl it—make sure your website is easy to navigate.
- Your site contains some basic code called “crawler directives” that’s blocking search engines—get that outta there.
- Your site has been penalized by Google for spammy tactics—don’t be a spamster.
Google is pretty complicated. Why don’t you grab a cup of coffee, and we’ll handle it from here. Get in touch with Uptick today.
