I was invited to speak in a Clubhouse room dedicated to Web Dev Ideas by Ketevan Social. Questions regarding the good old Sitemap emerged, and there appears to be a massive misunderstanding among many web designers.
What Is A Sitemap, and how does it work?
It is an XML file that outlines the hierarchy & URLs of the pages within your domain for search engines to reference to in order to index your website. Index is a nerdy term for including them in potential search results.
What is The (Google)Bot, and what does it do?
The Googlebot is a computerized explorer (other search engines have their own too) that discovers content on the web. It crawls through your page through links (like a human would, for the most part).
The bots do this without a Sitemap:
Moreover, having a page URL on a sitemap does not guarantee that it will be indexed – only Google’s algorithm decides what to index or ignore.
“Google does not index your pages just because you asked nicely.” – Michael Cottam at MOZ.com
Find Google’s Motive – Better Results, Better Traffic, More $$$
One only needs to reverse engineer the motivation and purpose behind Google’s business model. Their main source of revenue is advertising which comes from high volume of search engine usage. They want everybody to use Google to search the web, and they do that by delivering the most relevant results.
Google will rank higher websites that have relevant information that is easy to find. If it’s linked well for humans to easily navigate to, so will Google – and it will assign those links a higher relevance score when determining the quality of your website.
We haven’t used sitemaps since 2007, and our SEO campaigns always work.
Even Google’s developer documentation suggests that sitemaps are mostly unnecessary barring a few exceptions. In fact, the SEO campaigns we run at Brixwork Real Estate Marketing Inc. have returned incredible results on websites with tens of thousands of pages, and we haven’t used sitemaps since 2007!
To Actually Get Indexed Faster & Rank:
We All Like Easy & Quick (don’t we?)
Going back to the point of how Google’s crawlers work, it’s critical to understand that links are what made the internet what it is – it connects pages to allow discovery. Make them easy for Google.
- Proper navigation structure and clear links can dramatically improve indexing.
- Pre-set links of internal search results such as real estate listings by type/price/location can help bring more listings to the forefront to be indexed better & faster.
- This most often will increase Google’s crawl frequency & volume of your domain.
It is a Popularity Contest After All
Inbound links from other websites & social media are a critical method for crawlers to find your pages. It is also a way for it to determine how relevant & reputable your domain is in general. It’s a win-win.
- Our experiments show that sharing early on Facebook, Twitter & LinkedIn speeds up indexing, and improves ranking.
- Our Realtors are encouraged to share their new listing URLs before the same listings are found on thousands of other websites through MLS® Reciprocity. REW.ca, Zillow.ca, Realtor.ca are competing will show them. Early & plentiful sharing worked to keep their listings on page 1 often.
Backlinks from other domains or blogs is extremely powerful, although you may need professional SEO services to achieve this – and this has longer term implications than just social shares.
And never forget – content is king when it comes to long-term SEO!
From proper meta title & relevant meta descriptions to well-denoted ALT tags on images, content on your website is ultimately the best weapon you have when it comes to long-term Google rankings.
As Always, There Are Exceptions Where You May Need A Sitemap:
- If your website has archives of articles & historic data that must be searched, but is simply too much content to show within the surface level pages and require users to click on and on to find, sitemaps will help Google find the older pages. In most consumer-facing websites, this is not an issue. If you’re running a legal case archive, or an academic journal/resource, a dynamic sitemap generator will work well for you.
- If much of your website needs to be hidden from indexes for privacy or security reasons (while remaining publicly accessible), but you want to selectively showcase certain URLs only, then a sitemap may work for you.