This article is part of an SEO series from WooRank. Thank you for supporting the partners who make SitePoint possible.
Organic search traffic is vital to any commercial website: Almost half of online shoppers begin their shopping process with a search engine, and a third of e-commerce traffic comes from search results. Even if you’re a brick and mortar shop, you’re likely reliant on organic traffic since half of mobile searches result in a store visit within a day of the search. All that adds up to one fact: if you’re trying to make money with your website, you need organic traffic. And how do you bring in that traffic? SEO.
SEO is typically viewed as the realm of marketers and writers. However, developers have a large role to play as well. If sites aren’t built correctly, search engines could struggle, or even fail entirely, to find and index pages. One false move with your robots.txt file, for example, could prevent your entire site from showing up in Google search results.
That’s why we’ve put together this 9-point checklist to help developers build sites in a way that’s optimized to rank highly in search results.
Crawling and Indexing
Since the purpose of SEO is to appear in search results for your target audience, one of the most important considerations when creating a site is getting crawled and indexed. The easiest way to get indexed is to submit your site directly to Google and Bing. Use Google Search Console to submit your URL to Google. This doesn’t require a Google Search Console account, but if you do have one, you can use the Fetch as Google tool in the Crawl section. After Googlebot successfully fetches your site, click the "Submit to index" button.
Submitting your site to Bing requires a Bing Webmaster Tools account.
XML Sitemaps
A basic description of XML sitemaps is a list of every URL on your site, stored as a text file in your site’s root directory. In reality, there’s a little bit more to them than that. Yes, they list every URL on your site (or at least the URL for every page you want crawled and indexed), but they also list extra information about each page and serve an important SEO function. Search engines use the information in sitemaps to crawl sites more intelligently and efficiently so they won’t waste their crawl budget on unimportant or unchanged content. When done correctly, your basic sitemap looks like this:
<?xml version="1.0" encoding="UTF-8”?>
<urlset xmlns="http://ift.tt/xwbjRF” xmlns:xhtml=”http://ift.tt/2bTDKEE;
<url>
<loc>http://ift.tt/2byyU2M;
<lastmod>2016-8-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.9</priority>
<xhtml:link rel="alternate” hreflang=”fr” href=”http://ift.tt/2bTCP7v;
</url>
What does all that mean? Here’s a synopsis:
<urlset>
: This tells crawlers that the sitemap is starting and ending.
<url>
: Denotes the beginning and end of each URL entry in the sitemap.
<loc>
: This defines the URL of the page. While the rest of the attributes found in the <url>
tag are optional, <loc>
required.
<lastmod>
: The date, in YYYY-MM-DD format, the page was updated or modified.
<changefreq>
: This indicates how frequently you update the page, which will help search engines decide how often to crawl it to make sure they’re indexing the freshest content. You might be tempted to lie to increase your crawl frequency, but don’t. If search engines see <changefreq>
doesn’t jive with the actual change frequency, they’ll just ignore this parameter.
<priority>
: Sets the priority of the page in relation to the rest of the site. Valid values range from 0.0 to 1.0, from least to most important. Use this tag to help search engines crawl your site more intelligently. Note that this only tells crawlers how important your pages are compared to your other pages. It does not affect how your pages are compared to other sites.
<xhtml:link>
: This tag points to alternate versions of the page. In this example it indicates the French version of https://www.example.com.
Sitemaps aren’t a ranking signal, but they help search engines find all your pages and content, which makes it easier for you rank well.
If you don’t want to write your own sitemap, there are plenty of tools out there that can help you create one. Once you have your XML sitemap, validate and submit it using Google Search Console. You can also submit your sitemap to Bing via Webmaster Tools. Make sure you fix any errors so you don’t wind up impeding your site’s indexing.
Robots.txt
Like XML sitemaps, robots.txt files are plain text files stored in the root directory of your site, and help crawlers navigate your site. The file contains lines of code that specify which user agents have access to which files, file types or folders. The code is broken up into blocks, with one user agent line per section. Basic robots.txt code looks like this:
User-agent: *
Disallow:
User-agent: googlebot
Disallow: *.ppt$
The asterisk (*) is used as a wild card. In the user agent line, the wild card represents all bots. In a disallow line, it represents the URL up to a specified point. In our example above, our robots.txt disallows Googlebot from crawling pages that end with a PowerPoint file extension — the $ denotes the end of the URL.
You can block bots from crawling your entire site by using a slash in the disallow line like this:
User-agent: *
Disallow: /
It’s good practice to disallow all robots from accessing the entire server when you’re building, redesigning or migrating your site. However, you have to be sure to restore access once you’re done, or your shiny new site won’t get indexed.
Use Google Search Console to test your robots.txt file for syntax errors or other problems.
One problem with the robots.txt file is that it won’t stop search engines from following external links to your site, so disallowed pages could still wind up indexed. Add an extra layer of protection to individual pages using the robots meta tag:
Continue reading %9-Point Technical SEO Checklist for Developers%
by Sam Gooch via SitePoint