How to understand data from a Screaming Frog crawl

You may have been told that Screaming Frog is a great tool for SEO, so you downloaded the tool, crawled a website, and now you’ve got a sheet full of data but you’re not really sure what to do next. Well in this guide, we’ll talk you through that report tab by tab, so you can audit your website, check performance, and make recommendations based on the output.

What is Screaming Frog?

Screaming Frog is an SEO tool that replicates what search engine bots do when they crawl a website. You can crawl up to 500 URLs for free, or buy a licence to crawl an unlimited amount of URLs.

In order to rank websites Google has bots/machines that crawl web pages and store information about that website which is then used for the search engine’s index. Things these crawlers are checking includes:

That pages are working (200 not 404s)
The content of pages (length, tags, quality of content)
Websites structure

In order to audit what the search engine bots are seeing, you can replicate it with Screaming Frog’s crawling tool.

How to read your Screaming Frog report?

The first step in understanding how to audit your website is to understand the data the Screaming Frog crawl is displaying.

Address: this is the list of URLs that have been crawled. They will all start with https://
Content Type: this tells you what type of content the page is e.g. webpage (html), styling sheets (CSS), image assets or javascript files
Status Code: this tells you if the page is working (200), a redirect (301, 302) or not found (401). Most of your pages should be a status code 200.
Status: the status should match the status code e.g. 200 = OK, 301 = permanent redirect, 302 = temporary redirect etc..
Indexability: this simply tells you if your pages are indexable or not e.g. Google can index them in their search index.
Indexability status: if your page is non-indexable, this column will tell you why. For example it might be because it’s redirected to another page, or it’s canonicalised.
Title 1: your title 1 is your meta title. This is the title that will show up in the search results. Every page should have a custom and optimised title. You can see the same data for your meta descriptions.
Title length: to avoid truncation, your meta title should be around 69 characters long.
Title pixel width: this is similar to length, but breaks done the width in pixels rather than characters.
H1: every page should also have a unique and optimised H1. This is your page header and should be a short summary of the page. The report also shows this data for your H2s.
Meta Robots: meta robots are tags that you add to your site to tell Google bots what to do or not do, e.g. index, follow, no follow
X-robots tag: these are HTTP header directives for the URL
Canonical Link Element: canonical links are listed here for each page (to see more on Canonicals, check out our canonical tag deep dive)
Word Count: this is the number of words on the page
Crawl Depth: this is how deep your page is linked to from the main nav. The deeper the depth, the harder it is for Google bots to find it
Inlinks: number of internal links (from the same domain)
Outlinks: number of external links (going to a different, external domain)

What to look for in your screaming frog report

Once you have crawled a website using the Screaming Frog too, you should export the data into an excel or Google sheet. Once you have done this, and you have it in a format that is easily readable, there are a few things that you want to look into in order to audit the SEO performance of your site:

1. Status of pages

The first thing you want to check in your screaming frog report is the status of your pages. All pages that you want to be indexed should be a status 200 and indexable.

If you have pages that are 301s, or 404s, double check that they should be. It’s okay to have redirects and 404s, just make sure they are intentional.

2. Meta titles and descriptions

The next important thing to check in your Screaming Frog report is the presence and length of your meta titles and meta descriptions. Ideally every page should have a meta title and a meta description. The report will also show you the length, so you can easily see if some are too short or too long.

Ideally, to avoid truncation on the search engine results page, your meta titles should be ideally 69 characters long, and your meta description should be 150-160 characters long.

3. Headers (H1s and H2s)

Every page on your site should have a H1. Your H1 carries the most weight in terms of SEO value on your page. This is one of the main ways that a search engine understands what your page is about and how to index it. Your H1 should be a concise summary of what your page is about, and should contain the primary keyword you are targeting. If your page does not include a H1 tag, you should add one.

You should also check H2s, which are the second most important header on your page. Your H2 tags should include terms that are relevant to the main topic of the page.

Header tags should be used in order, from H1 (unique) to H2, H3 etc.. etc..

4. Length of content

Screaming Frog shows you the length of pages so you can see at a top level if some pages are shorter than others and need extra content. Some pages can be shorter in length than others like contact pages, but Google can penalise pages that are too light on content. For example your blogs shouldn’t be too thin on the content side.

Screaming Frog is a great tool to have in your SEO tool belt. We hope this guide helps you understand how Screaming Frog works and what to look for in your reports when auditing your site. If you would like some assistance or advice on how to go further with your SEO reporting, reach out to one of our team today.