Most SEO people in the business have heard of tools like Semrush and Ahrefs, which, most of the time, get the job done.
But there comes a time when we need tools that we can download and use advanced features that are not available online because of browser limitations and heavy pricing.
However, Screaming Frog, on the other hand, provides a feature that is easy to implement and execute, such as list mode.
Unlike most SEO crawlers with a crawl limit, Screaming Frog’s List Mode lets you enter a list of URLs to crawl, making it perfect for checking a specific set of pages.
Another feature of Screaming Frog is that it can render and crawl AJAX-heavy websites, which is a must-have feature for modern SEO as more and more websites rely on Javascript for content delivery as fast as possible when accessing content.
There are many features we will be discussing in this article, as well as simple tricks you didn’t know. So, let’s get started!
A. What is Screaming Frog?
Screaming Frog is a desktop software that helps you scan websites and gather information about SEO and website structure.
It’s made by a UK-based technical SEO agency, also named Screaming Frog. They launched the tool in 2010, calling it the “Screaming Frog SEO Spider,” but most people just say “Screaming Frog.”
With Screaming Frog, you can scan individual web pages on a site to get detailed data. It works a lot like the web crawlers operated by search-engines like Google and Bing.
It examines the HTML of a page and identifies elements like headings, images, meta descriptions, page titles, and structured data.
The software then follows links within the HTML to find other pages, analyzing them in the same way.
All this information is displayed in Screaming Frog’s interface, which provides SEO experts with a lot of useful details to help fix issues that could harm a website’s search engine rankings, such as broken links, slow loading times, or duplicate content.
B. 8 Most Useful Screaming Frog Features
1. Look for Broken Links
Broken links can harm your SEO and annoy your visitors. Here’s how to use Screaming Frog SEO to find them all.
Navigate to the Response Codes tab after doing a crawl. Proceed to filter the results for Client Error (4XX). A list of all 404 and 403 issues on your website will appear as a result.
Click the Inlinks button at the bottom of Screaming Frog after licking on a mistake. This will let you know which page the broken link was on, so you can go there and fix it.
By selecting Bulk Create > Response Codes > Client Error (4XX) Inlinks, you can also create a report on broken links for your website.
2. Check Page Titles and Meta Descriptions
With Screaming Frog, you can check the page titles and meta descriptions for your website to ensure they’re unique and relevant and give a clear snapshot of your content for search engines.
To do this, navigate to the Page Titles or Meta Descriptions tabs. There, you’ll find the specific details for each URL on your site.
These tabs let you filter titles and descriptions that need to be added, duplicated, too long, or too short.
You can also export this information to a spreadsheet for further analysis or sharing with your team.
2. The Request Data API Function
To obtain historical data from Analytics, Search Console, and SEO tools used when doing a site audit, linking Google Analytics, Search Console, Moz, PageSpeed Insights, Ahrefs, or any other data source will save you a ton of time and provide you with more insights.
It’s rather typical, though, to discover these tools after completing the initial crawl or to forget to connect them to Screaming Frog.
In certain situations, you may actually collect data from APIs and match it to the relevant URL in the current crawl by using Screaming Frog’s “Request API Data” button.
4. Analyze Internal and External Links
Screaming Frog is great for reviewing the internal and external links on your website, which is crucial for SEO. Links are important for helping search engines understand your site’s structure and for keeping visitors engaged.
Internal links help visitors navigate your site and make it more comfortable for search engines to index your content.
External links, which point to other websites, add credibility and value to your site while also passing “link juice” that can influence search engine rankings.
Analyze internal links with Screaming Frog:
- Go to the “Internal” report. You might want to filter the results to only show HTML files (pages).
- Click on the URL you’re interested in.
- At the bottom of the page, click the “Inlinks” tab to see which other pages on your site link to this URL. The “Outlinks” tab shows the internal links on this page, including anchor text and whether they’re following or nofollow.
- Export the data to a spreadsheet if you need a closer look or want to share it with your team.
For external links:
- Go to the “External” report to see all the links that point to sites outside of yours.
- Click on a URL to get detailed information.
- The “Inlinks” tab shows which external sites link to this page. The “Outlinks” tab indicates which external URLs you are promoting on this page.
- You can also export this data for further analysis.
With this information, you can identify opportunities to improve your link structure, address broken links, and confirm that both internal and external links are working to boost your site’s SEO.
5. Custom Extraction
Using Screaming Frog’s custom extraction capability to scrape websites in search of more data is another tactic.
For instance, Screaming Frog does not automatically gather information on H3 tags; however, custom extraction may be used to do this.
In addition, you may detect a variety of site-related data by using the custom extraction tool. It may be used to determine which cookies are active, whether social network sharing buttons are present on each page, and whether any connections in your documents are broken.
6. Duplicate Content
Screaming Frog can help you locate perfect or almost exact content for your website.
Duplicate material might confuse search engines, but it won’t destroy your SEO rankings. Select the Content tab and apply the Exact and Near Duplicates filters.
Once duplicate material has been found. You may analyze these pages and add fresh keywords to provide your visitors with original content.
7. Check just “Clean” HTML pages
If you wish to improve your sites’ information, build a list of URLs, or do any other job that only requires HTML pages, you may instruct Screaming Frog to crawl these pages and disregard other file types.
Go to Configuration > Spider, uncheck all of the boxes in the Resource Links area, and then start the crawl as normal.
However, if you have a large number of URLs containing arguments, you might want to go a little further and use a RegEx expression to omit all parameters or a specific syntax like:
- To exclude all URLs with a question mark, use \? (the question mark must be backsplashes for this to function).
- You may also exclude a single parameter, such as a price filter, by specifying its syntax. For example: \?price.
You may discover more Exclude ideas in Screaming Frog’s settings guide.
8. Explore Advanced Features
Unlock the full potential of Screaming Frog by delving into these advanced features:
- Schedule Audits: Set up audits to run automatically, either as one-time events or at regular intervals. Simply navigate to File > Scheduling to configure your crawl settings.
- Integration with External Tools: Enhance Screaming Frog’s capabilities by integrating with other tools. For example, you can integrate PageSpeed Insights for insights on page load speed or Google Analytics for data on page visits and bounce rates. Access this feature via Configuration > API Access.
- JavaScript Crawling: Dive deep into your website’s JavaScript elements, including AJAX content, to identify any dependencies or blocked elements. Navigate to the JavaScript tab to access this information.
- Site Comparison: Compare two websites within Screaming Frog to assess technical changes and track improvements. Head to File > Crawls and select the reports you wish to compare.
- XML Sitemap and robots.txt Generation: Seamlessly generate XML sitemaps and robots.txt files for your website. Access these features through Sitemaps > XML Sitemap or Configuration > robots.txt, and follow the guided instructions.
- Visualizations: Gain valuable insights into your website’s structure with visually engaging representations. Explore the Visualizations menu, select your preferred visualization type, and customize it to suit your needs.
C. Summarizing 8 Screaming Frog Features and Tricks You Didn’t Know
Screaming Frog is a sophisticated tool for website research and SEO improvement, with extensive features that go beyond simple web crawling.
This easy-to-skim article highlighted other less visible but extremely important capabilities, such as scheduling audits, connecting external tools, scanning JavaScript parts, comparing websites, and making infographics.
These capabilities allow business owners to acquire deeper insights into their websites, enhance technical SEO, and optimize content for higher search engine rankings.
Whether you’re a seasoned SEO expert or new to the area, learning these features will help you get the most out of Screaming Frog and guarantee your website runs optimally.
Suggested read: Best Backlink Checker Tool: Boost Your SEO
D. Common FAQs on 8 Screaming Frog Features and Tricks You Didn’t Know
How does Screaming Frog manage JavaScript content?
Screaming Frog can’t crawl JavaScript components and AJAX material. This feature allows you to determine whether pages use JavaScript and whether any components are banned. You may find this information on the JavaScript tab.
Can I compare two site crawls using Screaming Frog?
Yes, you may compare two crawls to track changes and evaluate progress. Navigate to File > Crawls, choose the reports you wish to compare, and see the differences.
What are some visualizations I can create with Screaming Frog?
Screaming Frog offers various visualizations to represent your website’s structure and link relationships. Visit the Visualizations menu, choose your preferred type, and customize it as needed to get a clear picture of your site’s architecture.
How can I exclude specific URLs or parameters when crawling with Screaming Frog?
To exclude specific URLs or parameters, go to Configuration > Spider and adjust the settings to ignore certain file types or apply regular expressions (RegEx) to filter out URLs with specific patterns, such as those containing question marks or other parameters.
