Screaming Frog is a crawling software that I use for auditing websites. It allows me to crawl website URLs, fetch, analyze and audit technical and on-page SEO elements. This can make auditing websites much easier since I can pull in a lot of website data into one area and analyze it.
Download the free version of Screaming Frog here: https://www.screamingfrog.co.uk/seo-spider/
The free version allows you to crawl up to 500 pages in each crawl. So for most small sites, this should be sufficient.
I use the paid version but if you are just getting started, the free version is perfect to test things out.
In Screaming Frog, past the URL of the website in the top bar and click “Start”.
You can see how much is left to crawl in the bottom right corner. Once it’s at 100% the crawl is done.
HEADS UP: Screaming Frog will crawl the entire site so if the site has a poor hosting service there’s a chance it could take the site down temporarily. Or if the site has a lot of traffic going to it, running Screaming Frog on it could cause users to see error messages temporarily. “By default the SEO Spider crawls at 5 threads, to not overload servers.” but I wanted to share this info with you just incase.
Was this article helpful?