Crawl budget problems are common and mostly invisible

On smaller sites, crawl budget rarely causes trouble. But once a site crosses a few thousand URLs — especially with faceted navigation, large product catalogues, or duplicate URL patterns — crawl waste becomes a real issue. Pages stop getting indexed, updates take weeks to reflect in search results, and it is not obvious why.

This webinar is run by Liezel van Niekerk, an SEO architect who has worked with retail and publishing platforms across southern Africa. Her background is in backend development, which means her approach to crawl issues is structural rather than surface-level. She does not just point at Screaming Frog output — she explains why the problem exists at the architecture level.

What the session addresses

The first half covers how Googlebot allocates crawl rate and crawl demand, and how those two things combine to determine crawl budget. A lot of practitioners confuse crawl rate limits with crawl budget — this session clears that up with documented Google guidance and practical examples.

The second half moves into solutions: parameter handling in Google Search Console, internal link auditing to reduce crawl path noise, log file analysis to see actual Googlebot behaviour, and sitemap structuring for large sites. There is a worked example using a fictional e-commerce site with 85,000 product URLs.

Tools used during the session

Screaming Frog SEO Spider, Google Search Console, and a basic log file analysis workflow using free tools. Participants do not need paid software to follow along, though having Screaming Frog installed is recommended.

The session is recorded and available for 45 days. A log file analysis template and URL parameter worksheet are included with registration.

Webinar Programme

Module 1 — Crawl budget explained
Crawl rate, crawl demand, and the difference between the two. How site size, server response times, and link popularity factor in.
Module 2 — Diagnosing crawl waste
Identifying problematic URL patterns, session IDs, faceted navigation, and orphaned pages. Using GSC coverage reports and log files as evidence.
Module 3 — Fixing the architecture
Parameter handling in GSC, internal link consolidation, sitemap segmentation strategies, and when to use noindex versus canonical versus disallow.
Module 4 — Worked example
Step-by-step review of a 85,000-URL e-commerce site structure. Identifying waste, prioritising fixes, estimating crawl recovery time.
Module 5 — Tools walkthrough
Live demo of Screaming Frog crawl configuration and log file import. Basic log analysis using Google Sheets.
What is included with registration

45-day recording access, log file analysis template in Google Sheets, URL parameter handling worksheet, and a recommended reading list on crawl architecture

Was this article helpful?

Thanks for your feedback — it helps us write better content.