vinnymo

Screaming Frog's paid license is about $250 per year and is definitely one of the most useful tools for SEOs. I built my own crawler with Claude last week, im gonna give you the prompt for free. 😜 Here's what it does: -> Crawls every page on your site -> Flags broken links (4xx/5xx) -> Catches missing/duplicate title tags & meta descriptions -> Finds thin content and missing H1s -> Maps redirect chains -> Exports it all to CSV Self-hosted. No license. No crawl limits. Paste this into Claude Code and you're done THE PROMPT: "Build me a Python SEO site crawler I can run locally. Requirements: Takes a starting URL and crawls all internal pages on the same domain (respect robots.txt, configurable max depth and crawl delay) For each page, record: URL, status code, title tag, meta description, H1(s), word count, indexability (noindex check), and canonical tag Flag issues: 4xx/5xx errors, missing/duplicate titles, missing/duplicate meta descriptions, missing H1, multiple H1s, thin content (<300 words), and redirect chains Handle errors gracefully and don't re-crawl the same URL twice Show a live progress counter in the terminal Export two CSVs: a full page inventory and an issues-only report Use requests + BeautifulSoup, keep it single-file, and include a requirements.txt and a short README on how to run it" Tweak the thresholds and checks to fit your workflow but yea that's most of what you need to have your own sick crawler. Enjoy fam 💜