Free Tool · Spider Mode

Site Crawler — a browser-based Screaming Frog alternative

Spider your site link by link or paste a URL list. Every page is scored against 25 SEO checks — broken links, redirect chains, duplicate titles, orphan pages, crawl depth, snippet quality, schema, and more. No install, no license — up to 1,000 pages free.

Crawl setup

Spider mode starts at your URL and follows every internal link. List mode crawls only the URLs you paste. Keep this tab open while it runs.

URL rewriting — clean up discovered links before crawling (optional)

Remove specific tracking parameters, or apply regex find/replace rules to every discovered URL.

CDNs — treat other subdomains as internal (optional)

Add CDN subdomain+subfolder combos (one per line, e.g. cdn.example.com/assets/) to be treated as Internal URLs even though they're on a different host than the crawl start URL.

Include — crawl only URLs matching these patterns (optional)

One regex pattern per line. If any patterns are set, a URL must match at least one to be crawled — separate from the simple path filter above.

🔒 Auditing a staging or password-protected site? Read this first

This crawler runs on our server, so it can't get past login walls, Cloudflare Access, or VPN-only staging sites — the same way no outside visitor could. Browser security also means no page, including this one, can reach across and silently crawl a different domain's staging site for you — that request has to originate from the staging site's own tab to carry your login. This is the same protection that stops any website from quietly reading your bank account in the background.

The closest thing to "one click, no copy-paste": drag this bookmarklet into your bookmarks bar once. Afterwards, whenever you're on a staging site, just click it — it runs the same 28-check engine using your own logged-in session, no console, no pasting.

📌 BP Staging Auditor (drag me →) ← drag this to your bookmarks bar
  1. Drag the button above into your bookmarks bar (one-time setup)
  2. Log into the staging site normally in your browser
  3. Click the bookmark — the audit runs right there in that tab
  4. A CSV downloads automatically when it finishes, plus a summary prints to console
Bookmarklet not working, or on mobile? Use the manual copy-paste method instead
  1. Log into the staging site normally in your browser
  2. Open DevTools (F12, or Cmd+Option+I on Mac) → Console tab
  3. Paste the copied script and press Enter
Customize checks — score only what matters to you (all on by default)
Custom extraction — pull specific content per page (optional)

Give each field a label and a CSS selector (e.g. .price or span.author-name). The first match on each page is extracted.

Source code search — find a pattern across every page (optional)

Search the raw HTML of every crawled page for a regular expression (e.g. gtag\(.config to find GA4 tags, or a phone number pattern). Case-insensitive.

Every page scored against 28 checks

Not just an inventory — each crawled page gets a health score, and the report shows pass rates across your whole site so you know exactly what to fix first.

Crawl & links

Broken links with the page they were found on, redirect chains, crawl depth, internal inlinks, and orphan pages (in your sitemap but never linked).

Snippet quality

Title 30–60 chars and under ~580px, description 70–160 chars and under ~920px, duplicate titles and descriptions flagged across the whole crawl.

Content & structure

Single H1, heading hierarchy gaps, word count and thin-content flags, Flesch readability, boilerplate ratio.

Indexing

Status codes, canonical presence and match, noindex in meta tags or HTTP headers — the pages Google can and can't see.

Schema & social

JSON-LD presence and validity, required properties per type (Product, Review, LocalBusiness, Article and more), FAQPage and BreadcrumbList completeness, plus og:title, og:description, og:image and twitter:card coverage.

Technical hygiene

Mixed content, images without alt text, favicon/viewport/charset, html lang, internal linking (pages with zero inlinks).

Need to crawl more than 500 pages?

Bigger background crawls (no tab open), saved crawl history and white-label reports are coming for agencies and large sites. Join the waitlist and we'll notify you at launch.