The browser doesn't show the parts that decide whether a search engine may index the page. A noindex from a staging config can leak into production. Canonicals copied from a template can point at another page. Someone adds a Disallow to robots.txt and it covers a whole folder. Affiliate links go out without rel="sponsored". None of these change how the page looks.

So after every deploy that adds or changes a public page, I run a few commands against the live URL. Total time is under a minute.

What these commands can and can't tell you

The commands read what your server sends: status codes, redirects, headers, HTML and robots.txt. They don't tell you whether Google indexed the page. Google makes that call later, and it treats a canonical as a strong signal it weighs against other signals.

For actual index status, use URL Inspection in Search Console. It also shows the HTML Googlebot received, which helps when your server sends different content to bots.

What the commands do catch: the mistakes you made yourself, on the page you just shipped.

Status and redirects

Start with the response code and where the URL ends up:

curl -sIL -o /dev/null -w '%{http_code} %{url_effective} (%{num_redirects} redirects)\n' http://getlinkbunny.com/

200 https://getlinkbunny.com/ (1 redirects)

You want a 200 on the URL you intend to publish. If the final URL differs from the one in your sitemap or internal links, fix the links. If the page returns a 404 or redirects to the homepage, the route didn't deploy the way you think.

X-Robots-Tag header

A noindex can come from an HTTP header instead of the HTML. Hosting configs, CDN rules and preview environments can add it. Check the headers on every hop:

curl -sIL https://example.com/new-page | grep -iE '^(x-robots-tag|link):'

No output means neither header was sent. If you see x-robots-tag: noindex on a production URL, find the config that adds it. Google's block indexing docs say the meta tag and the header have the same effect, so one is enough to keep the page out. The link: match also catches a canonical sent as a header, which some sites use for PDFs and other non-HTML files.

Meta robots tag

Next, the robots meta tag in the HTML:

curl -sL https://getlinkbunny.com/ | grep -ioE '<meta[^>]+name="(robots|googlebot)"[^>]*>'

<meta name="robots" content="index, follow">

Look for noindex or none in content. Missing tag is fine; indexing is the default. The grep assumes double quotes around the attribute value. If your framework writes single quotes, adjust the pattern.

Keep in mind that curl doesn't run JavaScript. If your app injects the meta tag on the client, you won't see it here. For prerendered or server-rendered pages, the curl output is what a crawler reads first.

Canonical URL

curl -sL https://getlinkbunny.com/ | grep -ioE '<link[^>]+rel="canonical"[^>]*>'

<link rel="canonical" href="https://getlinkbunny.com/">

Check three things: the canonical points at this page and not at the homepage or a sibling page, it uses the production host and not a staging or preview domain, and it's an absolute URL. Google's duplicate URL docs recommend absolute URLs for this reason. When an agent creates a page by copying an existing one, check this first, since it can keep the old head tags.

robots.txt

curl -s https://getlinkbunny.com/robots.txt | grep -iE '^(user-agent|disallow|allow):'

User-agent: *
Allow: /

Compare each Disallow line with the path of your new page. A Disallow rule for /docs also blocks /docs/new-guide.

Check robots.txt before you trust a noindex. If robots.txt blocks a URL, Google never fetches the page, so it never sees a noindex on it. The block indexing docs say such a page can still show up in results when other sites link to it. Don't use robots.txt to remove a page from search. Use noindex and leave the page crawlable.

Outgoing link rel values

Last, the links. Google's outbound link docs ask for rel="sponsored" on paid and affiliate links, rel="ugc" on user-submitted links like comments, and rel="nofollow" when neither fits and you don't want to vouch for the link. You can combine them, e.g. rel="ugc nofollow".

This lists outgoing links that carry none of the three:

url=https://example.com/new-page
origin=https://example.com
curl -sL "$url" \
  | grep -oiE '<a [^>]*href="https?://[^"]*"[^>]*>' \
  | grep -viE "href=\"$origin" \
  | grep -viE 'rel="[^"]*(nofollow|sponsored|ugc)'

Most links in the output are normal editorial links and should stay that way. Read the list for anything paid. To pull out likely affiliate links that lack sponsored, filter on your affiliate URL patterns:

curl -sL "$url" \
  | grep -oiE '<a [^>]*href="https?://[^"]*"[^>]*>' \
  | grep -iE 'amzn\.to|[?&](tag|ref|aff|affiliate)=|/go/' \
  | grep -viE 'rel="[^"]*sponsored'

The pattern list is a guess. Replace it with the patterns your affiliate programs use. The grep also misses links built with JavaScript, and links whose rel uses single quotes.

One script for all of it

I keep the checks in one file and pass the URL:

#!/usr/bin/env bash
# usage: ./check-page.sh https://example.com/some/page
page_url="$1"
origin=$(echo "$page_url" | grep -oE '^https?://[^/]+')

echo "== Status and final URL"
curl -sIL -o /dev/null -w '%{http_code} %{url_effective} (%{num_redirects} redirects)\n' "$page_url"

echo "== X-Robots-Tag and Link headers"
curl -sIL "$page_url" | grep -iE '^(x-robots-tag|link):'

page_html=$(curl -sL "$page_url")

echo "== Meta robots"
echo "$page_html" | grep -ioE '<meta[^>]+name="(robots|googlebot)"[^>]*>'

echo "== Canonical"
echo "$page_html" | grep -ioE '<link[^>]+rel="canonical"[^>]*>'

echo "== robots.txt rules"
curl -s "$origin/robots.txt" | grep -iE '^(user-agent|disallow|allow):'

echo "== Outgoing links without nofollow, sponsored or ugc"
echo "$page_html" | grep -oiE '<a [^>]*href="https?://[^"]*"[^>]*>' \
  | grep -viE "href=\"$origin" \
  | grep -viE 'rel="[^"]*(nofollow|sponsored|ugc)'

An empty section means nothing matched. That's the result you want for headers, and fine for meta robots.

A prompt for your coding agent

The agent shipped the page, so the agent can check it too. I add this to the end of a page task, or keep it in my agent instructions file:

After deploying, run ./check-page.sh <the new page's production URL> and report:
- status code and final URL
- any noindex or none in X-Robots-Tag or meta robots
- the canonical URL, and whether it matches the page URL on the production host
- any robots.txt Disallow rule that covers the page path
- outgoing links that look paid or affiliate but have no rel="sponsored"
Fix anything that comes from our code or config. Do not change rel values on editorial links. Do not claim the page is indexed; these checks only read what the server sends.

The last line keeps the agent from reporting "page indexed" when it only saw a 200.

Free checkers that do the same reads

If you'd rather paste a URL than run a script, I built two free checkers that need no signup. The indexability checker reads the HTTP response, redirects, meta robots, X-Robots-Tag, canonical declarations and robots.txt availability for one page. The nofollow link checker lists the rel value of every link on one page, plus page-level nofollow from meta robots and X-Robots-Tag. Both read server HTML only and don't run JavaScript, same as curl, and neither reports Google index status.

I also build LinkBunny, where coding agents handle nofollow link exchanges between sites.