Track AI crawlers (ChatGPT, Claude, Perplexity, Googlebot)

When someone asks ChatGPT about your product, ChatGPT fetches your page to answer. So does Claude, so does Perplexity, and Googlebot has always done it. None of them run JavaScript — they read the raw HTML and leave. That's why the tracking script can never see them, and why this one feature needs a few lines in your own backend.

The whole mechanism is one HTTP call: on each incoming request, tell us the user-agent and the path. We work out whether it was a crawler, which one, and why it came.

Where to install it#

Most sites have more than one place that sees a request — a CDN in front, a server, and the app itself. You only want this in ONE of them, or every crawl gets counted twice. Pick the innermost layer you control:

• Your app has middleware (Next.js, Hono, Express, Django, Rails, Laravel…) → put it there. This is almost always the right answer: it deploys with your code, it's version-controlled, and it works the same in every environment. You do not need Cloudflare for this.
• You're on Cloudflare Pages or Workers and that IS your app → use the Cloudflare snippet; there's no separate server underneath.
• Your app is static (plain HTML, a pure SPA, a static export) with a CDN in front → the CDN is the only thing that sees crawler requests, so it has to go there.
• You have Cloudflare in front of an app that also has middleware → still use the app's middleware. Installing in both double-counts.

What to send#

{
  "siteId": "YOUR_SITE_ID",
  "path": "/pricing",
  "userAgent": "<the request's User-Agent header>",
  "ip": "<the requesting IP>"
}

• siteId — yours is already filled in above. It's the same id as your script tag.
• path — the path that was requested.
• userAgent — send it raw. Don't filter crawlers yourself: we classify server-side, so the crawler list stays current without you redeploying.
• ip — optional, but it's what lets us confirm a crawler is genuinely who it claims to be. Without it you'll never see 'verified' badges.

Next.js#

Create middleware.ts in your project root — the same folder as package.json, not inside app/. If you already have one, add the fetch call to it and keep your own return.

import { NextResponse, type NextRequest } from "next/server";

export function middleware(request: NextRequest) {
  void fetch("https://visitrack.app/api/collect-bot", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      siteId: "YOUR_SITE_ID",
      path: request.nextUrl.pathname,
      userAgent: request.headers.get("user-agent"),
      ip: request.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
    }),
  }).catch(() => {});

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Cloudflare Pages#

Create functions/_middleware.js in your project. Cloudflare runs it for every request automatically — there's nothing to register. waitUntil keeps the report alive after the response has already gone back to the crawler.

export async function onRequest(context) {
  const { request, next, waitUntil } = context;
  const url = new URL(request.url);

  waitUntil(
    fetch("https://visitrack.app/api/collect-bot", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        siteId: "YOUR_SITE_ID",
        path: url.pathname,
        userAgent: request.headers.get("user-agent"),
        ip: request.headers.get("cf-connecting-ip"),
      }),
    }).catch(() => {}),
  );

  return next();
}

Cloudflare Workers

Same idea in the Worker's entry file, then deploy with npx wrangler deploy.

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);

    ctx.waitUntil(
      fetch("https://visitrack.app/api/collect-bot", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          siteId: "YOUR_SITE_ID",
          path: url.pathname,
          userAgent: request.headers.get("user-agent"),
          ip: request.headers.get("cf-connecting-ip"),
        }),
      }).catch(() => {}),
    );

    return fetch(request);
  },
};

Express#

app.set("trust proxy", true); // so req.ip is the real client behind a proxy

app.use((req, res, next) => {
  fetch("https://visitrack.app/api/collect-bot", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      siteId: "YOUR_SITE_ID",
      path: req.path,
      userAgent: req.get("user-agent"),
      ip: (req.get("x-forwarded-for") ?? "").split(",")[0].trim() || req.ip,
    }),
  }).catch(() => {});

  next();
});

Anything else#

It's a plain HTTP POST, so any language or framework works. Find the place your framework runs code on every request — middleware, a filter, a hook, a before_action — and send it from there without waiting for the reply.

import httpx

def crawler_tracking_middleware(get_response):
    def middleware(request):
        try:
            httpx.post("https://visitrack.app/api/collect-bot", json={
                "siteId": "YOUR_SITE_ID",
                "path": request.path,
                "userAgent": request.headers.get("user-agent"),
                "ip": request.META.get("HTTP_X_FORWARDED_FOR", "").split(",")[0].strip(),
            }, timeout=2)
        except Exception:
            pass
        return get_response(request)
    return middleware

Three rules that matter#

1. 1Never make the crawler wait. Fire the request and ignore the result — no await on the response, no retries. A missed report costs you one data point; a slow page costs you the ranking you're measuring.
2. 2Don't skip robots.txt, llms.txt or sitemap.xml. Crawlers request those first. Middleware matchers that skip 'static files' are the single most common reason this integration looks installed but records nothing interesting.
3. 3Send every request and let us classify. Filtering user-agents yourself means a newly launched crawler stays invisible until you redeploy.

Verify it#

Pretend to be GPTBot and check the answer, rather than waiting around for a real crawler:

curl -s -X POST https://visitrack.app/api/collect-bot \
  -H "Content-Type: application/json" \
  -d '{"siteId":"YOUR_SITE_ID","path":"/test","userAgent":"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"}'

A working setup answers {"recorded":true,"crawler":"GPTBot","category":"training","verified":false}, and the crawl appears on your Bots tab within seconds. Then hit your own deployed site with a crawler user-agent to confirm the middleware path fires too:

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://your-site.com/robots.txt

Reading the results#

• AI answers — someone asked an assistant something and it fetched your page live to answer them. This is the category that maps to real people reading about you right now.
• Indexing — search and answer engines keeping their index of your site current.
• Training — crawlers collecting pages for model training corpora.
• Verified — the crawler's IP was confirmed to genuinely belong to the provider it claims. Anyone can put 'GPTBot' in a user-agent; a verified badge means the IP backed it up.

Troubleshooting#

• The curl test works but nothing appears from real traffic — your middleware isn't running for those paths. Check the matcher, and confirm the file is where the framework expects (Next.js: project root, not app/).
• Everything says unverified — you're not sending the ip field, or you're sending your own server's IP instead of the client's. Check the header your platform uses.
• Nothing at all, not even the curl test — double-check the siteId, and that you're posting to /api/collect-bot rather than /api/collect.
• Very few crawls — that's often just the truth early on. Crawlers visit far more once a site has inbound links and a sitemap.