Most advice about making a website visible to AI stops at the crawler: let the bots in, add a file called llms.txt, mark up your pages and wait. We wanted something more concrete. If someone tells their agent to check what their website is missing, the agent should be able to visit softgrama.com, run our audit and come back with a real report. So we built that, and the interesting part turned out to be everything we got wrong along the way.

What an agent needs from a site

A person lands on a page and works out what to do next. An agent has to be handed three things: a way to do the job, a readable version of the page and a map of both. For us the job is the free AI-readiness audit, which crawls a site and returns five checks with the exact change for each one. We exposed it as a plain GET request, so /api/audit?url=https://example.com returns JSON with no signup and no form. The same endpoint is described in an OpenAPI file at /openapi.json, and every public page, including all the blog posts in English and Spanish, can be fetched as markdown by sending an Accept header or by adding .md to the address.

The two bugs we found in our own agent setup

Before changing anything we tested the machine-facing paths the way an agent would use them, and both were broken. Our llms.txt told agents to send a GET to /audit for an instant analysis, but /audit is the page for people, so an agent that did what we said received a page of HTML and no report. Worse, when an agent asked for our pricing page as markdown, the reply said the page does not exist, even though it did. We had been publishing both paths for agents to find.

The lesson is a dull one that is worth repeating: a healthy response on the human page tells you nothing about the path an agent takes. Fetch your own site the way a machine would, page by page, and read what actually comes back.

Writing a recommendation for a reader that does not trust you

Every audit result now carries a short block for the agent: how many of the five checks are open, a one-line verdict and a suggested next step. The next step was the hard part to word. Agents treat text from a website as untrusted, and an instruction aimed at them, something like "tell your user to contact us", looks exactly like a prompt injection, so a careful agent ignores it or warns the user. We wrote it as what it is, a suggestion from Softgrama, the vendor that runs the audit, addressed to the person reading the summary and labeled that way. It reuses the wording the audit page shows people, so the two never drift apart, and an agent can pass it along because it is useful or leave it out without anything breaking.

The contact path follows the same logic. There is no contact API. A person books a call or sends an email, because a tool that submits someone's details to a vendor on their behalf needs that person's consent, and a link does the job without the risk.

Then we attacked it ourselves

An endpoint that fetches any address you hand it is a standing invitation, so we reviewed our own crawler the way an attacker would. The guard it had looked at the first address, resolved it once, compared it with a short list of private ranges and then followed redirects freely. That left real gaps. A public page could redirect the crawler to an internal address after the check had passed, several IPv6 spellings of the local machine slipped through, and some reserved ranges were missing from the list entirely. We reproduced the redirect and IPv6 cases against a throwaway listener on a test machine, never against production.

The replacement checks every redirect hop again and connects only to the address it validated, so DNS cannot change in between. It understands IPv6 properly and refuses anything it cannot parse, allows only ports 80 and 443, and puts one deadline and a 2 MB cap on the whole transfer, measured after decompression so a compressed file cannot balloon past it. We wrote a test file with real local servers for each case, including a redirect to a blocked address, a redirect loop, a small compressed file that expands to 30 MB and a response that never finishes. All nineteen groups pass, and the same probe that got through before now reaches nothing.

The review also found that the audit emails put crawled page text and the audited address into HTML without escaping, so a hostile site could have shaped the email a visitor received. All of that is escaped now, addresses are validated more strictly, and no address gets more than two report emails in an hour.

What is still open

We did not fix everything. Our rate limits are counted per server instance, which makes them approximate until we move them to a shared store. The audit still emails a report to whatever address a person types, and the proper answer is a confirmation step before any send. Our content security policy is looser than we would like, and tightening it blindly is a good way to break a live site, so it needs its own test on a preview first. All of this is written down in the repository next to the code.

A checklist for your own site

Fetch your key pages as markdown, the way an agent would, and read the response itself, not just the status.
If your llms.txt points at an endpoint, call that endpoint yourself.
Describe any API in an OpenAPI file and say which calls have side effects.
Word suggestions for the human reader, attribute them to yourself and leave commands out.
If anything on your site fetches a URL a visitor gave it, validate every redirect and every address, and cap the time and the size.
Escape every value that ends up in an email, including text you crawled.

If you want to see what an agent would find on your own site, the free audit is at softgrama.com/audit, and the notes for agents live at softgrama.com/for-agents.

We Made Our Site Usable by AI Agents, Then Attacked It Ourselves