Sitemaps and indexing

This page tells you how Vantage SEO builds the XML sitemap and llms.txt, when it sends IndexNow requests, how to serve its files at the root of your site, and how it counts 404 hits.

Where the files are

Instatic publishes page HTML only. It cannot write files such as sitemap.xml or robots.txt at the root of your site. The plugin therefore serves its files from its own public runtime path:

Add your site URL before each address, for example https://harborandpine.coffee/admin/api/cms/plugins/monkeywebs.seo/runtime/sitemap.xml. Step 3 of the setup wizard shows the full addresses for your site.

XML sitemap

The plugin keeps a list of published pages. It updates the list each time Instatic publishes a page. The sitemap is built from this list.

• Each entry has the page URL and a <lastmod> date.
• The <lastmod> date changes only when the slug, the title or the page content changes. A publish with no changes keeps the old date.
• Pages with noindex are not in the sitemap.
• The sitemap lists up to 5,000 URLs.
• The sitemap shows "not available" until you save a site URL and publish at least one page.
• Browsers and crawlers can cache the sitemap for 5 minutes.

To submit the sitemap, add its full address in Google Search Console and Bing Webmaster Tools. With Pro and a Search Console connection, you can submit it from the plugin. See Google Search Console.

llms.txt

llms.txt is a Markdown index of your site for AI crawlers. It follows the format from llmstxt.org:

# Harbor & Pine Coffee

> Small-batch coffee roasted in Portland.

## Pages

- [Menu](https://harborandpine.coffee/menu)
- [Visit us](https://harborandpine.coffee/visit)

The heading is your site name. The quote line is your default meta description. The page list is the same list as the sitemap. llms.txt has no separate settings.

IndexNow

IndexNow is a protocol that tells search engines, such as Bing and Yandex, that a URL changed. The plugin sends one request to https://api.indexnow.org/indexnow. The request contains:

• your site host,
• your IndexNow key,
• the address of the key file,
• the URLs of the pages that changed.

When the plugin sends IndexNow requests

1. When Instatic publishes a page, the plugin compares the page with the last version. If the slug, the title or the content changed, it marks the page as pending. The publish itself sends nothing.
2. A maintenance task runs every 15 minutes. It sends the pending URLs in one request, up to 50 URLs. Other pending URLs wait for the next run.
3. After a successful request, the plugin clears the pending mark.

The plugin creates the IndexNow key at the first request. The key file returns "no key yet" until then. The SEO Settings page shows a warning when IndexNow requests fail.

Turn IndexNow off

1. Open SEO Settings, General tab.
2. Under Indexing, turn off IndexNow instant indexing.
3. Save.

IndexNow is on by default. It also needs a saved site URL. Without a site URL, the plugin sends nothing.

Serve the files at the root of your site

You can make /sitemap.xml, /llms.txt and /indexnow-key.txt work at the root of your site. Add rewrite rules on the reverse proxy in front of Instatic. The rules send each root path to the runtime path.

Pro The redirect exports on the Redirects tab include these rules for nginx and Caddy. See Redirects. With the free edition, you can write the same rules by hand.

nginx

Put these lines inside the server { } block that serves your site. A location for /admin/api/ must also send requests to the Instatic host. Remove any other exact location for these paths first, or nginx -t rejects the block.

location = /sitemap.xml { rewrite ^ /admin/api/cms/plugins/monkeywebs.seo/runtime/sitemap.xml last; }
location = /llms.txt { rewrite ^ /admin/api/cms/plugins/monkeywebs.seo/runtime/llms.txt last; }
location = /indexnow-key.txt { rewrite ^ /admin/api/cms/plugins/monkeywebs.seo/runtime/indexnow-key.txt last; }

Caddy

Put these lines inside the site block. A handler for /admin/api/ must also send requests to the Instatic host.

@vseo_root_file_sitemap path /sitemap.xml
rewrite @vseo_root_file_sitemap /admin/api/cms/plugins/monkeywebs.seo/runtime/sitemap.xml
@vseo_root_file_llms path /llms.txt
rewrite @vseo_root_file_llms /admin/api/cms/plugins/monkeywebs.seo/runtime/llms.txt
@vseo_root_file_indexnow_key path /indexnow-key.txt
rewrite @vseo_root_file_indexnow_key /admin/api/cms/plugins/monkeywebs.seo/runtime/indexnow-key.txt

Cloudflare

The Cloudflare export uses Bulk Redirects. Bulk Redirects cannot do an internal rewrite, so the Cloudflare export does not include root-file rules. Use a Cloudflare rewrite rule of your own, or put nginx or Caddy in front of Instatic.

Robots control

The plugin does not create a robots.txt file. Instatic cannot serve files at the root, and the plugin has no robots.txt route. The plugin controls robots with the <meta name="robots"> tag on each page. See Robots tags.

Breadcrumbs

The plugin adds a BreadcrumbList node to the JSON-LD graph of each regular page. It builds the trail from the slug. For the slug menu/espresso, the trail is Home, Menu, then the page title. The plugin does not add visible breadcrumb HTML to the page.

404 counter

When you turn analytics on, the plugin also counts visits to your 404 page. The analytics script on the 404 page sends the missed path to the plugin. The plugin keeps a count per path per day, for 30 days.

• Free The dashboard widget shows the total of 404 hits for the last 7 days.
• Pro The Redirects tab lists the missed paths with their counts. You can create a redirect from a path in one step.

The 404 counter needs analytics on. See Analytics.

URL changes

When you change the slug of a page, Instatic does not redirect the old address. The old address returns 404. The plugin records each slug change when you publish. To send visitors and search engines from the old address to the new one, you need a redirect rule on your proxy.

Pro The Redirects tab lists the recorded slug changes under Host URL changes, and it creates the redirect rules for your proxy. See Redirects.