How we collect local business data

Our sources

Business counts: Australian Bureau of Statistics

The number of businesses in each area, by industry and by number of employees, comes from the ABS Counts of Australian Businesses, published through the ABS Data API (Data by Region). Figures are at 30 June of the latest year released. Each location page names the ABS areas it uses: SA2 areas for suburbs, SA4 areas for regions, and Greater Capital City areas for capitals.

Business list: OpenStreetMap

To find local business websites we use businesses listed in OpenStreetMap (© OpenStreetMap contributors, available under the Open Database Licence), in categories that match six industries: construction and trades, health care, real estate, professional services, accommodation and food, and personal services. We take businesses that list a website, up to 12 per industry per location, chosen by a fixed rule so the same businesses are picked on each run.

Website check: our own crawler

For each sampled business we fetch three things: robots.txt, the home page and /llms.txt. Our crawler identifies itself as GalactiaLocalData/0.1 (+https://galactia.co), obeys robots.txt for its own name, and fetches nothing else from the site in that run. From those three files we record:

• whether robots.txt blocks any AI search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot), any user-triggered fetcher (ChatGPT-User, Perplexity-User, Claude-User) or any AI training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent);
• the structured data types on the home page, and whether any is LocalBusiness or one of its subtypes;
• whether the home page has a meta description, a mobile viewport tag and HTTPS;
• which site builder it appears to use, from its code;
• whether an llms.txt file exists and, where the file says so or the platform is clear, what generated it.

Answer engine check: a second home page fetch

For our answer engine optimisation pages we fetch each home page once more, only where the first check reached it and robots.txt allowed our crawler. From that page we record:

• whether the title, meta description or main heading (H1) names the location, or the business's own suburb or town as listed in OpenStreetMap;
• how many H1 to H3 headings there are, and whether any ends in a question mark;
• whether the page has FAQ or Q&A structured data;
• whether it links to Google Maps or a Google Business Profile.

Generative engine check: a third home page fetch

For our generative engine optimisation pages we fetch each home page once more, on the same terms, and record what an AI answer could quote and whether the site links to the business's other profiles:

• whether the visible text states a specific figure: a number with a unit, such as years, jobs or reviews, or a dollar amount, ignoring copyright years;
• whether the page shows a testimonial, review or quote, and whether it carries Review or AggregateRating structured data;
• whether it links to an official or industry source on a .gov, .edu or .org.au domain;
• whether its structured data uses sameAs to connect the site to other profiles.

Why we separate the crawlers

AI companies run different crawlers for different jobs, and blocking each has a different effect. OpenAI says blocking OAI-SearchBot keeps a site out of ChatGPT search answers, while blocking GPTBot only opts out of model training and ChatGPT-User is not used to decide search inclusion (OpenAI). Perplexity and Anthropic draw similar lines (Perplexity, Anthropic). So we only count a site as blocking AI search when it blocks a search crawler.

Samples by location

What the data cannot tell you

• The samples are small: tens of sites per location, not thousands. Percentages are a fair pointer to common gaps, not a census. Every figure on the site shows its sample size.
• OpenStreetMap does not list every business. It covers shops, food and health well and trades less well, and businesses that list a website there may not be typical of all businesses in the area.
• We check home pages only. A site may have structured data or a meta description on other pages.
• A check is a snapshot. Sites change, and each page shows the date of its check.

Support programmes

Government and council programmes on location pages come from the official programme pages, which we read and link. We list a programme only when an official page confirms it, and we record whether it is open or closed on the date we checked.

How we name businesses

We publish totals, not lists. We never name a business for a gap on its website.

Corrections

If you think a figure is wrong, or you run one of the sites we checked and want to know what we found, email liam@galactia.co. We correct errors and say so on the page. Our services are described on the AI SEO services page.