What does Robots.txt Checker check?
If you need to check robots.txt online, simply enter the website or file address and run the analysis. Robots TXT Analyzer displays the current User Agent, Allow, and Disallow values, helps you see blocked URLs, and check whether important pages are blocked from crawling.
The robots.txt checker analyzes the file's rules and shows how they are applied to website pages. This robots.txt analysis is more useful than simply viewing the contents, as a long file can contain multiple User-Agent groups, intersecting paths, and separate rules for different search engines.
Checking robots.txt is especially necessary after technical edits, when one extra line can limit site crawling. Robots.txt tester helps detect such situations before they affect search engine crawling of important URLs.
Checking robots.txt syntax
Robots.txt validator checks the file structure, recognizes key robots.txt directives, and helps detect syntax errors, incorrect paths, or incorrectly written rules. Syntax checking also reveals suspicious constructs that are formally present in the file but may function differently than the website owner intended.
Robots.txt validity is especially important for sites with many sections, filters, and technical URLs. A single character error can sometimes change the scope of a disallow, so it's helpful to re-check the file and several actual pages after each change.
Checking access to a specific URL
Testing a specific page answers a practical question: whether a given search robot can request a specific URL. The robots.txt test tool compares the address with the current rules and displays the final status of the URL as allowed or blocked.
This format is convenient for diagnosing categories, product cards, articles, filter pages, and other landing URLs. Instead of manually comparing multiple lines, the user immediately sees the result and the rule that affected access.
Checking rules for different search robots
The same URL may have different crawling rules for multiple bots if robots.txt contains separate User-Agent groups. Therefore, checking only the general User-Agent: * doesn't always provide a complete picture of access.
For search engines, Googlebot and Bingbot are typically checked, while for AI services, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot can be used. A specific AI crawler can receive a different set of rules if its own group is created in the file.
What directives does the robots.txt validator parse?
The main robots.txt check is built around directives that determine crawler access to website sections. For accurate diagnostics, it's important to consider not just a single line, but the entire set of rules for the selected User Agent.
| Directive | What is it used for? | Example |
|---|---|---|
| User-agent | Specifies a robot or group of robots | User-agent: Googlebot |
| Disallow | Prevents traversal of the specified path | Disallow: /private/ |
| Allow | Allows a more precise path | Allow: /private/public/ |
| Sitemap | Specifies the sitemap address | Sitemap: https://example.com/sitemap.xml |
RFC 9309 describes the basic logic of the Robots Exclusion Protocol. When checking, it's useful to consider the path precision, group order, and whether the rules match the actual page address.
User-agent
The User-Agent determines which search engine the next group of rules applies to. The User-Agent: * entry applies to all crawlers for which a more appropriate group is not specified in the file.
If a site sets its own restrictions for Googlebot or Bingbot, these should be verified separately. Otherwise, the overall test results may differ from the actual behavior of the selected search robot.
Disallow
Disallow specifies a path that the specified crawler should not request. For example, the rule Disallow: /private/ restricts crawling of addresses starting with the corresponding directory.
An overly broad value can sometimes block commercial categories, service pages, or other useful content. Therefore, after adding a Disallow, it's a good idea to check several real URLs that are structurally close.
Allow
Allow is used when access to a more specific address within a restricted section needs to be preserved. This combination is common on sites with complex directory structures, filters, and technical pages.
When multiple matches occur, it's important to look at the most suitable path and the result of rule processing. Simply visually reading the file is often insufficient, especially with long URL masks.
Sitemap
The Sitemap directive points the search engine robot to the XML Sitemap address, which lists the website's pages. Robots.txt can contain one or multiple Sitemap files for different content types.
Having a Sitemap doesn't negate the need to check the availability of the specified URLs. A page can be in the Sitemap and still be subject to the Disallow setting, so it's important to check these settings together.
How do the * and $ symbols work?
Symbol*used as a wildcard when it is necessary to designate an arbitrary sequence of characters within a path. The symbol$helps to mark the end of the address being checked and make the rule more precise.
Such constructions should be used with caution, as an overly broad mask can affect more pages than intended. After changing such rules, it's best to run the robots.txt checker online on several URLs with different endings.
How to check robots.txt online?
You can check robots.txt online without manually parsing each line of the file. The user specifies the domain, selects the desired page, and, if necessary, specifies the search engine crawler. The system then matches the URL with the found rules.
The verification scheme looks simple:
Website URL → download robots.txt → select User-agent → analyze rules → Allowed or Disallowed → recheck after changes.
This order helps to separate errors in the file itself from problems with a specific page and to quickly understand the cause of the limitation.
Enter your domain or robots.txt address
The standard robots.txt file is located in the website's root directory and is accessible at a URL like https://example.com/robots.txt Rules for a different path or subdirectory do not replace the main file that search bots request in the host's root directory.
The robots.txt validator can accept a domain or a full file address, depending on the interface. Before testing, it's important to ensure that the server actually serves robots.txt and returns the expected HTTP status.
Select a URL and User-Agent to test
After uploading the file, specify the specific URL whose access you want to check. Then select Googlebot, Bingbot, the generic User-Agent, or another crawler if it has specific restrictions specified in robots.txt.
This test displays crawling rules specifically for the selected page and robot combination. This is especially useful when one directory is closed by a general rule, but individual pages within it are opened via the Allow rule.
Check the analysis result
The result usually shows whether URL crawling is allowed or disallowed, as well as the rule that determined the response. If the page is closed unexpectedly, you should check the Disallow path, the corresponding User-Agent group, and more specific rules.
The robots checker itself doesn't change the file on the server or edit site settings. After editing robots.txt, you need to run the test again to ensure the new rule is applied correctly.
What we actually did
Dental clinic · Kyiv and Chernihiv
+44% clicks from search
A domain with no history on a website builder. We built the semantic core for both cities, reworked the landing pages and built the link profile from zero. Four months: 34.8k clicks, impressions 1.32 → 1.76M, DR 0 → 41.
E-commerce · international
+96% clicks in two months
A catalog of digital 3D models. We clustered the semantics, rebuilt the hub pages and closed duplicates and indexing errors. Google users 247 → 532, CTR 2.4% → 4%.
Medical center · Ukraine
+68.75% visibility in the first month
Narrow visibility and a small semantic core at the start. Semantics, landing page structure, metadata and internal linking, then gradual link building.
Answers to your questions
What does Robots.txt Checker check?
Robots.txt Checker downloads the file, analyzes the directives, and determines whether the selected crawler has accessed a specific URL. The check reveals the user agent, allow, disallow, and the rule that influenced the final result.
This tool is useful for technical audits and after robots.txt modifications. It reduces the need for manual review of long files and helps quickly identify erroneous restrictions.
How to check robots.txt online?
Enter the domain or full robots.txt address, then run the file check. For a more accurate test, select the desired User Agent and paste the URL of the page you want to check.
After analysis, check the Allowed or Disallowed result and the rule found. If the configuration has been changed, repeat the test with the same address and compare the results.
Can robots.txt prevent a page from being indexed?
Robots.txt controls crawling, so Disallow can't be considered a guaranteed prohibition of indexing. The address may remain known to the search engine from links or other sources.
To control indexing, noindex or other suitable mechanisms are typically used. However, the crawler must be able to retrieve the page and read the corresponding directive.
What to do if robots.txt closes the page you need?
First, find the Disallow that matches the address and check the entire group for the selected User-Agent. Then determine whether you need to change the path, add a more specific Allow, or revise the rule structure.
After saving the new version, re-test the same URL. Additionally, test adjacent pages to ensure the fix doesn't affect access to other sections.
What happens if there is no robots.txt file on the website?
If the file is missing, Robots Exclusion Protocol rules are not defined for this host. Search engine robots will be able to request accessible URLs unless they are restricted by the server, authorization, or other technical mechanisms.
The absence of a robots.txt file does not automatically index all pages. The indexing decision depends on the document's accessibility, its content, and other search engine signals.
How is robots.txt different from meta robots?
Robots.txt defines the rules for crawler access to the URL before the page loads. Meta robots is located within the HTML and tells the search engine how to process the document once it's retrieved.
For non-HTML files, the X-Robots-Tag can perform a similar task. Therefore, when diagnosing indexing, it's advisable to check robots.txt, meta robots, and HTTP headers together.
Is it possible to check robots.txt for Googlebot separately?
Yes, if the checker supports user agent selection, you can check URLs specifically for Googlebot. This is useful when your file contains separate rules for Google and a common group for other robots.
The result will show which rule applies to the selected page. After changing the file, the same check should be repeated for verification.
Do I need to check robots.txt after changing my website?
Yes, especially after migration, redesign, CMS changes, URL changes, or the introduction of new filters. Old rules may continue to apply to the updated structure and accidentally close important pages.
It's a good idea to include this check in your technical SEO audit after every major release. Several test URLs usually allow you to quickly identify the most dangerous errors.
Related services
Page SEO analysis
Online SEO analysis: check URLs, technical errors, meta tags, content, and key SEO factors. Get clear optimization recommendations for free.
CMS Detector / Detect a site CMS
Identify a website's CMS online by domain or URL. CMS Detector checks for engine features, popular platforms, and web technologies and displays results in seconds.
Indexability check
Check if your page is indexable by Google: robots.txt, noindex, canonical, HTTP status, and other technical indicators. Online URL indexability checker.
Google index check
Check if a URL or website page is indexed in Google. Methods include Search Console, site:, the bulk checker, and reasons why pages are not indexed.
PageSpeed / Core Web Vitals Checker
Core Web Vitals Checker and PageSpeed test online: check your site's speed, LCP, INP, CLS, and get clear optimization recommendations.
HTTP Status Checker
HTTP Status Checker by Seo-Gen: check URL response codes, redirects, and 4xx/5xx errors online. Suitable for single-page and bulk URL checking.
Redirect Checker
Online Redirect Checker: Check 301 and 302 redirects, redirect chains, HTTP codes, and final URLs. Quickly check redirects without installation.
Broken Link Checker
Broken Link Checker by Seo-Gen: Find broken and inactive links, 404 errors, and problematic URLs on your website. Online verification of internal and external links.
Canonical Checker
Canonical Checker Online: Check rel=canonical, target URL, HTTP status, and common page canonicalization errors. Quick canonical check for SEO.
Hreflang Checker
Hreflang Checker by Seo-Gen: Check hreflang, x-default, canonical, language and region codes, backlinks and URL errors online.
Robots.txt Checker helps verify file syntax, rules for individual robots, and actual access to specific website pages. This test is especially useful after technical changes, when an error in a single path can limit scanning of an entire section.
Run a robots.txt check, test the main landing URLs, and compare the results for the relevant User Agents. If the checker shows an unexpected block, first find the rule that triggered it, correct the configuration on the site, and repeat the test.
We reply within one business day. No newsletters, no “just a reminder” calls.
He will look at the site himself instead of passing it to a manager.
More on: Robots.txt Checker – check robots.txt
What errors can robots.txt analysis find?
A file can be syntactically correct and still contain poor access logic. Therefore, technical SEO requires checking both the document structure and the actual results for important pages.
During an SEO audit, robots.txt is typically checked alongside meta robots, HTTP headers, XML sitemaps, and Google Search Console data. This combination helps understand why a search engine isn't crawling a page or why an expected URL isn't indexed.
Robots.txt is missing or incorrectly located
The search engine crawler looks for robots.txt in the root of the corresponding host. A file with a different name, an incorrect path, or a server error will not work as expected.
If robots.txt is missing, its restrictions cannot be applied. However, the absence of the file does not control page indexing or override other technical settings.
The entire site was accidentally closed.
One of the most dangerous configurations looks like this:
User-agent: *Disallow: /
This rule prevents crawling of the entire site for the corresponding group of robots. The error often appears after migrating a project from a test environment, if the staging site settings are transferred to production without verification.
Incorrect Allow and Disallow
Errors are often related to an invalid directory, an extra character, an overly broad mask, or an incorrect combination of Allow and Disallow. Case sensitivity in the path should be taken into account if the server distinguishes between such URLs.
If multiple rules overlap, check the specific URL using the robots txt tester. The final result is more reliable than manual evaluation when the file contains dozens of similar lines.
Errors in User-agent groups
Rules must be in the correct group and assigned to the correct crawler. If lines are accidentally added under a different user agent, Googlebot or another crawler may gain unexpected access.
Before publishing changes, it's a good idea to check the overall User Agent, Googlebot, and other groups important to the project. This approach reduces the risk that a single change will fix access for one robot while simultaneously breaking the settings of another.
Important pages or resources are closed
Disallowed URLs can accidentally include categories, articles, product cards, images, CSS, JavaScript, or filter pages. The solution depends on the site's purpose, so there is no universal list of allowed URLs.
Priority should be given to pages that should be crawled and included in search results. Technical URLs are checked separately to ensure that rules don't unnecessarily consume crawl budget or interfere with the processing of useful content.
How does robots.txt affect crawling and indexing?
Robots.txt controls which URLs a search engine crawler can request. Crawling and indexing are linked, but search engines treat these processes separately.
Because of this, disallowing a page cannot be considered a fully functional way to remove it from search results. Other mechanisms are used to control indexing, which the search robot must detect when loading the page.
Crawling and indexing are not the same thing
Crawling means receiving a URL from a search engine robot and processing its available content. Indexing means adding the obtained data to the search index after the system evaluates the page.
If robots.txt prohibits crawling, the robot may not see the page's content and its meta robots tag. Therefore, crawl settings should always be aligned with the indexing task.
Can a page blocked by robots.txt be crawled by Google?
Yes, an address can sometimes remain known to a search engine through links, old data, or other sources. In such a situation, the URL may be present in the search engine's database even without regular content scanning.
Therefore, Disallow shouldn't be used as a guaranteed block on indexing. If the goal is to specifically exclude a page from search results, you should choose an appropriate indexing management method and test it separately.
How is robots.txt different from noindex?
Robots.txt regulates search engine requests to URLs, and noindex tells the search engine not to index the page. To read noindex, the crawler must access the document or the corresponding HTTP header.
If a page is simultaneously blocked in robots.txt, the search engine may not see the required directive. Therefore, the combination of Disallow and noindex should be tested against the actual purpose of the setting.
Meta robots and X-Robots-Tag
Meta robots are placed in the page's HTML and can contain directives like noindex or nofollow. The X-Robots-Tag transmits similar instructions via an HTTP header and is suitable for files or other resources without regular HTML.
These mechanisms don't replace robots.txt, as they operate at a different stage of URL processing. When diagnosing a complex issue, it's worth checking all three sources of restrictions.
How to check if a page is accessible to Googlebot?
To check, select Googlebot and paste the URL of the page you want to crawl. After processing robots.txt, the checker will show whether the current configuration allows crawling of this address.
If the result unexpectedly shows "Disallowed", first find the rule that matched the URL. Then check the entire Googlebot group and general rules, as multiple paths may intersect.
Checking URLs using robots.txt rules
First, upload the file, then select Googlebot and enter the desired URL. Robots.txt validator will match the address with the rules and display the resulting access.
After fixing the file, repeat the check with the same URL. This approach allows you to compare the results before and after the change without making any additional assumptions.
Why do you need to check a specific URL?
A long robots.txt file is difficult to evaluate visually when multiple rules match a single page. A specific test shows the actual result for the URL that's truly important for SEO.
It's especially useful to check URLs after changing the site structure or filtering rules. Even a small change in the path can move a page from an allowed group to a restricted one.
Checking robots.txt for AI bots
A modern robots.txt file can contain separate rules for search engines and AI crawlers. Checking these groups helps understand which automated systems access site content and what restrictions apply to each user agent.
Access settings should be chosen wisely, as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot use different identifiers. Enabling or disabling one user agent does not automatically apply the same rule to all AI services.
Which AI crawlers can be tested?
Robots.txt may contain separate identifiers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and other identifiers. Some projects also define separate rules for Google services and other systems that use automatic data collection.
The check must show an actual match between the user agent and the current rule group. Granting access does not guarantee that the site will be cited, appear in AI responses, or otherwise use the content.
How to fix errors in robots.txt after verification?
Fixes should begin with the rule that's causing the undesirable result. Removing all restrictions indiscriminately can open up service sections and increase the number of unnecessary requests to the site.
After editing, the file should be re-checked with a robots checker and tested with several different types of URLs. This monitoring is especially necessary after migrations, CMS changes, and directory structure changes.
First define a rule that blocks the URL
If the page is closed, find the corresponding Disallow and check its path. Then see if there's a more specific Allow or a separate group for the desired User Agent.
Don't change a rule just because the checker shows a block. First, understand whether a specific page should be crawled and what role it plays in the site's structure.
Edit robots.txt on the website side
The editing method depends on the project's infrastructure. On some sites, robots.txt is stored as a file on the server, while on others, it is generated by the CMS, SEO module, or server-side logic.
Before making changes, it's important to identify the source of the generated file to ensure the new version isn't overwritten after an update or deployment. After saving, check the public robots.txt URL directly in your browser.
Recheck the file and important URLs
After making changes, test your homepage, categories, product cards, articles, and other landing URLs. Also, check your sitemap and technical sections for which special restrictions were set.
If a site uses multiple languages, subdomains, or separate hosts, each robots.txt file is checked separately. A rule on one host does not control crawler behavior on another.
Examples of robots.txt rules
Examples help quickly understand the relationship between User-Agent, Allow, and Disallow. They should be adapted to the actual site structure, as the same configuration rarely works across multiple projects without modification.
Before publishing your own file, check all used paths and actual URLs. Be especially careful when testing wildcard rules and specific search engine crawler groups.
Allow crawling of the entire site
A minimal configuration without private sections might look like this:
User-agent: *Disallow:
This entry doesn't create URL crawling restrictions for the general group. Additional restrictions may exist in individual groups for other User-Agents, so the entire file still needs to be reviewed.
Disable partition scanning
To close a specific directory, you can use the following example:
User-agent: *Disallow: /private/
After adding the rule, check several URLs within the directory and pages with similar names. This will help ensure that the rule hasn't affected adjacent sections of the site.
Set a rule for a specific search engine robot
A separate group can be configured for a specific crawler:
User-agent: GooglebotDisallow: /example-section/
Other robots will use their own groups or common rules if there's no matching entry. Therefore, before making any changes, it's important to understand which user agent the setting applies to.
Recommendation from an SEO specialist
I check robots.txt not only for syntax errors, but also for the actual results for specific URLs and User Agents. A formally correct file may still block important pages due to an overly broad disallow, an incorrect path, or the intersection of multiple rules.
After any change to robots.txt, I separately check key landing pages, categories, and technical URLs. Then I compare the results with the XML Sitemap, meta robots, X-Robots-Tag, and Google Search Console data to ensure that crawl settings are consistent with indexing.