How to use the robots.txt generator?
This tool is suitable for creating robots.txt online for a new website, updating rules after changing the structure, or checking the configuration before launch. In English-language documentation and services, this tool is also referred to as robots.txt generator, robots.txt creator, or robots file generator.
First, specify which crawler the rules will apply to. Then, add paths that are allowed or blocked from crawling, specify the sitemap, and test the resulting code. After testing, you can copy or download the file and place it at https://example.com/robots.txt.
If you need to generate robots.txt for a small website, a single User-Agent group and a few rules are usually sufficient. For large online stores, catalogs, and services, the configuration may include separate groups for Googlebot, Yandex, Bingbot, and AI crawlers.
Select a search robot
The User-agent directive specifies which crawler the following set of rules applies to. Meaning*applies to all robots that support the Robots Exclusion Protocol and do not have a more appropriate separate group.
You can set different conditions for different robots. For example, you can allow Googlebot to crawl commercial pages, while restricting access to service sections for a specific technical bot. This approach is more convenient than trying to describe all the logic in a single universal rule.
Add Allow and Disallow rules
Disallow specifies a path the robot should not crawl, while Allow can open a specific URL within a broader prohibition. Rules are written relative to the site root, so the full domain is usually not needed in these lines.
For example, the entry Disallow: /admin/ restricts bypassing the admin section. If there's a public page within it, you can add Allow: /admin/public/. Before publishing, it's a good idea to test several real URLs to ensure the rule works as expected.
Specify your Sitemap and get a ready-made file
The Sitemap directive specifies the full URL of the sitemap, for example, https://example.com/sitemap.xml. If there are multiple sitemaps, each can be added on a separate line to help the search crawler find the relevant addresses more quickly.
Once configured, the generator displays the final code, which you can check, copy, and save. This is the principle used by the robots.txt generator online, the SEO robots.txt generator, and other tools that help generate robots.txt files without manually preparing the file from scratch.
I use robots.txt primarily to manage site crawling. If a URL is already known to a search engine, a single Disallow may not be enough to remove it from search results. Before publishing, I check the file's syntax, sitemap, and the accessibility of key commercial pages. I always test the Disallow: / rule separately, as an error in this line can block the entire site from the selected robot.2
Basic robots.txt directives
The basic configuration is built around User-agent, Disallow, Allow, and Sitemap. Additional directives depend on the specific search engine, so their support should be verified separately, rather than transferring settings from another site without testing.
| Directive | What is it used for? | Example |
|---|---|---|
| User-agent | Selects a robot or group of robots | User-agent: * |
| Disallow | Limits path traversal | Disallow: /search/ |
| Allow | Allows a more specific path | Allow: /search/help/ |
| Sitemap | Specifies an XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Specifies a delay for robots that support the directive. | Crawl delay: 5 |
| Clean-param | Helps Yandex take URL parameters into account | Clean-param: utm_source |
The table provides a general outline, but the rules need to be matched to the structure of a specific project. The same disallow may be safe for a technical section and critical for a page that receives organic traffic.
User-agent, Disallow, and Allow
User-agent: * is suitable for general rules, while separate groups are needed when different bots need to behave differently. The Disallow: / entry requires special attention, as it prevents the specified bot from crawling the entire site.
Allow is most often used within an already blocked section. If the logic is complex, it's best to test each group against real URLs before publishing the file. Copying a ready-made robots.txt file from someone else's website often leads to accidentally blocking necessary pages.
Crawl-delay, Clean-param and Host
Google doesn't use Crawl-delay in robots.txt, so adding this directive specifically for Googlebot is pointless. Other robots may handle it differently, and support should be checked in the specific crawler's documentation.
Clean-param is specific to Yandex and is used for URLs with parameters that don't change the main content of the page. Host shouldn't be added as a mandatory directive in a modern robots.txt file, especially if its use is solely due to an old template.
What we actually did
Dental clinic · Kyiv and Chernihiv
+44% clicks from search
A domain with no history on a website builder. We built the semantic core for both cities, reworked the landing pages and built the link profile from zero. Four months: 34.8k clicks, impressions 1.32 → 1.76M, DR 0 → 41.
E-commerce · international
+96% clicks in two months
A catalog of digital 3D models. We clustered the semantics, rebuilt the hub pages and closed duplicates and indexing errors. Google users 247 → 532, CTR 2.4% → 4%.
Medical center · Ukraine
+68.75% visibility in the first month
Narrow visibility and a small semantic core at the start. Semantics, landing page structure, metadata and internal linking, then gradual link building.
Answers to your questions
What is the purpose of the robots.txt file?
Robots.txt communicates site crawling rules to search engines and other standard-compliant robots. It can be used to restrict crawling of technical pages, individual directories, internal search results, and other URLs that don't require regular crawling.
The file also contains a link to the sitemap and can include different sets of rules for multiple user agents. The specific configuration depends on the project structure, CMS, and the number of technical URLs.
Where should robots.txt be located?
The file is placed in the root directory of the corresponding host and should be opened at the standard address /robots.txt. For the primary domain, the example would look like https://example.com/robots.txt.
If the site uses separate subdomains, the rules for them should be checked separately. After downloading, it's a good idea to open the file in a browser and ensure the server returns a valid HTTP response.
How to create robots.txt online?
To create a robots.txt file online, select a User Agent, add the necessary Allow and Disallow parameters, specify a Sitemap, and preview the resulting code. After reviewing it, you can copy it or save it as a file.
The online generator is especially convenient when you need to quickly assemble a basic configuration and avoid syntax errors. However, before publishing, you should still test the rules on actual website URLs.
Is it possible to prevent a page from being indexed using robots.txt?
Disabling crawling via robots.txt does not guarantee removal of the address from the index. Search engines can recognize the URL from links and store it without the full page content.
To control indexing, use the meta robots noindex or X-Robots-Tag if the search robot has access to the document and is able to read the corresponding instruction.
Should I add a sitemap to robots.txt?
Adding a sitemap is useful because it gives search engines the direct address of the current XML sitemap. You must specify the absolute URL, including the protocol and domain.
If you have multiple sitemaps, you can add multiple sitemap lines. However, the sitemap itself must contain correct, indexable URLs and be regularly updated along with the site structure.
Is it possible to create separate rules for Google and Yandex?
Yes, you can create separate User-Agent groups for different robots. This is necessary when Googlebot, Yandex, or another crawler require different settings.
With this configuration, it's important to carefully check the order and content of groups. An overly broad rule that accidentally targets the right robot could alter the crawling of a large section of the site.
Does Google support Crawl-delay?
Googlebot doesn't support the Crawl-delay directive in robots.txt. Therefore, adding this line doesn't affect Google's crawl speed.
Other crawlers may process the directive differently. Before using, please check the current documentation for the specific crawler, rather than assuming the same behavior for all systems.
Is it possible to block AI bots using robots.txt?
For individual AI crawlers, you can create a custom User-Agent group and specify prohibited paths. This option is suitable for GPTBot, ClaudeBot, CCBot, and other systems that claim to support robots.txt.
Robots.txt remains a public set of rules for voluntarily adhering robots. Server restrictions, WAFs, and other access control methods are used for strict technical blocking.
A good robots.txt file remains as short and clear as the site's structure allows. Its purpose is to provide crawlers with correct crawling rules without blocking access to the pages and resources needed for searching and proper rendering.
Configure the User-agent, Allow, Disallow, and Sitemap settings in the Seo-Gen generator, test several real URLs, and only then download the finished file. If the site structure changes, return to the robots.txt checker and update the rules along with the rest of the technical SEO settings.
We reply within one business day. No newsletters, no “just a reminder” calls.
He will look at the site himself instead of passing it to a manager.
More on: Robots.txt generator
What is robots.txt and why is it needed?
Robots.txt is a text file containing site crawling rules for search engines and other crawlers. It is located in the root directory of a specific host and helps manage site crawling, technical pages, and URLs that don't require regular crawling.
This file is especially useful on sites with numerous filters, sorting methods, GET parameters, and duplicate URLs. Properly configured access rules reduce the number of unnecessary requests to the server and help direct site crawling to the pages the search engine actually needs.
How does robots.txt affect website crawling?
Robots.txt regulates crawling, so it can't be considered a direct way to control indexing. A search engine can know a page's address from external or internal links, even if the content of that URL is blocked from crawling.
Crawl budget is more important for large resources where it's physically impossible for a crawler to regularly crawl all URL combinations. For a small site, artificially complicating robots.txt to save a few requests is usually unnecessary.
What is usually blocked in robots.txt?
Most often, rules are applied to technical URLs, internal search, individual service sections, shopping carts, checkout pages, and certain filter combinations. An online store may also need to work with sorting and parameters that create a large number of URL duplicates.
There's no universal list of prohibited directories. The admin panel, user account, or internal search are of a clear technical nature, while the directory filter itself sometimes serves as a useful landing page and receives search traffic.
Before banning, there are three things to check:
- whether the URL generates organic traffic and is involved in internal linking;
- does the page have its own search intent and unique content;
- whether the robot requires access to the address for correct rendering or processing of other signals.
After such a check, it is easier to decide where Disallow is needed, where canonical or noindex is needed, and where the page should not be restricted at all.
What shouldn't you blindly close in robots.txt?
Don't automatically disable CSS and JavaScript used to render a page. If a search engine bot doesn't receive the resources it needs, it may view the page differently than a regular user and render its content less effectively.
Robots.txt is also not suitable for protecting sensitive data. The file's address is publicly accessible, so sensitive documents, personal data, and administrative interfaces must be protected by authorization, server restrictions, and proper access rights.
Important landing pages need to be checked separately. A single character error or an overly broad rule can block an entire catalog, services section, or other commercial section of the site.
Robots.txt and noindex – what's the difference?
Robots.txt controls the ability to crawl a URL, while meta robots with the noindex directive or the X-Robots-Tag HTTP header instruct the search engine not to index the document. These mechanisms solve different technical problems.
If a page is blocked via Disallow, the search robot may not retrieve its HTML and may not see the noindex inside. Therefore, blocking crawling and simultaneously relying on meta robots processing without additional verification is not recommended.
How to set up robots.txt for different robots?
For Googlebot, clear User-Agent, Allow, Disallow rules, and a Sitemap link are usually sufficient. If you need to change Google's crawl rate, Crawl-Delay won't work, as Google doesn't support this directive in robots.txt.
For Yandex, standard rules and Clean-param can be used if the site has many URLs with parameters. Bingbot and other search robots are best configured according to their own documentation, especially when non-standard crawl rate limits are required.
Separate groups can also be created for AI crawlers, such as GPTBot, ClaudeBot, Google-Extended, or CCBot. Bot blocking via robots.txt only works for systems that adhere to the published rules, so the file shouldn't be used as a server-side security mechanism.
Robots.txt examples
For a typical website, a short configuration is sufficient, provided there are no complex technical sections. It can include a general robot group, crawl permissions, and a link to the sitemap.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
For an online store, the rules are usually more complex, as they include a shopping cart, internal search, filters, and utility pages. Before applying a template, you should verify the actual project URLs, rather than copying catalog names from another CMS.
User-agent: *
Disallow: /cart/
Disallow: /search/
Disallow: /account/
Sitemap: https://example.com/sitemap.xml
If a separate rule is required for a specific robot, a separate group is created. This approach is convenient when using a common robots generator to prepare multiple sets of rules in a single file.
Common mistakes when creating robots.txt
The most dangerous mistake is accidentally using Disallow: / in a group for a search robot that should crawl the site. Other common mistakes include incorrect sitemap paths, blocking necessary landing pages, attempting to manage indexing solely through robots.txt, and copying settings from another project.
Technical errors include unnecessary resource blocking, incorrect paths, redundant directives, and conflicting rules. Before publishing, it's a good idea to check the file with a robots.txt validator, then test several open and closed URLs.
The verification scheme looks simple:
Rules settings → file generation → URL check → publishing → recheck
This procedure reduces the risk of accidentally closing a section that already receives traffic or should be accessible to a search robot.
How to install and test robots.txt?
The finished file should open at a URL like https://example.com/robots.txt. After downloading, check the HTTP response, file contents, robots.txt syntax, and the availability of several important pages for each individual user agent.
After changing the site structure, it's worth repeating the check. The same applies to moving to a different CMS, launching new filters, changing indexing rules, or making major catalog edits.
For additional verification, you can use Google Search Console, Yandex Webmaster, and a separate robots.txt checking tool. This process helps you spot a technical error before it impacts crawling of a significant portion of your site.