Six years in numbers
Generator Robots.txt and Sitemap.xml online
The robots.txt and sitemap.xml generator is suitable for launching a new website, migrating to a different domain, changing the structure, adding sections, and performing scheduled technical testing. The finished file can be checked before publication, downloaded, placed on the website, and then submitted to Google Search Console.
Proper setup isn't limited to creating two files. You need to check which pages are allowed to be crawled, which URLs are included in the XML sitemap, and whether they contain redirects, errors, duplicates, noindex, or non-canonical URLs.
This online generator helps prepare essential technical files without manually writing directives and XML markup. Users specify a domain, set access rules for search robots, add the required URLs, and receive ready-made robots.txt and sitemap.xml files for further review.
The tool is suitable for corporate websites, online stores, blogs, catalogs, service websites, and small projects. After generation, the files must be verified against the actual site structure, as the automatic configuration doesn't know the business value of each page or its role in search engine optimization.
The work scheme looks like this:
Search engine → robots.txt → crawl rules → sitemap.xml → list of URLs → page crawling → search engine indexing decision.
This sequence helps us understand the key difference between the files. Robots.txt regulates crawler access, while an XML sitemap provides a list of URLs that search engines can discover and process.
What does the robots.txt generator create?
The robots.txt generator creates a text file with instructions for search robots. It allows you to specify which sections are allowed to be crawled, which technical paths should be excluded, and where the sitemap is located. The standard file address for the main domain is https://example.com/robots.txt.
After generation, the file should be manually verified, especially if the site is already live and receiving organic traffic. An erroneous Disallow: / directive can block the entire site from being crawled, so transferring rules from a test domain to a production domain without verification poses a serious indexing risk.
For most websites, a few clear rules are sufficient. The more complex the structure of an online store, catalog, or multilingual project, the more carefully you need to check filters, URL parameters, service pages, and different URL versions.
What does the sitemap.xml generator create?
The sitemap generator creates an XML sitemap with a list of pages that should be passed to search engines for discovery and crawling. It's recommended that the sitemap.xml include canonical URLs that respond with a 200 OK response, are open for indexing, and are accessible by the primary version of the domain.
Having a page in your sitemap.xml doesn't guarantee its indexing and doesn't directly impact rankings. Google independently evaluates page content, canonical tags, Meta Robots, X-Robots tags, internal links, HTTP response, and other signals before indexing a URL.
A sitemap is especially useful for new websites, large catalogs, online stores, and projects with a deep structure. It also helps quickly identify pages that currently have few internal links, although it can't replace proper interlinking.
What it includes
robots.txt generator
An online robots.txt generator for setting up user-agent, allow, disallow, and sitemap settings. Create a file, check the rules, and download your finished robots.txt file for free.
sitemap.xml generator
Sitemap.xml generator for creating an XML sitemap online. Enter your URL, get a ready-made sitemap, and add it to robots.txt and Google Search Console.
Common Mistakes When Setting Up Robots and Sitemaps
Most problems arise after migrations, structure changes, or automatic URL generation. Files continue to technically work, but they begin to contain old URLs, block useful sections, or pass pages to search engines that are no longer intended for indexing.
It's better to search for errors by patterns rather than by a single URL. If one filter is included in the sitemap, it's likely that all URLs of that type have a similar issue.
Regular testing is especially necessary for online stores and other dynamic projects. Their structure changes more frequently, so the initial settings gradually become outdated.
Accidental blocking of the entire site
The most dangerous error in robots.txt is the Disallow: / directive for the generic User-Agent on a production domain. This often persists after migrating a project from a test environment, where a complete disallow for crawling was justified.
Before releasing, you should separately check robots.txt through the public domain. Checking a local file in the repository doesn't guarantee that the server is serving the same version.
After fixing the issue, it's worth checking the availability of the main pages and the site's status in Google Search Console. It will take some time for the search engine robot to receive the updated rules and re-crawl the URL.
Adding closed pages to sitemap.xml
A sitemap shouldn't systematically contain URLs that the owner simultaneously blocks from crawling without clear reason. Such a configuration sends conflicting signals to search engines and complicates technical diagnostics.
If a page isn't needed in search results, you should determine the correct way to handle it. This could include noindexing, deleting, redirecting, authorizing, or excluding it from sitemap generation.
After changing the rules, check the XML generation source. Otherwise, the manually removed URL will reappear during the next automatic map update.
Adding redirects and 404 pages
The sitemap should contain final working URLs, not old addresses with 301, 302, or 404 errors. The presence of a large number of such pages indicates that the sitemap is not synchronized with the current structure.
After migration, old URLs should be removed from the XML and replaced with new canonical addresses. The redirect itself may remain valid for users and external links, but it's usually unnecessary in the sitemap.
The same applies to deleted pages. If a URL returns a 404 and should no longer exist, regularly submitting it via a sitemap is pointless.
Adding non-canonical URLs
It's best to use the same version of the URL in the sitemap as the page's canonical URL. Mismatches often occur between HTTP and HTTPS, www and non-www, addresses with and without a slash, and URLs with parameters.
If the XML systematically contains non-canonical addresses, the generation mechanism itself needs to be corrected. Manually cleaning the file will only resolve the issue until the next update.
Google takes into account the presence of a URL in a sitemap as one of the signals when choosing a canonical version, although the search engine determines the final canonical version itself.
Using a Sitemap Instead of Internal Linking
An XML sitemap helps search engines discover pages, but a healthy website structure should still connect important documents with internal links. If a valuable landing page exists only in the sitemap, it's worth checking its place in the architecture.
Internal links provide search engines with additional context, anchor text, and links between sections. They also allow users to navigate to the page in the usual way.
Orphaned pages should be analyzed separately. Sometimes the lack of internal links is accidental, and sometimes the URL shouldn't be used for search engine optimization at all.
Incorrect lastmod
The lastmod date should reflect a real, significant page update. Automatically assigning the current date to each URL every time it's generated creates a constant stream of changes that never actually happened.
Google uses lastmod if the value remains accurate and corresponds to actual changes. Updates to the main content, structured data, or significant links are considered significant, while a simple date change in the footer is not.
If the CMS doesn't store a reliable date, the optional field can be omitted. An inaccurate signal doesn't improve the sitemap's quality.
I recommend treating robots.txt and sitemap.xml as related but distinct parts of a site's technical setup. Robots.txt controls crawler access, while sitemap.xml passes a list of URLs we want to show to search engines. Before publishing, I check the response code, canonical, Meta Robots, and the availability of each page type. In the sitemap, I leave the loc attribute and, if an exact date is available, the lastmod attribute. I don't configure the changefreq or priority for Google, as the search engine ignores them.
How to use the Robots and Sitemap generator?
Start with the primary domain version and check the protocol used on the site. Then, configure robots.txt rules, add the required URLs to sitemap.xml, and preview the results before downloading the files.
The generator reduces manual work, but it doesn't make SEO decisions for the website owner. It's important to understand in advance which pages are intended for organic search, and which are technical, duplicate, or restricted sections.
After generating the sitemap, check the contents of both files together. The sitemap should not pass to search engines addresses that are also blocked by crawl rules without an obvious reason.
Specify the website domain
Enter the primary HTTPS version of the domain if it is used as the canonical version. For example, for a site like https://example.com, don't generate both http://example.com and https://www.example.com if they redirect to the primary version.
Ensure a consistent URL format before generating an XML sitemap. Duplicate URLs are particularly common due to trailing slashes, case inconsistency, GET parameters, and different versions of the same domain.
If the project is multilingual or operates on multiple subdomains, rules should be developed taking into account the site's architecture. Each individual host can have its own robots.txt file.
Set up robots.txt rules
Determine which URLs the search engine should crawl and which technical sections can be excluded. For basic configuration, use the User-agent, Disallow, Allow, and Sitemap directive with the full sitemap address.
Don't copy another project's robots.txt file without checking its structure. Identical CMS or template names don't mean the sites have the same filters, search pages, utility sections, and indexing rules.
After setting up, check your homepage, categories, products, services, and several technical URLs. This set will quickly show whether the ban has affected any useful landing pages.
Add URL to sitemap.xml
Add pages to the sitemap that are accessible via the canonical URL, return a 200 OK response, and are intended for indexing. For a sitemap, the quality of the list is more important than the maximum number of URLs.
Don't automatically add URLs just because the CMS can detect them. Filters, internal search, test pages, duplicates, and technical parameters often shouldn't be included in the XML sitemap.
Before bulk generation, it's helpful to check URL patterns by type. This reduces the risk of thousands of unnecessary URLs appearing in large directories.
Configure Sitemap Settings
For each URL, you must pass the correct absolute address via loc. The lastmod tag should only be added if the system stores the exact date of the page's last significant change.
There's no need to configure changefreq and priority for Google. Their values are ignored, so these fields don't affect crawl frequency, indexing, or page ranking.
If there's no exact update date, it's best to leave lastmod blank. A clean XML sitemap with correct URLs is more useful than a file with formally filled in but inaccurate technical parameters.
Generate and download files
After setting up, create a robots.txt and sitemap.xml file, then review the contents before publishing. Check the domain, protocol, key directives, URL list, and the absence of obvious technical pages.
The finished files can be downloaded and hosted on the server. After downloading, be sure to open both URLs in a browser and ensure the server returns the file without errors, authorization, or unexpected redirects.
Then add the sitemap.xml to Google Search Console and monitor the processing results. Errors in the report help you identify unreachable, non-canonical, or malformed URLs.
Answers to your questions
Is it possible to create robots.txt and sitemap.xml online for free?
Yes, you can generate a basic robots.txt file and sitemap.xml file online without any separate software. For a small website, simply specify the domain, configure access rules, and add canonical pages that should be included in the XML sitemap.
After generation, a technical review is still required. The tool creates a file based on the specified parameters, but it doesn't automatically determine which categories, filters, or service pages are needed for a specific project.
Before uploading, check the domain, protocol, and basic URL patterns. Be especially careful with existing sites, where an error could impact already indexed pages.
Does a small website need a sitemap.xml?
For a small website, a sitemap.xml isn't considered a mandatory requirement for indexing if all pages are accessible through clear internal linking. Search engines can discover such URLs through a simple crawl.
At the same time, the map remains a convenient technical resource. It can be used to submit current URLs and then monitor file processing in Google Search Console.
On a new website, a sitemap is especially useful after launch. It helps quickly show search engines the project's structure, although the final decision on whether to index each URL remains with the search engine.
How many URLs can be added to sitemap.xml?
A single standard sitemap can contain up to 50,000 URLs with an uncompressed file size of up to 50 MB. If one of these limits is reached, the URLs must be distributed among multiple sitemaps.
For a large website, it's common to create separate sitemaps for different page types. For example, products, categories, and articles can be placed in different XML files.
All such maps are then linked via the Sitemap Index. Search engines only need to retrieve the index file to discover other maps.
Can I add multiple sitemaps?
Yes, multiple maps are used on large and structurally complex websites. This approach helps adhere to technical limits and simplifies control over individual page types.
Each child sitemap must remain accessible and contain up-to-date canonical URLs. Deleted sitemaps should be promptly removed from the index file.
Splitting is also useful for diagnostics. If Google reports errors only in the product page, you can more quickly find the problem in the corresponding generation template.
Should I add sitemap.xml to robots.txt?
Specifying sitemap.xml in robots.txt is recommended because the search engine robot receives the sitemap's direct URL when accessing the rules file. This is achieved by using the Sitemap directive with an absolute URL.
You can also submit your sitemap via Google Search Console. These methods don't conflict and are used simultaneously on most websites.
After changing your domain, check both locations. The old address often remains in robots.txt or the webmaster dashboard after the migration is complete.
Are changefreq and priority needed in sitemap.xml?
Google doesn't need these parameters. The search engine ignores the <changefreq> and values, so they don't affect crawl frequency, indexing, or page ranking.
In a current sitemap, it's sufficient to pass the correct URL via <loc>. <lastmod> makes sense to add when the site can indicate the exact date of the last significant page change.
If the old generator continues to generate changefreq and priority, this in itself doesn't make the file invalid. For Google's modern SEO setup, spending time managing these values is pointless.
Is it possible to prevent a page from being indexed using robots.txt?
You shouldn't use robots.txt alone to remove a page from the index. Disallow prohibits or limits crawling, so search engines may not access Meta Robots and see the noindex directive.
To prevent indexing, a page typically must remain accessible to the robot and contain the appropriate directive. The specific scheme depends on the current URL status and the site owner's goals.
If the page is already in search results, you need to check the reasons for its appearance and how to remove it. Simply adding the path to Disallow may not produce the expected result.
Should I add noindex pages to my Sitemap?
A standard XML sitemap of indexed pages should not contain a noindex URL. Adding such a URL simultaneously informs the search engine about the page via the sitemap and prohibits its indexing via another technical signal.
If noindex is set temporarily, you need to understand what happens next. Once the restriction is lifted, the page can be returned to automatic sitemap generation.
If noindex pages appear en masse in XML, it's best to correct the generation rule. Manually removing individual URLs won't fix the problem.
How often should I update my sitemap.xml?
The sitemap should be updated after new indexed pages are added, old URLs are removed, and the site structure is significantly changed. For dynamic projects, it's best to automate this process via a CMS or server-side generation.
Don't recreate the file just to change the date without actually editing it. If lastmod is used, the date should correspond to the last significant change to the page, not the time the sitemap was last generated.
After major migrations, the sitemap is checked separately. It should not contain the old domain, old protocol, redirects, or deleted pages.
More on: Generator Robots.txt and Sitemap.xml
What is robots.txt and why is it needed?
Robots.txt stores instructions for crawlers that request pages and other website resources. A search bot typically accesses this file before crawling and checks the rules specified for its User Agent. The file is located in the root directory of the corresponding host.
Robots.txt can be used to restrict crawling of technical sections, parameters, internal search results, and other URLs that shouldn't consume crawl budget. However, disabling crawling isn't a reliable way to remove a known page from the index.
On a small website, the file often contains only a few directives. For a large online store, the rules may be more complex due to filters, sorting, search, a user account, a shopping cart, and various technical parameters.
What directives are used in robots.txt?
Basic directives define the search engine robot and access rules for specific paths. For most sites, User-agent, Disallow, Allow, and Sitemap are used. These are sufficient to create a clear basic configuration and point the search engine to the XML sitemap address.
Rules should be written with real URLs in mind. Disabling an overly broad path may accidentally affect useful categories, product cards, or service pages if their URLs are located within the same directory.
Before publishing, it's helpful to test several URLs of each type. This approach quickly identifies conflicts between robots.txt, canonical, and noindex rules and the actual site structure.
User-agent
The user-agent determines which robot the rules below apply to. The * value indicates that the group of instructions is intended for all crawlers that support the robots.txt standard.
Example of a basic entry:
User-agent: *
For projects with specialized requirements, rules can be separated for individual robots. This shouldn't be done unnecessarily, as a large number of groups complicates file maintenance and increases the risk of inconsistencies after changes to the site structure.
Every technical edit must take into account the order and scope of the rules. If a site uses separate settings for Googlebot, other search robots, or AI bots, they should be checked separately.
Disallow
Disallow specifies a path that the selected crawler should not crawl. For example, this directive can be used to close internal search, technical settings, or the administrative section if they are accessible via public URLs.
Example:
Disallow: /admin/
Don't close directories containing useful landing pages without proper verification. Search engines may stop loading their content, internal links, and other elements necessary for proper website processing.
If the goal is to remove a page from search results, you need to check crawlability and the noindex directive separately. Blocking in robots.txt and disabling indexing solve different technical problems.
Allow
Allow allows bypassing of a specific path within a broader restricted section. This setting is used on sites with complex structures where a common URL pattern needs to be blocked, but individual resources within it should remain accessible.
Exceptions should only be used after validating actual URLs. The more overlapping Allow and Disallow rules, the more difficult it is to maintain the configuration when URL patterns change.
After implementation, it's worth testing several allowed and blocked URLs. This test will show whether the actual behavior matches the expected robots.txt logic.
Sitemap
The Sitemap directive tells the crawler the absolute URL of the XML sitemap. It is typically placed in robots.txt on a separate line and contains the full URL with protocol and domain.
Example:
Sitemap: https://example.com/sitemap.xml
For a large project, a Sitemap Index, which contains links to multiple XML sitemaps, can be used instead of a single file. This option is convenient for large catalogs, separating products, categories, articles, or language versions.
After changing the sitemap address, you should update your robots.txt file and Google Search Console data. The old path should not be the only source of sitemap information.
What is the difference between Disallow and noindex?
Disallow refers to crawling: the robot is advised not to load the page at the specified path. Noindex refers to indexing and tells the search engine that the crawlable page should not be stored in the search index.
If a URL is already known to Google through external links, internal linking, or an old sitemap, disabling it in robots.txt alone may not be enough to remove the address from search results. The robot sees the URL itself, but it can't always load the page and read Meta Robots.
Therefore, you first need to determine the goal: reducing unnecessary crawling or removing a page from search results. After that, you can choose robots.txt, Meta Robots, X-Robots-Tag, URL removal, redirect, or another approach.
What is an XML Sitemap and why do you need a sitemap?
Sitemap.xml contains a list of URLs that the website owner considers relevant and crawlable. This file helps search engines find pages without having to wait for each one to be discovered through traditional internal linking.
An XML sitemap is especially useful on a new website, where search engines don't yet know most of the URLs. It's also essential for large online stores, catalogs, news projects, and sites with regularly updated pages.
For a small website, a sitemap is also useful as a technical source of control. It makes it easy to compare the list of submitted URLs with the actual indexed pages, canonical URLs, and Google Search Console data.
What data does sitemap.xml contain?
The main sitemap.xml entry contains the page address in the loc tag. Additionally, lastmod can be passed if the site can indicate the actual date of a significant document change. For Google, this data is of practical use for standard XML Sitemap processing.
The changefreq and priority fields are still present in older sitemap generators and are part of the historical sitemap format, but Google ignores them. There's no need to add them for SEO or configure them for each page. This is clearly stated in the current Google Search Central documentation.
The current basic structure looks like this:
| Tag | What does it contain? | How to use |
|---|---|---|
| loc | Absolute URL of the page | Specify the primary canonical address |
| lastmod | Date of last significant change | Transmit only if accurate data is available |
This is sufficient for a standard sitemap. Additional extensions can be used for images, videos, news content, and localized versions of pages if the project truly requires them.
loc
loc contains the absolute address of the page, including the protocol and domain. For a production site, it's recommended to use the canonical HTTPS version without an intermediate redirect and without unnecessary GET parameters, unless they form independent indexable pages.
In a sitemap, don't mix HTTP and HTTPS, or www and non-www versions of the same website unless absolutely necessary. This structure creates unnecessary URLs and makes it difficult to understand the primary version of the pages.
Google recommends specifying fully qualified absolute URLs and including in the sitemap the addresses the owner wants to appear in search results. For duplicate pages, it's preferable to submit the selected canonical version.
lastmod
lastmod shows the date of the last significant change to the page. This could be an update to the main text, structured data, links, product, features, or another part of the document that actually changed its content.
Don't automatically change the lastmod tag for all URLs daily if the pages haven't actually been updated. Google uses this tag when the data remains accurate and can be verified by actual page changes.
On dynamic websites, it's best to take the date from the actual update history. If the CMS can't reliably determine the time of the last significant change, the tag can be omitted instead of using a fictitious date.
What pages should be added to the Sitemap?
The XML sitemap should include URLs that are valuable to the user and intended for search engine indexing. These typically include categories, product cards, services, articles, informational pages, and other landing pages.
It's advisable to check each URL against several criteria: 200 OK response, self-canonical, no noindex, crawlability, and compliance with the primary domain version. This approach makes the sitemap a useful technical resource rather than a simple list of all CMS URLs.
For large sites, it's best to generate the list automatically using standard rules. Manually updating thousands of URLs quickly leads to outdated addresses and errors.
Which URLs should I include?
A sitemap typically includes canonical pages that are indexable and return a valid server response. For an online store, this could include main categories, indexable subcategories, products, and useful informational materials.
The following characteristics are worth checking:
- The URL returns a 200 OK code and opens without an intermediate redirect;
- the page has a correct canonical and does not canonically link to another document;
- Meta Robots or X-Robots-Tag does not contain a noindex prohibition;
- the page is accessible to search robots and corresponds to the main version of the domain;
- The URL is needed by users and has a clear role in the site structure.
Once you've compiled your list, it's helpful to compare it with your site crawl. This can help you identify pages that are in your sitemap but missing from your internal linking.
What URLs should not be added?
Sitemaps typically don't include pages with 301 or 302 redirects, 404 errors, noindex, technical duplicates, or URLs with canonical addresses. Avoid passing obviously unnecessary combinations of filters, sorting, and internal parameters to search engines.
Most often excluded:
- old URLs that redirect to current pages;
- search, shopping cart, authorization and user account pages;
- technical filters and sorting without separate search demand;
- non-canonical versions of addresses that create duplicate pages;
- test or closed materials that are not intended for organic search.
After removing unnecessary URLs, it's important to check your internal links. If the technical URL continues to appear frequently on the site, cleaning the sitemap alone won't be enough.
Sitemap XML Limitations
A single XML file isn't designed to store an unlimited number of pages. Standard sitemaps have technical limitations on the number of URLs and file size, so large projects use multiple sitemaps and a separate Sitemap Index.
It's important to consider this limitation when designing the generation process. If an online store contains hundreds of thousands of products, categories, and other indexed documents, a single file will quickly cease to meet the format requirements.
It's more practical to separate maps logically. This approach simplifies diagnostics and helps quickly identify which page type is experiencing errors.
How many URLs can be added to one sitemap.xml?
A single sitemap.xml file can contain up to 50,000 URLs, and the uncompressed file size must not exceed 50 MB. If any of these limits are reached, the list must be split into multiple files. These limits are detailed in Google's current documentation.
For most corporate websites, this limit is unattainable. This limitation becomes more relevant for large marketplaces, online stores, media outlets, aggregators, and projects with a large number of dynamic pages.
You should monitor both the number of URLs and the file size. Large entries with additional data may reach the size limit before the number of URLs reaches 50,000.
What to do if there are more than 50,000 pages?
If the number of indexed pages exceeds the set limit, multiple sitemap.xml files and a Sitemap Index are created. The index file contains the addresses of individual sitemaps, which the search engine then processes separately.
For example, a large online store might separate cards into categories, products, articles, and other document types. This separation is also convenient for technical analysis, as errors in a specific group are more quickly identified in the webmaster dashboard.
The simplified diagram looks like this:
Sitemap Index → sitemap-products.xml → sitemap-categories.xml → sitemap-blog.xml → individual indexed URLs.
When generating automatically, ensure that all child files are up-to-date. Deleted maps should not remain in the Sitemap Index.
Where to upload robots.txt and sitemap.xml?
Once generated, both files need to be placed at URLs accessible to search engines. In a standard configuration, robots.txt is located at the root of the host, and sitemap.xml is usually placed on the main domain at a permanent URL.
After downloading, open the files in a browser and check the server's response. The search engine crawler should retrieve the content without requiring authorization, encountering 404 errors, loop redirects, or being blocked by security systems.
If your site uses a CDN, WAF, or a complex server configuration, double-check Googlebot access. Sometimes the file exists, but certain security rules prevent it from being retrieved normally.
Where should robots.txt be located?
For the main domain, the standard address looks like this: https://example.com/robots.txt. The file is specific to a specific host, so the rules for the main domain don't automatically override the separate robots.txt file for another subdomain.
After uploading, check the exact URL manually. The server should return a text file with the correct directives without redirecting to an HTML page or login page.
When migrating a site, be especially careful in checking the old rules. The production domain sometimes retains a staging environment configuration with complete disallowed crawling.
Where should sitemap.xml be located?
A typical XML sitemap address looks like https://example.com/sitemap.xml. The name may vary if the CMS creates multiple sitemaps or a sitemap index, but the path used should remain accessible and stable.
If the sitemap is split into multiple files, each one should be checked. The Sitemap Index should not contain any links to deleted or inaccessible documents.
After changing the structure, it's advisable to update the sitemap automatically. A manual file quickly becomes outdated when products, articles, or new sections are regularly added to the site.
How to link robots.txt to sitemap.xml?
A separate directive with the full sitemap address is added to robots.txt:
Sitemap: https://example.com/sitemap.xml
If you're using a Sitemap Index, it's best to specify the URL of the index file. Google, Bing, and other major search engines support the Sitemap field in robots.txt. The URL must be absolute, including the protocol and host.
After changing a domain or protocol, you should check this line separately. The old sitemap address in robots.txt often goes unnoticed after a site migration.
How to add a Sitemap to Google Search Console?
After publishing the sitemap, open the relevant Google Search Console resource and add the current sitemap URL to the appropriate section. The system will process the file and display the status, detected pages, and any errors.
Having a sitemap in robots.txt doesn't prevent you from submitting it separately through Search Console. This panel is useful for diagnostics because it shows processing issues and helps track changes after updating the structure.
Don't submit the same unmodified sitemap multiple times. Google specifically recommends against submitting the same file multiple times a day without modifications.
How to check robots.txt and sitemap.xml after generation?
Post-generation verification is essential, especially if the site is already ranking and receiving organic traffic. One incorrect directive can affect the accessibility of a large section, and an erroneous sitemap will regularly send unnecessary URLs to search engines.
It's best to check not only the files themselves, but also the pages affected by their rules. This helps detect discrepancies between robots.txt, sitemap.xml, Meta Robots, canonical, and actual HTTP responses.
For a large project, it's useful to automate such testing. Changes to the structure, CMS, and templates can gradually disrupt an initially correct configuration.
Checking robots.txt
First, make sure the file opens at the expected address and contains the current version of the rules. Then, check the home page, main categories, products, services, articles, and technical sections.
Particular attention should be paid to prohibitions that are too broad. Disallow: / closes the site crawl for the corresponding User-agent, and an inaccurate mask can affect more URLs than intended.
Additionally, check your sitemap directive and the availability of important resources. If CSS or JavaScript is required to render a page, accidentally blocking them can impair search engine crawlers' understanding of the page.
Checking sitemap.xml
In the sitemap, you should check HTTP statuses, canonical, noindex, redirects, duplicates, and compliance with the primary domain version. Errors are especially noticeable after migrations, mass URL changes, and directory structure reorganizations.
For each page type, it's advisable to manually check several examples and then perform a bulk technical analysis. If the sitemap contains thousands of URLs, a single check won't reveal a systemic template error.
Also, check the XML validity and lastmod accuracy if this tag is used. Incorrect dates, corrupted markup, or inaccessible daughter maps can reduce the quality of technical signals.
Should AI bots be blocked in robots.txt?
Some website owners set separate rules for GPTBot, Google-Extended, ClaudeBot, and other AI crawlers. The decision depends on whether the owner wants to allow such systems to retrieve content and the requirements for the specific project.
Rules can be added as separate User-Agent groups. Before doing so, it's recommended to check the current bot name and the official documentation for the relevant service, as the list of AI bots and their purposes are subject to change.
Robots.txt functions as a standard that conscientious crawlers voluntarily adhere to. If technically necessary to deny access to certain clients, server restrictions, WAFs, request analysis, and other control mechanisms are additionally applied.
For SEO purposes, it's not a good idea to massively block unknown crawlers without proper analysis. Some crawlers are linked to search engines, monitoring services, or tools the website owner uses.
Specify your domain, set up crawl rules, and create ready-made robots.txt and sitemap.xml files for your website.
The Robots and Sitemap Generator helps you prepare two basic technical files and immediately test their combined logic. Robots.txt defines crawling rules, while sitemap.xml provides up-to-date canonical URLs that search engines can crawl and evaluate for indexing.
For a modern sitemap, focus on correct URLs and an accurate lastmod if the system actually knows the date of a significant page change. Configuring changefreq and priority for Google is not necessary, as the search engine ignores these values.
Before publishing, check disallow, noindex, canonical, HTTP status codes, redirects, and the primary domain version. After uploading, open both files on your live site and submit the updated sitemap.xml to Google Search Console.
We reply within one business day. No newsletters, no “just a reminder” calls.
He will look at the site himself instead of passing it to a manager.