Does robots.txt affect the results?

Yes, the robots.txt file directly affects crawlability and, consequently, page indexing.

If a URL or an entire section of a website is blocked in robots.txt, a search engine crawler may simply be unable to access the page's content. In this case, even submitting the URL through an indexing service will not yield the desired results, as the crawler will be physically unable to crawl it correctly.

How does this work

robots.txt controls crawl access, not indexing directly:

• If access is allowed → the robot can visit the page and analyze it;

• if access is denied → the robot cannot read the page content;

• Without scanning, indexing is either impossible or severely limited.

An important nuance

Sometimes a URL can be indexed even if it is blocked in robots.txt, but:

• the search engine does not see the page content;

• only the URL without the snippet may be displayed in the search results;

• The page is considered "unclear" for analysis and may be removed later.

Why sending a URL doesn't help when blocked

Indexing services or manual URL submission:

• do not bypass robots.txt rules;

• do not allow the search robot to “bypass the ban”;

• only speed up the attempt to visit the page if access is allowed.

If access is denied, the robot simply will not be able to complete the task.

What needs to be done correctly

Before submitting the URL, it is important to:

• check if the path is blocked in robots.txt;

• make sure that the required directories are allowed for the search engine User-agent;

• open access for important pages (Allow/Disallow are configured correctly);

• Check the page's availability for Googlebot/YandexBot.

Result

Robots.txt is one of the basic technical filters. If a page is blocked by it, no amount of submitting the URL for indexing will replace crawling access. First, you need to open the page to search robots, and only then does it make sense to speed up indexing.