As an online store owner, you want your Magento 2 store to be visible and easily discoverable by search engines. One way to achieve this is by creating and properly configuring your robots.txt file. In this article, we’ll discuss what robots.txt is, how it works, and how to optimize it for SEO.

Related blogs

1. Is magento 2 SEO optimization worth it in 2023?

2. How to Use Magento 2 Blog Posts to Build Brand Awareness

What is robots.txt?

Robots.txt is a text file that instructs search engine robots or crawlers how to crawl and index your website’s pages. It tells the search engines which pages to crawl and which pages to exclude. The robots.txt file resides in the root directory of your website and can be accessed by adding “/robots.txt” to your domain name (e.g., https://store.webbytroops.com/robots.txt).

How does robots.txt work?

When a search engine crawls your website, it looks for the robots.txt file in the root directory. If it finds the file, it reads the instructions and follows them accordingly. If it doesn’t find the file, it assumes that all pages on the website are open for crawling and indexing.

Configuration of Robots.txt

1. Navigate to Content > Configuration.

2. Locate the line Global in the grid that has been opened, then select the Modify link to bring up the Search Engine Robots panel.

3. Choose the necessary Default Robots from the list below:

• INDEX, FOLLOW: Regularly, search engines will index your sites and monitor any updates
• NOINDEX, FOLLOW: Although a search engine bot won’t index your website, it will nevertheless keep an eye out for updates.
• INDEX, NOFOLLOW: The search engine bot will index your store only once and never return to monitor updates. 
• NOINDEX, NOFOLLOW: This setting will prevent search engine bots from accessing your store.
  You can enter the required custom instructions in the robots.txt file’s Modify custom instruction line.

4. You can enter the required custom instructions in the robots.txt file’s Modify custom instruction line.

5. Click the Reset To Default option to restore the default settings and erase all of your customised instructions.

6. Remember to Save Configuration so that the changes will take effect.

Optimizing Magento 2 Robots.txt for SEO

Here are some tips for optimizing your Magento 2 robots.txt file for better SEO:

1. Allow access to important pages: Ensure that the important pages of your website are accessible by search engines. These include your homepage, category pages, product pages, and other pages that you want to appear in search results.
2. Disallow irrelevant pages: You should disallow pages that are not relevant to your website’s content or are of low quality. These include login pages, admin pages, checkout pages, and other pages that are not meant for public viewing.
3. Block duplicate content: Duplicate content can harm your SEO efforts as search engines penalize websites that have identical content on multiple pages. Use the “Disallow” directive to block duplicate content pages, such as print pages, archive pages, or tags.
4. Allow CSS and JS files: Ensure that CSS and JavaScript files are accessible to search engine crawlers. These files help search engines to understand the structure of your website and improve its indexing.
5. Use relative URLs: Use relative URLs instead of absolute URLs in your robots.txt file. This helps search engines to understand the structure of your website and crawl it more efficiently.

Here’s an example of a basic Magento 2 robots.txt file:

User-agent: *
Disallow: /checkout/
Disallow: /customer/
Disallow: /wishlist/
Disallow: /compare/
Disallow: /catalogsearch/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /.js$
Disallow: /.css$
Sitemap: https://store.webbytroops.com/robots.txt

In this example, we have disallowed search engine crawlers from accessing the checkout, customer, wishlist, compare, catalogsearch, sendfriend, and review pages. We have also blocked JavaScript and CSS files from being crawled.

What happens if I don’t have a robots.txt file on my Magento 2 website?

Not having a robots.txt file on your Magento 2 website means that search engine crawlers will assume that all pages on your website are open for crawling and indexing. This can lead to indexing irrelevant or low-quality pages, duplicate content, and other issues that can negatively impact your website’s SEO performance. It’s essential to have a properly configured robots.txt file to guide search engines to crawl and index the right pages of your website.

Can I use robots.txt to block specific search engines from crawling my website?

Yes, you can use the robots.txt file to block specific search engines from crawling your website. However, it’s not recommended as search engines can still find ways to access your website’s pages through other sources. It’s best to use other methods like no-index tags or password protection to block access to specific pages.

How often should I update my robots.txt file?

Updating your robots.txt file depends on your website’s content and changes in the page structure. If you add new pages to your website or change the URLs of existing pages, you need to update your robots.txt file accordingly. It’s also recommended to periodically review your robots.txt file to ensure it’s properly configured and optimized for search engine crawlers. However, if there are no significant changes in your website’s content or structure, there’s no need to update your robots.txt file frequently.

In the end

Optimizing your Magento 2 robots.txt file is an essential step in improving your website’s SEO. By properly configuring your robots.txt file, you can ensure that search engines can easily crawl and index your website’s important pages while excluding irrelevant and duplicate content. Keep in mind that improper configuration of robots.txt can harm your SEO efforts, so it’s important to understand its functionality and use it wisely.