Digital Marketing

08

November

Robots.txt file Structure

Robots.txt is a text file used to prevent specific websites or resources like pdf documents, docs, or other images from being indexed. Robots.txt file instructs robots of the search engines not to search for specific pages.

The Major Search Engines (Google, Yahoo, Bing) read our site's robots.txt file, and then determine which pages should be listed and which pages shouldn't be indexed.

Why Should You Use Robots.txt file?

As we've already discussed, robots.txt file is a crucial file utilized to block specific websites or pages on the web site.

Let's look at three principal reasons to use the robots.txt file. It includes one of them:

Block Non-Public Pages: Sometimes, we are required to block our admin pages that are not intended for users who are not random like login page, profile page and site settings. We don't want to index these pages , so this is where we utilize robots.txt to block these pages. robots.txt file to block these pages.

Block Resources: we are required to block our resources like pdf documents, files from docs, images, CSS, js and more. If we don't want Google to index these resources on our site, so we make use of robots.txt file to prevent these media files from being indexed.

Block Particular Character in URL: When our website is affected by viruses and then our website will create random URLs, and it's indexed in search engines. It can affect our rankings in the search engines. To avoid these types of issues, we utilize the robots.txt file to block the characters to be used in URLs. It is typically used on dynamic websites, for instance, which are built using PHP frameworks like Codeigniter and Laravel.

Robots.txt file for CodeIgniter/Laravel PHP Framework

User-agent: *

Disallow: /index.php/

Disallow:/admin/

Disallow:/media/

Sitemap: your-website-url/sitemap.xml

In this structure, we are disallowing admin pages of our websites, media files of our websites such as CSS, js, image files, and pages that have index.php in URL. At last, we are allowing our sitemap.xml file to index.