How to write a robotstxt file – 8 Easy Steps – Coddu

How To Write a robotstxt File

This tutorial belongs to robots.txt file. robots.txt file use to make restriction / allow files, folders/directories for search engine. We will read about to create and write robots.txt file step by step.

Tutorial Details:

Topic: robots.txt
Difficulty: Average
Estimated Time Completion: 10 Minutes
How to do:

1. Open a text editor like notepad and create a robots.txt file. Filename should be lower case and host to the root of web server i.e. https://coddu.com/robots.txt

2. User-agent: robot

3. Edit robots.txt file and disallow files and folders / directories.

 

# robots.txt for https://coddu.com

User-agent: *

#Disallow: /filename.extension or /directoryname

Disallow: /index.html
Disallow: /includes
Disallow: /admin
Disallow: /settings

 

4. “User-agent: *” use to specify all robots.

5. “User-agent: A*” use to specifiy that robots which they start with letter A.

6. “User-agent: robot” use to specify which that start with “robot” letters.

7. Disallow syntax start from front slash “/” i.e. “Disallow: /file or /directory name and path”. If don’t allow robots to view any file than use “Disallow: /”.

8. To allow Googlebot to see all site than use: “User-agent: Googlebot”

Please feedback after reading this article.
Thanks For Time