How to write a robotstxt file – 8 Easy Steps – Coddu
How To Write a robotstxt File
This tutorial belongs to robots.txt file. robots.txt file use to make restriction / allow files, folders/directories for search engine. We will read about to create and write robots.txt file step by step.
Tutorial Details:
Topic: robots.txt
Difficulty: Average
Estimated Time Completion: 10 Minutes
How to do:
1. Open a text editor like notepad and create a robots.txt file. Filename should be lower case and host to the root of web server i.e. https://coddu.com/robots.txt
2. User-agent: robot
3. Edit robots.txt file and disallow files and folders / directories.
# robots.txt for https://coddu.com
User-agent: *
#Disallow: /filename.extension or /directoryname
Disallow: /index.html
Disallow: /includes
Disallow: /admin
Disallow: /settings
4. “User-agent: *” use to specify all robots.
5. “User-agent: A*” use to specifiy that robots which they start with letter A.
6. “User-agent: robot” use to specify which that start with “robot” letters.
7. Disallow syntax start from front slash “/” i.e. “Disallow: /file or /directory name and path”. If don’t allow robots to view any file than use “Disallow: /”.
8. To allow Googlebot to see all site than use: “User-agent: Googlebot”

Please feedback after reading this article.
Thanks For Time
Leave a Reply
You must be logged in to post a comment.