Testing the Security of Your Website – Part 1
The Robots.txt File
Contents
The robots.txt file is a text file you should put in the root directory (folder) of your website (https://www.yourwebsite.com/robots.txt), telling search engines such as Google what to scan and what should not be scanned on your website. It is a good practice to configure this file.
However, some webmasters add the location of the control panel on the list of files not to be scanned by the search engine. Since the robots.txt file is public, anyone can open it to check if there is any unusual directory (folder) listed under “Disallow.”
Consider the real example presented in Figure 2. Why is the/Comment/NewComment directory listed under “Disallow?” That is definitely a place a hacker would open to see what is there. Opening this directory on this particular website produces the login screen shown in Figure 3. Bingo!
Figure 2: Robots.txt file
Figure 3: A login screen found through the robots.txt file
Therefore, you must not add the directory (folder) of your control panel in the robots.txt file.