Block robots from folder and sub folders

1

I think I’ve got this right, but need to check…

folder structure is like this…

level1
level1 > sub 1
level1 > sub 2
level1 > sub3

If I want to stop all folders contained within > sub1 do I add a robots.txt file to > level1 with the following in it?

User-agent: *
Disallow: sub1/*

Or, do I put that robots.txt file in > level1 > sub1 ?

Ta.

2

The robots.txt file must be at the root level of the website.

Format and location rules:

  • The file must be named robots.txt
  • Your site can have only one robots.txt file.
  • The robots.txt file must be located at the root of the website host that it applies to. For instance, to control crawling on all URLs below http://www.example.com/ , the robots.txt file must be located at http://www.example.com/robots.txt . It cannot be placed in a subdirectory ( for example, at http://example.com/pages/robots.txt ). If you’re unsure about how to access your website root, or need permissions to do so, contact your web hosting service provider. If you can’t access your website root, use an alternative blocking method such as meta tags.
  • A robots.txt file can apply to subdomains (for example, http://website.example.com/robots.txt ) or on non-standard ports (for example, http://example.com:8181/robots.txt ).
  • Comments are any lines that start with a # mark.

They (google) have an excellent tester:

1 Like
3

Thanks Teefers. I completely missed that!

2 Likes