Whitehouse.gov Robots.txt

Excluded Directories

Update below

Here are the directories that are excluded by the robots.txt file currently (Oct 24, 2003) at whitehouse.gov. According to the Robots Exclusion Specification, exclusions given to a directory extend to all the directories that are under it (so-called "child" directories), so some of these directories are not explicitly listed in the robots.txt file.


http://www.whitehouse.gov/infocus/iraq
http://www.whitehouse.gov/infocus/iraq/text
http://www.whitehouse.gov/infocus/iraq/100days
http://www.whitehouse.gov/infocus/iraq/photoessay
http://www.whitehouse.gov/infocus/iraq/news
http://www.whitehouse.gov/infocus/iraq/disarmament
http://www.whitehouse.gov/infocus/iraq/decade
http://www.whitehouse.gov/infocus/iraq/swf
http://www.whitehouse.gov/infocus/iraq/100days/text
http://www.whitehouse.gov/infocus/iraq/photoessay/essay6
http://www.whitehouse.gov/infocus/iraq/photoessay/essay1
http://www.whitehouse.gov/infocus/iraq/photoessay/essay5
http://www.whitehouse.gov/infocus/iraq/photoessay/essay4
http://www.whitehouse.gov/infocus/iraq/photoessay/essay3
http://www.whitehouse.gov/infocus/iraq/photoessay/essay2
http://www.whitehouse.gov/infocus/iraq/disarmament/text
http://www.whitehouse.gov/infocus/iraq/decade/text
http://www.whitehouse.gov/infocus/iraq/decade/text/text
http://www.whitehouse.gov/news/timeline/iraq
http://www.whitehouse.gov/news/releases/2003/09/iraq
http://www.whitehouse.gov/news/releases/2003/09/iraq/text
http://www.whitehouse.gov/news/releases/2003/05/iraq
http://www.whitehouse.gov/news/releases/2003/05/iraq/text
http://www.whitehouse.gov/news/releases/2003/05/iraq/
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news/releases
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news/releases/2003
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news/releases/2003/05
http://www.whitehouse.gov/news/releases/2003/03/iraq
http://www.whitehouse.gov/news/releases/2003/03/iraq/text
http://www.whitehouse.gov/news/releases/2003/01/iraq
http://www.whitehouse.gov/news/releases/2003/01/iraq/text
http://www.whitehouse.gov/news/releases/2003/10/iraq
http://www.whitehouse.gov/news/releases/2003/10/iraq/text
http://www.whitehouse.gov/news/releases/2003/08/iraq
http://www.whitehouse.gov/news/releases/2003/08/iraq/text
http://www.whitehouse.gov/news/releases/2003/07/iraq
http://www.whitehouse.gov/news/releases/2003/07/iraq/text
http://www.whitehouse.gov/news/releases/2003/06/iraq
http://www.whitehouse.gov/news/releases/2003/06/iraq/text
http://www.whitehouse.gov/news/releases/2003/04/iraq
http://www.whitehouse.gov/news/releases/2003/04/iraq/text
http://www.whitehouse.gov/news/releases/2003/02/iraq
http://www.whitehouse.gov/news/releases/2003/02/iraq/text
http://www.whitehouse.gov/news/releases/2003/02/iraq/powell-slides
http://www.whitehouse.gov/news/releases/2002/12/iraq
http://www.whitehouse.gov/news/releases/2002/12/iraq/tex
http://www.whitehouse.gov/news/releases/2002/11/iraq
http://www.whitehouse.gov/news/releases/2002/11/iraq/text
http://www.whitehouse.gov/news/releases/2002/10/iraq
http://www.whitehouse.gov/news/releases/2002/10/iraq/text
http://www.whitehouse.gov/news/releases/2002/09/iraq
http://www.whitehouse.gov/news/releases/2002/09/iraq/text
http://www.whitehouse.gov/news/releases/2002/08/iraq
http://www.whitehouse.gov/news/releases/2002/03/iraq
http://www.whitehouse.gov/news/releases/2002/01/iraq
http://www.whitehouse.gov/news/releases/2001/08/iraq
http://www.whitehouse.gov/news/releases/2001/07/iraq
http://www.whitehouse.gov/news/releases/2001/04/iraq
http://www.whitehouse.gov/news/releases/2001/02/iraq
http://www.whitehouse.gov/news/releases/2003/09/images/iraq
http://www.whitehouse.gov/news/releases/2003/05/images/iraq
http://www.whitehouse.gov/news/releases/2003/03/images/iraq
http://www.whitehouse.gov/news/releases/2003/01/images/iraq
http://www.whitehouse.gov/news/releases/2003/10/images/iraq
http://www.whitehouse.gov/news/releases/2003/08/images/iraq
http://www.whitehouse.gov/news/releases/2003/07/images/iraq
http://www.whitehouse.gov/news/releases/2003/04/images/iraq
http://www.whitehouse.gov/news/releases/2003/02/images/iraq
http://www.whitehouse.gov/news/releases/2002/12/images/iraq
http://www.whitehouse.gov/news/releases/2002/11/images/iraq
http://www.whitehouse.gov/news/releases/2002/10/images/iraq
http://www.whitehouse.gov/news/releases/2002/09/images/iraq
http://www.whitehouse.gov/news/releases/2002/08/images/iraq
http://www.whitehouse.gov/news/releases/2002/03/images/iraq

Note: some few of these directories actually exist but are empty. Some of them are populated with files but merely have no index file (so you get a 404 when opening a URL above). If I have time and feel it's worthwhile I will try to get a list of all the actual files that have been excluded.

UPDATE:

 

I haven't gotten a list of all the files that have been excluded, but I did look for some issues.

Some of the directories that 404 truly are empty of files. FOr instance:
http://www.whitehouse.gov/news/timeline/iraq

doesn't have files.

But at least some of the files that 404 above Do have files in the directory, just not an index file. For instance:

http://www.whitehouse.gov/infocus/iraq/100days

does not have an index page, so just entering that URL will give a 404.

However, the directory has the following files in it:

http://www.whitehouse.gov/infocus/iraq/100days/100days.pdf
http://www.whitehouse.gov/infocus/iraq/100days/introduction.html
http://www.whitehouse.gov/infocus/iraq/100days/part1.html
http://www.whitehouse.gov/infocus/iraq/100days/part2.html
http://www.whitehouse.gov/infocus/iraq/100days/part3.html
http://www.whitehouse.gov/infocus/iraq/100days/part4.html
http://www.whitehouse.gov/infocus/iraq/100days/part5.html
http://www.whitehouse.gov/infocus/iraq/100days/part6.html
http://www.whitehouse.gov/infocus/iraq/100days/part7.html
http://www.whitehouse.gov/infocus/iraq/100days/part8.html
http://www.whitehouse.gov/infocus/iraq/100days/part9.html
http://www.whitehouse.gov/infocus/iraq/100days/part10.html

All those files are excluded by the directory disallow entry in robots.txt

I'll try to work some more on this if I get time.

 

Back to Home: Whitehouse.gov Robots.txt