Excluded Directories

Update below

Here are the directories that are excluded by the robots.txt file currently (Oct 24, 2003) at whitehouse.gov. According to the Robots Exclusion Specification, exclusions given to a directory extend to all the directories that are under it (so-called "child" directories), so some of these directories are not explicitly listed in the robots.txt file.

http://www.whitehouse.gov/infocus/iraq
http://www.whitehouse.gov/infocus/iraq/text
http://www.whitehouse.gov/infocus/iraq/100days
http://www.whitehouse.gov/infocus/iraq/photoessay
http://www.whitehouse.gov/infocus/iraq/news
http://www.whitehouse.gov/infocus/iraq/disarmament
http://www.whitehouse.gov/infocus/iraq/decade
http://www.whitehouse.gov/infocus/iraq/swf
http://www.whitehouse.gov/infocus/iraq/100days/text
http://www.whitehouse.gov/infocus/iraq/photoessay/essay6
http://www.whitehouse.gov/infocus/iraq/photoessay/essay1
http://www.whitehouse.gov/infocus/iraq/photoessay/essay5
http://www.whitehouse.gov/infocus/iraq/photoessay/essay4
http://www.whitehouse.gov/infocus/iraq/photoessay/essay3
http://www.whitehouse.gov/infocus/iraq/photoessay/essay2
http://www.whitehouse.gov/infocus/iraq/disarmament/text
http://www.whitehouse.gov/infocus/iraq/decade/text
http://www.whitehouse.gov/infocus/iraq/decade/text/text
http://www.whitehouse.gov/news/timeline/iraq
http://www.whitehouse.gov/news/releases/2003/09/iraq
http://www.whitehouse.gov/news/releases/2003/09/iraq/text
http://www.whitehouse.gov/news/releases/2003/05/iraq
http://www.whitehouse.gov/news/releases/2003/05/iraq/text
http://www.whitehouse.gov/news/releases/2003/05/iraq/
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news/releases
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news/releases/2003
http://www.whitehouse.gov/news/releases/2003/05/iraq/_/news/releases/2003/05
http://www.whitehouse.gov/news/releases/2003/03/iraq
http://www.whitehouse.gov/news/releases/2003/03/iraq/text
http://www.whitehouse.gov/news/releases/2003/01/iraq
http://www.whitehouse.gov/news/releases/2003/01/iraq/text
http://www.whitehouse.gov/news/releases/2003/10/iraq
http://www.whitehouse.gov/news/releases/2003/10/iraq/text
http://www.whitehouse.gov/news/releases/2003/08/iraq
http://www.whitehouse.gov/news/releases/2003/08/iraq/text
http://www.whitehouse.gov/news/releases/2003/07/iraq
http://www.whitehouse.gov/news/releases/2003/07/iraq/text
http://www.whitehouse.gov/news/releases/2003/06/iraq
http://www.whitehouse.gov/news/releases/2003/06/iraq/text
http://www.whitehouse.gov/news/releases/2003/04/iraq
http://www.whitehouse.gov/news/releases/2003/04/iraq/text
http://www.whitehouse.gov/news/releases/2003/02/iraq
http://www.whitehouse.gov/news/releases/2003/02/iraq/text
http://www.whitehouse.gov/news/releases/2003/02/iraq/powell-slides
http://www.whitehouse.gov/news/releases/2002/12/iraq
http://www.whitehouse.gov/news/releases/2002/12/iraq/tex
http://www.whitehouse.gov/news/releases/2002/11/iraq
http://www.whitehouse.gov/news/releases/2002/11/iraq/text
http://www.whitehouse.gov/news/releases/2002/10/iraq
http://www.whitehouse.gov/news/releases/2002/10/iraq/text
http://www.whitehouse.gov/news/releases/2002/09/iraq
http://www.whitehouse.gov/news/releases/2002/09/iraq/text
http://www.whitehouse.gov/news/releases/2002/08/iraq
http://www.whitehouse.gov/news/releases/2002/03/iraq
http://www.whitehouse.gov/news/releases/2002/01/iraq
http://www.whitehouse.gov/news/releases/2001/08/iraq
http://www.whitehouse.gov/news/releases/2001/07/iraq
http://www.whitehouse.gov/news/releases/2001/04/iraq
http://www.whitehouse.gov/news/releases/2001/02/iraq
http://www.whitehouse.gov/news/releases/2003/09/images/iraq
http://www.whitehouse.gov/news/releases/2003/05/images/iraq
http://www.whitehouse.gov/news/releases/2003/03/images/iraq
http://www.whitehouse.gov/news/releases/2003/01/images/iraq
http://www.whitehouse.gov/news/releases/2003/10/images/iraq
http://www.whitehouse.gov/news/releases/2003/08/images/iraq
http://www.whitehouse.gov/news/releases/2003/07/images/iraq
http://www.whitehouse.gov/news/releases/2003/04/images/iraq
http://www.whitehouse.gov/news/releases/2003/02/images/iraq
http://www.whitehouse.gov/news/releases/2002/12/images/iraq
http://www.whitehouse.gov/news/releases/2002/11/images/iraq
http://www.whitehouse.gov/news/releases/2002/10/images/iraq
http://www.whitehouse.gov/news/releases/2002/09/images/iraq
http://www.whitehouse.gov/news/releases/2002/08/images/iraq
http://www.whitehouse.gov/news/releases/2002/03/images/iraq

Note: some few of these directories actually exist but are empty. Some of them are populated with files but merely have no index file (so you get a 404 when opening a URL above). If I have time and feel it's worthwhile I will try to get a list of all the actual files that have been excluded.

UPDATE:

I haven't gotten a list of all the files that have been excluded, but I did look for some issues.

Some of the directories that 404 truly are empty of files. FOr instance:
http://www.whitehouse.gov/news/timeline/iraq

doesn't have files.

But at least some of the files that 404 above Do have files in the directory, just not an index file. For instance:

http://www.whitehouse.gov/infocus/iraq/100days

does not have an index page, so just entering that URL will give a 404.

However, the directory has the following files in it:

http://www.whitehouse.gov/infocus/iraq/100days/100days.pdf
http://www.whitehouse.gov/infocus/iraq/100days/introduction.html
http://www.whitehouse.gov/infocus/iraq/100days/part1.html
http://www.whitehouse.gov/infocus/iraq/100days/part2.html
http://www.whitehouse.gov/infocus/iraq/100days/part3.html
http://www.whitehouse.gov/infocus/iraq/100days/part4.html
http://www.whitehouse.gov/infocus/iraq/100days/part5.html
http://www.whitehouse.gov/infocus/iraq/100days/part6.html
http://www.whitehouse.gov/infocus/iraq/100days/part7.html
http://www.whitehouse.gov/infocus/iraq/100days/part8.html
http://www.whitehouse.gov/infocus/iraq/100days/part9.html
http://www.whitehouse.gov/infocus/iraq/100days/part10.html

All those files are excluded by the directory disallow entry in robots.txt

I'll try to work some more on this if I get time.