of robots

October 27, 2003

Apparently the whitehouse.gov robots.txt file is blocking searches of iraq . Well, I’ve been digging and what was the reason to change the (basically empty) Whitehouse.gov robots.txt file that was being used until August 29, 2001 to the one from September 1, 2001 that blocked everything?

What brought about that change, 11 days before the attacks on NY, DC, and Penn? This is a giant head scratcher. Why would they go from a file with 0 entries and change to a file with dozens of entries in the course of 48 hours? Is this related to 9/11/01?