# Url from Sitemap Sitemap: http://www.newyorker.de/index.php?id=19&type=841132 User-agent: Googlebot Disallow: # Do not allow ia_archiver to access any part of the server # ia_archiver is the User Agent from archive.org User-agent: ia_archiver Disallow: /