Search notes:

robots.txt

Special characters

# starts a comment that spans to the end of the line
* zero or more characters (including /)
$ specifies a hard end of the match pattern

Example

User-agent: facebookexternalhit
User-agent: Twitterbot
Allow: /foo
Allow: /bar
Allow: /*/shop/*/offers$
Disallow: /baz

Sitemap: https://server.xyz/sitemap.xml

crawl-delay

crawl-delay is an unofficial directive for crawlers.
This directive is honored, for example, by Common Crawl, but ignored by Google or Yandex.

Misc

e-Helvetica is a project by the Swiss National Library with the aim to collect and archive digital publications related to Switzerland among which are websites.
In this project, robots.txt and robots meta tags are explicitely ignored.

See also

RFC 9309
«Known» HTTP request paths
The Python standard library class urllib.robotparser.RobotFileParser
Web crawling
/ads.txt, /lmms.txt

humans.txt

Some websites also have a humans.txt file (such as for example init7.net).

Links

The Web Robots Pages: about robots.txt
The robotstxt github repository contains Google's robots.txt parser and matcher as a C++ library

Index

Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 8 attempt to write a readonly database in /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php:78 Stack trace: #0 /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php(78): PDOStatement->execute(Array) #1 /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php(30): insert_webrequest_('/notes/developm...', 1788432490, '216.73.217.21', 'Mozilla/5.0 App...', NULL) #2 /home/httpd/vhosts/renenyffenegger.ch/httpsdocs/notes/development/web/HTTP/robots_txt(97): insert_webrequest() #3 {main} thrown in /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php on line 78