Search notes:

Web crawling

TODO

https://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/ tries to create a »combined, focused web crawling system that collects relevant documents from the web that is particularly suited for harvesting publications in the educational domain«

Firecrawl

FIRECRAWL_API_KEY=‥

url=https://www.scrapethissite.com/pages/simple/

curl https://api.firecrawl.dev/v2/scrape       \
-H "Authorization: Bearer $FIRECRAWL_API_KEY"  \
-H 'Content-Type: application/json'            \
-d @- <<DATA
  "url": "$url"
}
DATA

See also

Azure Bot Service

web

Links

Learn to scrape a website on scrapethissite.com.

Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 8 attempt to write a readonly database in /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php:78 Stack trace: #0 /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php(78): PDOStatement->execute(Array) #1 /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php(30): insert_webrequest_('/notes/developm...', 1788449510, '216.73.217.21', 'Mozilla/5.0 App...', NULL) #2 /home/httpd/vhosts/renenyffenegger.ch/httpsdocs/notes/development/web/crawling/index(67): insert_webrequest() #3 {main} thrown in /home/httpd/vhosts/renenyffenegger.ch/php/web-request-database.php on line 78