Search notes:

Web crawling

TODO

https://www.kdsl.tu-darmstadt.de/de/kdsl/research-program/crawling-and-semantic-structuring/ tries to create a »combined, focused web crawling system that collects relevant documents from the web that is particularly suited for harvesting publications in the educational domain«
Heritrix

Firecrawl

FIRECRAWL_API_KEY=‥

url=https://www.scrapethissite.com/pages/simple/

curl https://api.firecrawl.dev/v2/scrape       \
-H "Authorization: Bearer $FIRECRAWL_API_KEY"  \
-H 'Content-Type: application/json'            \
-d @- <<DATA
  "url": "$url"
}
DATA

See also

robots.txt
Azure Bot Service
web

Links

Learn to scrape a website on scrapethissite.com.

Index