Search notes:

cURL

Download files

By default, curl prints the content of a resource to stdout: 
curl https://www.google.com/robots.txt
In order to save the file to a file with the same name as the downloaded file, -O must be used.
This command creates or overwrites the file robots.txt in the current directory: 
curl -O https://www.google.com/robots.txt
The previous command prints some statistics to stderr.
-s suppresses that: 
curl -Os https://www.google.com/robots.txt
-o can be used to specify the name of the written file
The -L option here is suggested in order to follow redirects: 
$ curl -sL http://some.url.xyz/path/to/file -o downloaded-file-name

Download and untar in one go

It's possible to pipe a downloaded tar file directly to tar so that the tar file can be extracted in one go. 
curl -L https://.tar.xz | tar xJ -C /tmp/mesa
Note: the J option of tar is to extract a xz archive.

No clobbering of downloaded files / resume interrupted download

-C - specifies to resume a download where it was interrupted.
Additionally, if the file is already fully downloaded, it does not clobber the already downloaded file: 
curl -C - https://dumps.wikimedia.org/other/pageviews/2024/2024-04/pageviews-20240401-000000.gz -o 20240401-20240401.gz

Interesting command line options

-i

-i (or --include) includes protocol response headers in the output.

-u

-u (or --user) send username and password (for basic authentication?) 
curl -u user:password

-H

-H (or --header) passes specific HTTP headers to the web server. 
curl -H "Authorization: token …" https://foobar.baz/xyz/
curl -H @file

-T

When used with HTTP, -T (or --upload-file) initiates a PUT request to upload a file to the web server.

-d / --data

d specifies the data for the body of a POST request (unless -G is also specified)

Data from a file

-d can be followed by @path/to/file which specifies the name of a file that contains the data for the body.

Using a here document to create the data

With @-, curl is instructed to read the data from stdin. The following here document then provides the data that will be read from stdin: 
curl -s $url   \
  -H …         \
  -d @- <<DATA

   Insert the data of the body here.
   It's possible to use shell variables.

DATA
In order to pipe the result of the server down a pipeline, the pipe symbol must appear after the here document delimiter (here: DATA): 
curl -s $url   \
  -H …         \
  -d @- <<DATA     | … pipeline …

   Insert the data of the body here.
   It's possible to use shell variables.

DATA

--data-urlencode

--data-urlencode can be combined with -G to create a GET request rather than a POST request.
The following example displays the resulting request URL by using -v and then grepping for GET: 
$ curl -G -v                                             \
  https://nominatim.openstreetmap.org/search             \
  --data-urlencode "q=White House, Washington"           \
  --data-urlencode  format=jsonv2                        \
   2>&1 > /dev/null                              |       \
  grep GET
When executed, the command prints 
GET /search?q=White%20House%2C%20Washington&format=jsonv2 HTTP/2

-v

Show verbose output of how the HTTP connection is established, for example TLS and certificate related information: 
$ curl -X HEAD -v   https://letsencrypt.org/
…
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=lencr.org
*  start date: Nov 28 21:20:24 2023 GMT
*  expire date: Feb 26 21:20:23 2024 GMT
*  subjectAltName: host "letsencrypt.org" matched cert's "letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
…

-F / --form

-F allows to simulate the following piece of HTML 
<form enctype="multipart/form-data" action="upload-file" method="post">
   Specify file to uplad: <input type="file" name="filename">
   <input type="submit" value="Upload">
</form>
with 
$ curl -X POST http://xyz/upload-file -H "Content-type: multipart/form-data"  -F "filename=@/tmp/file-to-upload.txt;type=text/plain"

--ntlm

Using --ntlm helped me fetch a resource that needed NTLM authentication. I had to enter the password on the commandline 
C:\> curl.exe --ntlm -u rene https://xyz/abc
TODO: How (if) is this option related to --negotiate?

All command line options

short long form argument description
--abstract-unix-socket <path> Connect via abstract Unix domain socket
--anyauth Pick any authentication method
-a --append Append to target file when uploading
--basic Use HTTP Basic Authentication
--cacert <CA certificate> CA certificate to verify peer against
--capath <dir> CA directory to verify peer against
-E --cert <certificate[:password]> Client certificate file and password
--cert-status Verify the status of the server certificate
--cert-type <type> Certificate file type (DER/PEM/ENG)
--ciphers <list of ciphers> SSL ciphers to use
--compressed Request compressed response
-K --config <file> Read config from a file
--connect-timeout <seconds> Maximum time allowed for connection
--connect-to <HOST1:PORT1:HOST2:PORT2> Connect to host
-C --continue-at <offset> Resumed transfer offset
-b --cookie <data> Send cookies from string/file
-c --cookie-jar <filename> Write cookies to <filename> after operation
--create-dirs Create necessary local directory hierarchy
--crlf Convert LF to CRLF in upload
--crlfile <file> Get a CRL list in PEM format from the given file
-d --data <data> HTTP POST data
--data-ascii <data> HTTP POST ASCII data
--data-binary <data> HTTP POST binary data
--data-raw <data> HTTP POST data, '@' allowed
--data-urlencode <data> HTTP POST data url encoded
--delegation <LEVEL> GSS-API delegation permission
--digest Use HTTP Digest Authentication
-q --disable Disable .curlrc
--disable-eprt Inhibit using EPRT or LPRT
--disable-epsv Inhibit using EPSV
--dns-interface <interface> Interface to use for DNS requests
--dns-ipv4-addr <address> IPv4 address to use for DNS requests
--dns-ipv6-addr <address> IPv6 address to use for DNS requests
--dns-servers <addresses> DNS server addrs to use
-D --dump-header <filename> Write the received response headers to <filename>. Compare with -i.
--egd-file <file> EGD socket path for random data
--engine <name> Crypto engine to use
--expect100-timeout <seconds> How long to wait for 100-continue
-f --fail Fail silently (no output at all) on HTTP errors
--fail-early Fail on first transfer error, do not continue
--false-start Enable TLS False Start
-F --form <name=content> Specify HTTP multipart POST data
--form-string <name=string> Specify HTTP multipart POST data
--ftp-account <data> Account data string
--ftp-alternative-to-user <command> String to replace USER [name]
--ftp-create-dirs Create the remote dirs if not present
--ftp-method <method> Control CWD usage
--ftp-pasv Use PASV/EPSV instead of PORT
-P --ftp-port <address> Use PORT instead of PASV
--ftp-pret Send PRET before PASV
--ftp-skip-pasv-ip Skip the IP address for PASV
--ftp-ssl-ccc Send CCC after authenticating
--ftp-ssl-ccc-mode <active/passive> Set CCC mode
--ftp-ssl-control Require SSL/TLS for FTP login, clear for transfer
-G --get Put the post data in the URL and use GET
-g --globoff Disable URL sequences and ranges using {} and []
-I --head Show document info only
-H --header <header/@file> Pass custom header(s) to server
-h --help This help text
--hostpubmd5 <md5> Acceptable MD5 hash of the host public key
-0 --http1.0 Use HTTP 1.0
--http1.1 Use HTTP 1.1
--http2 Use HTTP 2
--http2-prior-knowledge Use HTTP 2 without HTTP/1.1 Upgrade
--ignore-content-length Ignore the size of the remote resource
-i --include Include received response headers in the output. Compare with -D.
-k --insecure Allow insecure server connections when using SSL
--interface <name> Use network INTERFACE (or address)
-4 --ipv4 Resolve names to IPv4 addresses
-6 --ipv6 Resolve names to IPv6 addresses
-j --junk-session-cookies Ignore session cookies read from file
--keepalive-time <seconds> Interval time for keepalive probes
--key <key> Private key file name
--key-type <type> Private key file type (DER/PEM/ENG)
--krb <level> Enable Kerberos with security <level>
--libcurl <file> Dump libcurl equivalent code of this command line
--limit-rate <speed> Limit transfer speed to RATE
-l --list-only List only mode
--local-port <num/range> Force use of RANGE for local port numbers
-L --location Follow redirects (See HTTP status 301).
--location-trusted Like --location, and send auth to other hosts
--login-options <options> Server login options
--mail-auth <address> Originator address of the original email
--mail-from <address> Mail from this address
--mail-rcpt <address> Mail from this address
-M --manual Display the full manual
--max-filesize <bytes> Maximum file size to download
--max-redirs <num> Maximum number of redirects allowed
-m --max-time <time> Maximum time allowed for the transfer
--metalink Process given URLs as metalink XML file
--negotiate Use HTTP Negotiate (SPNEGO) authentication
-n --netrc Must read .netrc for user name and password
--netrc-file <filename> Specify FILE for netrc
--netrc-optional Use either .netrc or URL
-: --next Make next URL use its separate set of options
--no-alpn Disable the ALPN TLS extension
-N --no-buffer Disable buffering of the output stream
--no-keepalive Disable TCP keepalive on the connection
--no-npn Disable the NPN TLS extension
--no-sessionid Disable SSL session-ID reusing
--noproxy <no-proxy-list> List of hosts which do not use proxy
--ntlm Use HTTP NTLM authentication
--ntlm-wb Use HTTP NTLM authentication with winbind
--oauth2-bearer <token> OAuth 2 Bearer Token
-o --output <file> Write to file instead of stdout. Compare with -O which chooses the name of the written file according to the downloaded file
--pass <phrase> Pass phrase for the private key
--path-as-is Do not squash .. sequences in URL path
--pinnedpubkey <hashes> FILE/HASHES Public key to verify peer against
--post301 Do not switch to GET after following a 301
--post302 Do not switch to GET after following a 302
--post303 Do not switch to GET after following a 303
--preproxy [protocol://]host[:port] Use this proxy first
-# --progress-bar Display transfer progress as a bar
--proto <protocols> Enable/disable PROTOCOLS. --proto =http,https permits only HTTP and HTTPS. --proto -https disables HTTPS. --proto +https adds HTTPS to the list of permitted protocols.
--proto-default <protocol> Use PROTOCOL for any URL missing a scheme
--proto-redir <protocols> Enable/disable PROTOCOLS on redirect
-x --proxy [protocol://]host[:port] Use this proxy
--proxy-anyauth Pick any proxy authentication method
--proxy-basic Use Basic authentication on the proxy
--proxy-cacert <file> CA certificate to verify peer against for proxy
--proxy-capath <dir> CA directory to verify peer against for proxy
--proxy-cert <cert[:passwd]> Set client certificate for proxy
--proxy-cert-type <type> Client certificate type for HTTS proxy
--proxy-ciphers <list> SSL ciphers to use for proxy
--proxy-crlfile <file> Set a CRL list for proxy
--proxy-digest Use Digest authentication on the proxy
--proxy-header <header/@file> Pass custom header(s) to proxy
--proxy-insecure Do HTTPS proxy connections without verifying the proxy
--proxy-key <key> Private key for HTTPS proxy
--proxy-key-type <type> Private key file type for proxy
--proxy-negotiate Use HTTP Negotiate (SPNEGO) authentication on the proxy
--proxy-ntlm Use NTLM authentication on the proxy
--proxy-pass <phrase> Pass phrase for the private key for HTTPS proxy
--proxy-service-name <name> SPNEGO proxy service name
--proxy-ssl-allow-beast Allow security flaw for interop for HTTPS proxy
--proxy-tlsauthtype <type> TLS authentication type for HTTPS proxy
--proxy-tlspassword <string> TLS password for HTTPS proxy
--proxy-tlsuser <name> TLS username for HTTPS proxy
--proxy-tlsv1 Use TLSv1 for HTTPS proxy
-U --proxy-user <user:password> Proxy user and password
--proxy1.0 <host[:port]> Use HTTP/1.0 proxy on given port
-p --proxytunnel Operate through a HTTP proxy tunnel (using CONNECT)
--pubkey <key> SSH Public key file name
-Q --quote Send command(s) to server before transfer
--random-file <file> File for reading random data from
-r --range <range> Retrieve only the bytes within RANGE
--raw Do HTTP "raw"; no transfer decoding
-e --referer <URL> Referrer URL
-J --remote-header-name Use the header-provided filename
-O --remote-name Write output to a file named as the remote file. Use -o to specify a different file name.
--remote-name-all Use the remote file name for all URLs
-R --remote-time Set the remote file's time on the local output
-X --request <HTTP-request-method> Specify the HTTP request method to use.
--request-target Specify the target for this request
--resolve <host:port:address> Resolve the host+port to this address
--retry <num> Retry request if transient problems occur
--retry-connrefused Retry on connection refused (use with --retry)
--retry-delay <seconds> Wait time between retries
--retry-max-time <seconds> Retry only within this period
--sasl-ir Enable initial response in SASL authentication
--service-name <name> SPNEGO service name
-S --show-error Show error even when -s is used
-s --silent Silent mode: for example to suppress printing of statistics.
--socks4 <host[:port]> SOCKS4 proxy on given host + port
--socks4a <host[:port]> SOCKS4a proxy on given host + port
--socks5 <host[:port]> SOCKS5 proxy on given host + port
--socks5-basic Enable username/password auth for SOCKS5 proxies
--socks5-gssapi Enable GSS-API auth for SOCKS5 proxies
--socks5-gssapi-nec Compatibility with NEC SOCKS5 server
--socks5-gssapi-service <name> SOCKS5 proxy service name for GSS-API
--socks5-hostname <host[:port]> SOCKS5 proxy, pass host name to proxy
-Y --speed-limit <speed> Stop transfers slower than this
-y --speed-time <seconds> Trigger 'speed-limit' abort after this time
--ssl Try SSL/TLS
--ssl-allow-beast Allow security flaw to improve interop
--ssl-no-revoke Disable cert revocation checks (WinSSL)
--ssl-reqd Require SSL/TLS
-2 --sslv2 Use SSLv2
-3 --sslv3 Use SSLv3
--stderr Where to redirect stderr
--suppress-connect-headers Suppress proxy CONNECT response headers
--tcp-fastopen Use TCP Fast Open
--tcp-nodelay Use the TCP_NODELAY option
-t --telnet-option <opt=val> Set telnet option
--tftp-blksize <value> Set TFTP BLKSIZE option
--tftp-no-options Do not send any TFTP options
-z --time-cond <time> Transfer based on a time condition
--tls-max <VERSION> Use TLSv1.0 or greater
--tlsauthtype <type> TLS authentication type
--tlspassword TLS password
--tlsuser <name> TLS user name
-1 --tlsv1 Use TLSv1.0 or greater
--tlsv1.0 Use TLSv1.0
--tlsv1.1 Use TLSv1.1
--tlsv1.2 Use TLSv1.2
--tlsv1.3 Use TLSv1.3
--tr-encoding Request compressed transfer encoding
--trace <file> Write a debug trace to FILE
--trace-ascii <file> Like --trace, but without hex output
--trace-time Add time stamps to trace/verbose output
--unix-socket <path> Connect through this Unix domain socket
-T --upload-file <file> Transfer local FILE to destination
--url <url> URL to work with
-B --use-ascii Use ASCII/text transfer
-u --user <user:password> Server user and password
-A --user-agent <name> Send User-Agent <name> to server
-v --verbose Make the operation more talkative
-V --version Show version number and quit
-w --write-out <format> Use output FORMAT after completion
--xattr Store metadata in extended file attributes

See also

Windows 10 apparently comes with curl.exe installed under %SYSTEMROOT%\System32.

Index