Table of contents
- Download and save to file (Follow redirection)
- Resume interrupted/broken download
- Username and Password
- Use Proxy Server
- Show document information (header)
- Pass through headers
- POST/Upload data
- Read email
- Send email
- Silent cURL
- Download a large file in chunks
- Verify/Login with certificate rather than username and password
- cURL vs Wget
cURL (pronounced ‘curl’) is a computer software project providing a library (libcurl) and command-line tool (curl) for transferring data using various network protocols. The name stands for “Client URL”, It was first released in 1997 . It can be used to transfer (including upload and download) files, emails, web pages etc. Supports more than 20 protocols.
cURL uses HTTP as default protocol to download, e.g. download a web page.
Output of the downloaded web page in terminal
Download web page and save as html file
curl www.bing.com --output bing.htm
3 Download and save file (Follow redirection)
curl -L https://arxiv.org/pdf/1904.00733.pdf --output myfile.pdf
Note: “-L” switch tells cURL to follow redirection, in most cases, we should add “-L” switch
4 Resume interrupted/broken download
While downloading, we can use ctrl + C to stop/interrupt the download.
Later, if we want to continue with the download, we can use following command
curl -C - mywebsite.com/file.7z --output myfile.7z
Note: “-C” switch is used to resume the download, following by ” – ” .
By default, cURL will not timeout. We can add timeout switch to make sure it will not hang indefinitely.
e.g. To timeout after 1 minute, no matter the download finished or not
curl -m 60 bing.com
e.g. To define timeout for connection (So that curl doesn’t spend to much time on waiting for connection or try to connect a host that is offline) (1 minute in this example)
curl --connect-timeout 60 localwebsite.com
6 Username and Password
It can be used for any protocols, e.g. http, https, ftp etc.
#HTTP curl -u username:password http://myownserver.com/myfile.7z #FTP curl -u username:password ftp://myownserver.com/myfile.7z
7 Use Proxy Server
We can use “-x” switch to define proxy server. If protocol is not defined, curl will assume it’s a HTTP proxy server.
curl -x 10.0.0.10:3128 http://awebsite.com
In this case, 10.0.0.10 is the proxy server, 3128 is the proxy server’s port. awebsite.com is the destination you want to proxy through.
We can also use proxy server with other protocols. e.g.
curl -x 10.0.0.10:3128 ftp://awebsite.com/myfile.7z
curl also supports other protocols. e.g. SOCKS, tunnelling etc.
8 Show document information (header)
curl -I awebsite.com
Note: It can be used with “-L”, which means cURL will return headers for each redirection as well.
9 Pass through headers
curl -H 'Connection: keep-alive' -H 'Accept-Charset: uef-8' http://awebsite.com
10 POST/Upload data
If we want to upload a file from /tmp/a.txt to server http://server/upload, which process file input from parameter named my_file.
curl -F '[email protected]/tmp/a.txt' http://server/upload #POST/upload Multiple files curl -F '[email protected]/path/to/file1' -F '[email protected]/path/to/file2' http://server/upload
Upload to FTP server
curl -T file.7z ftp://aftpsite.com/folder/
11 Read Email
cURL supports POP3, IMAP, IMAPS, these protocols can be used to read emails.
To list usable mailbox.
curl -u username:password imap://mail.theserver.com
To read email, we need to add “-X” switch to specify UID of the message which we want to read.
curl -u username:password imap:mail.theserver.com -X 'UID FETCH 1529'
12 Send email
curl smtp://mail.theserver.com --mail-from [email protected] --mail-rcpt [email protected] –upload-file email.txt
The email.txt must be properly formated
From: Web Administrator <[email protected]> To: Alice K <[email protected]> Suject: My title Date: Fri, 17 Jan 2020 00:00:00 Dear Alice, ....... ........ ......... ...... Bob
13 Silent cURL
“-s” switch is used for silent cURL, progress bar and error will be hidden, though your data will still be processed and output. If we want complete silence, we need to output to a file.
#output to current directory curl -s -O http://theserver.com
curl -s http://theserver.com --output webpage.htm
14 Download a large file in chunks
Here we use “–range <bytes>” switch
e.g. Download a big 7z file, we download in 100MB chunks
curl --range 0-99999999 http://aserver.com/file.7z myfile-part1
Second command to download second part
curl --range 100000000-199999999 http://aserver.com/file.7z myfile-part2
Till all parts are downloaded, we can use “cat” command to assemble all parts
cat myfile-part? > my-complete-file.7z
15 Verify/Login with certificate rather than username and password
curl --cert /path/to/certificate.crt:password ftps://theserver.com
man page for cURL’s –cert
--cacert <file> (TLS) Tells curl to use the specified certificate file to verify the peer. The file may contain multiple CA certificates. The certificate(s) must be in PEM format. Normally curl is built to use a default file for this, so this option is typically used to alter that default file. curl recognizes the environment variable named 'CURL_CA_BUNDLE' if it is set, and uses the given path as a path to a CA cert bundle. This option overrides that variable. The windows version of curl will automatically look for a CA certs file named curl-ca-bun-dle.crt, either in the same directory as curl.exe, or in the Current Working Directory, or in any folder along your PATH. If curl is built against the NSS SSL library, the NSS PEM PKCS#11 module (libnsspem.so) needs to be available for this option to work properly. (iOS and macOS only) If curl is built against Secure Transport, then this option is supported for backward compatibility with other SSL engines, but it should not be set. If the option is not set, then curl will use the certificates in the system and user Keychain to verify the peer, which is the preferred method of verifying the peer's certificate chain. (Schannel only) This option is supported for Schannel in Windows 7 or later with libcurl 7.60 or later. This option is supported for backward compatibility with other SSL engines; instead it is recommended to use Windows' store of root certificates (the default for Schannel). If this option is used several times, the last one will be used.
16 cURL vs Wget
wget is the best to use when we just want to download files from websites. It can be used to download whole website.
If we want to use protocols other than HTTP and HTTPS to upload file, we need to use cURL.
 Wikipedia – cURL