curl - Linux

Overview

curl is a command-line tool used for transferring data with URLs. It supports multiple protocols, including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, LDAP, and more. curl is widely used for testing, developing, and interacting with web services, as well as downloading files from the internet. Its versatility makes it an essential tool in web development and system administration.

Syntax

The basic syntax of curl is:

curl [options] [URL...]

Where [options] are the command line options you can pass to modify the behavior of curl, and [URL...] represents one or more URLs where the data should be transferred to or from.

Options/Flags

Here are some commonly used options in curl:

  • -o, --output <file>: Write output to <file> instead of stdout.
  • -I, --head: Fetch the headers only.
  • -H, --header <header>: Pass custom header(s) to server.
  • -d, --data <data>: HTTP POST data.
  • -X, --request <command>: Specify request command to use.
  • --data-urlencode <data>: URL encode POST data.
  • -u, --user <user:password>: Server user and password.
  • -A, --user-agent <name>: Send User-Agent <name> to server.
  • -s, --silent: Silent mode. Don’t show progress meter or error messages.
  • -L, --location: Follow redirects.
  • -C, -continue-at <offset>: Resume a previous file transfer at the given offset.
  • --retry <num>: Retry request if transient problems occur.

Each option alters how curl behaves, from changing the request method to handling authentication or output.

Examples

  1. Simple Download: Download a file and save it with a specific name.

    curl -o example.html http://example.com

  2. Sending Data with POST: Send data via POST method to a server:

    curl -d "param1=value1&param2=value2" -X POST http://example.com/resource

  3. Using Custom Headers: Send a request with custom headers:

    curl -H "Content-Type: application/json" -H "Authorization: Bearer your_token" http://api.example.com

  4. Downloading with Redirects: Downloading while handling location redirects:

    curl -L -o filename.png http://example.com/redirected-image

Common Issues

  • Permissions: Make sure you have the necessary permissions to write files when using -o.
  • Proxy Issues: When behind a proxy, curl may not behave as expected without proper configuration (-x or setting environment variables).
  • Transfer Incompleteness: Network issues can lead to incomplete transfers. Use -C - to resume failed transfers.

Integration

curl can be combined with other tools and shell scripts to automate tasks. Here’s an example of downloading a file only if it doesn’t exist:

[ ! -f "archive.zip" ] && curl -O http://example.com/archive.zip

Or use curl with jq for processing JSON data:

curl -s http://api.example.com/data | jq '.user.name'
  • wget: Similar to curl but primarily used for downloading files.
  • httpie: A user-friendly http client, alternative to curl with more readable outputs.
  • scp: For secure file transfer over SSH protocol, where curl supports SCP through the command line.

For further details, visit the official curl documentation.