Monday man page: curl

Today's man page covers one of my favorite utilities: curl. No, it's not a haircare product -- it's one of the most flexible download tools in the kit bag, with the ability to handle almost any protocol that can be addressed via a URL (hence the name, short for "client for URLs"). If there's a server out there that's reachable via HTTP, HTTPS, FTP, SFTP, SCP, and lots of other alphabet soup, curl can talk to it.

curl http://www.tuaw.com/2007/03/05/monday-man-page-curl/ -- display the source of this very article in Terminal
curl ftp://ftp.panic.com -- list the contents of a remote FTP site, in this case one with a pretty good FTP client
curl -o ~/Desktop/curl-man.html http://curl.haxx.se/docs/manpage.html -- copy the curl manpage to your desktop; if you use capital -O, the local file mirrors the remote filename

curl has an excellent usage manual at its site, detailing examples of use and advanced techniques. While there are zillions of ways to use curl in site testing, analysis and uploading, my favorite way of using it is as a quick file downloader. Read on for the details.

curl does an awesome job of quickly and cleanly downloading files, making it perfect for DIY software distribution schemes. In my case, I keep a web-shared folder of current installers and packages zipped up for delivery, and then when I'm managing a remote user's machine, I can trigger the downloads via ssh and curl -- all behind the user's back:

curl http://myserver.example.com/Installer.mpkg.zip -o /tmp/install.zip unzip -o install.zip sudo installer -pkg /tmp/Installer.mpkg -target /

If you wanted to do something really cool, though, you could take advantage of curl's ability to compare file timestamps, and trigger a download only when the source file is updated -- great for a login script:

curl -z updater.zip http://server.example.com/updater.zip

thereby making sure that the client always has the latest file downloaded. So handy!

Hannes adds a comment below that he always uses the -C option to allow curl to resume interrupted file downloads, a trick that Safari hasn't mastered yet:

curl -C -O http://example.com/bigfile.zip

One last fun trick: using curl to do dictionary lookups...

curl dict://dict.org/d:macintosh

20 aspen.miranda.org dictd 1.9.15/rf on Linux 2.6.18-3-k7  <7204009.9729.1173071849@aspen.miranda.org>

250 ok

150 2 definitions retrieved

151 "Macintosh" gcide "The Collaborative International Dictionary of English v.0.48"

Mac \Mac\ prop. n.

Shortened form of {Macintosh}, a brand name for a personal

computer; as, the latest Mac has great new features.

[PJC]

.

151 "Macintosh" gcide "The Collaborative International Dictionary of English v.0.48"

Macintosh \Mac"in*tosh\, n.

1. Same as {Mackintosh}.

[1913 Webster]



2. [Trademark.] (Computers) A brand of personal computer

featuring an integrated system in which the hardware and

system-operating software were designed by or under the

control of a single company, the Apple Computer

Corporation; among personal computers, distinguished from

the {IBM-compatible} or {Intel-based} series of computers.

[PJC]

View Tags

Source: http://curl.haxx.se/docs/manpage.html

Categories

Features UNIX / BSD TUAW Tips

Today's man page covers one of my favorite utilities: curl. No, it's not a haircare product -- it's one of the most flexible download...

Add a Comment

Sign in »

8 Comments

Filter by:

Steve M

Just want to give a shout out to developers. On a recent project I had a bunch of old SOAP code that was no longer working with a new version of Java we were upgrading to. I converted all of the C code to using the curl libraries (libcurl) in about two days. It is a really easy API to learn and use if you need to write applications that transfer information with a web application, or for simple file transfers.

March 07 2007 at 11:55 AM Report abuse Permalink rate up rate down Reply

Hannes

Michael, I think you forgot the dash after the "-C" flag. It must be "-C - " in order to resume from the correct point.

March 06 2007 at 3:03 PM Report abuse Permalink rate up rate down Reply

Jason

Another simple use for curl is to use it as your download manager with Flashgot in Firefox. Running 10 simultaneous downloads in Firefox will slow your machine to a crawl, but handing them off to curl works great.

March 06 2007 at 12:14 AM Report abuse Permalink rate up rate down Reply

brian

Curl is the greatest thing in the world. To download a series of files:
curl -O example.org/[1-9].jpg
it's smart--it can handle leading zeroes just fine:
curl -O example.org/[01-20].jpg
You can even do a sequence of images in a series of folders, but you've got to use '-o' to name the local files--otherwise it'll download the first 10 as 1-10.jpg, then overwrite those with the next batch. So use '-o' and '#' to give unique names to the resulting files:
curl example.org/[1-5]/[1-9].jpg -o "#1-#2.jpg"
'#1' means 'the first number' and '#2' means 'the second number'
This will result in 1-1,jpg, 1-2.jpg ... 1-9.jpg, 2-1.jpg, 2-2.jpg, etc.

And since 'curl' by default just produces plain text output (if fed a plain text file) it's also handy for system administration. Want to run a shell script on a bunch of machines and you don't have ARD? You don't have to connect to a file server from each Mac to download the script, or walk around with it on a USB drive. Just post your script to a server (as 'script.sh' in this example) and then launch Terminal.app on each client and enter just one command:
curl your.ip.address.here/script.sh | sh

1,001 uses.

March 05 2007 at 3:15 PM Report abuse Permalink rate up rate down Reply

Michael Rose

Thanks guys -- both tips included in the post now.

March 05 2007 at 2:15 PM Report abuse Permalink rate up rate down Reply

bob

I use curl with geektool to give me almost live headlines from some favarorite blogs right on my desktop.

March 05 2007 at 2:01 PM Report abuse Permalink rate up rate down Reply

mroach

Tip: If you use "curl -O" (capital "O") instead of lowercase "o" you don't have to specify a filename; it will use the remote file's name.

Also, curl is a good way to FTP single files. For that, "curl -T filename.ext ftp://username@ftp.somedomain.com"

March 05 2007 at 1:50 PM Report abuse Permalink rate up rate down Reply

Hannes

I always use curl when it comes to downloading big files, as Safari's got no (working) possibility to resume a download which for some reason (lost airport signal or safari crash for example) has been aborted. Running curl with "-C - " makes the download resume where it stopped last time. I found the combination "curl -OC - " so good that I bound the alias "C" to it. So when I want to download a big file, I simply type "C http://example.com/bigfile.zip" into the prompt.

March 05 2007 at 1:26 PM Report abuse Permalink rate up rate down Reply