Filed under: Features, UNIX / BSD, TUAW Tips
Monday man page: curl
Today's man page covers one of my favorite utilities: curl. No, it's not a haircare product -- it's one of the most flexible download tools in the kit bag, with the ability to handle almost any protocol that can be addressed via a URL (hence the name, short for "client for URLs"). If there's a server out there that's reachable via HTTP, HTTPS, FTP, SFTP, SCP, and lots of other alphabet soup, curl can talk to it.
curl http://www.tuaw.com/2007/03/05/monday-man-page-curl/ -- display the source of this very article in Terminalcurl ftp://ftp.panic.com -- list the contents of a remote FTP site, in this case one with a pretty good FTP clientcurl -o ~/Desktop/curl-man.html http://curl.haxx.se/docs/manpage.html -- copy the curl manpage to your desktop; if you use capital -O, the local file mirrors the remote filenamecurl has an excellent usage manual at its site, detailing examples of use and advanced techniques. While there are zillions of ways to use curl in site testing, analysis and uploading, my favorite way of using it is as a quick file downloader. Read on for the details.
curl does an awesome job of quickly and cleanly downloading files, making it perfect for DIY software distribution schemes. In my case, I keep a web-shared folder of current installers and packages zipped up for delivery, and then trigger the downloads via ssh and curl, all behind the user's back:
curl http://myserver.example.com/Installer.mpkg.zip -o /tmp/install.zip
unzip -o install.zip
sudo installer -pkg /tmp/Installer.mpkg -target /If you wanted to do something really cool, though, you could take advantage of curl's ability to compare file timestamps, and trigger a download only when the source file is updated -- great for a login script:
curl -z updater.zip http://server.example.com/updater.zipthereby making sure that the client always has the latest file downloaded. So handy!
Hannes adds a comment below that he always uses the -C option to allow curl to resume interrupted file downloads, a trick that Safari hasn't mastered yet:
curl -C -O http://example.com/bigfile.zipOne last fun trick: using curl to do dictionary lookups...
curl dict://dict.org/d:macintosh
20 aspen.miranda.org dictd 1.9.15/rf on Linux 2.6.18-3-k7 <auth.mime> <7204009.9729.1173071849@aspen.miranda.org>
250 ok
150 2 definitions retrieved
151 "Macintosh" gcide "The Collaborative International Dictionary of English v.0.48"
Mac \Mac\ prop. n.
Shortened form of {Macintosh}, a brand name for a personal
computer; as, the latest Mac has great new features.
[PJC]
.
151 "Macintosh" gcide "The Collaborative International Dictionary of English v.0.48"
Macintosh \Mac"in*tosh\, n.
1. Same as {Mackintosh}.
[1913 Webster]
2. [Trademark.] (Computers) A brand of personal computer
featuring an integrated system in which the hardware and
system-operating software were designed by or under the
control of a single company, the Apple Computer
Corporation; among personal computers, distinguished from
the {IBM-compatible} or {Intel-based} series of computers.
[PJC]
![TUAW [Cafepress]](http://www.blogsmithmedia.com/www.tuaw.com/media/tuaw-cafepress-promo.png)
Reader Comments (Page 1 of 1)
Hannes Petri said 2:15PM on 3-05-2007
I always use curl when it comes to downloading big files, as Safari's got no (working) possibility to resume a download which for some reason (lost airport signal or safari crash for example) has been aborted. Running curl with "-C - " makes the download resume where it stopped last time. I found the combination "curl -OC - " so good that I bound the alias "C" to it. So when I want to download a big file, I simply type "C http://example.com/bigfile.zip" into the prompt.
Reply
mroach said 1:55PM on 3-05-2007
Tip: If you use "curl -O" (capital "O") instead of lowercase "o" you don't have to specify a filename; it will use the remote file's name.
Also, curl is a good way to FTP single files. For that, "curl -T filename.ext ftp://username@ftp.somedomain.com"
Reply
bob said 2:52PM on 3-05-2007
I use curl with geektool to give me almost live headlines from some favarorite blogs right on my desktop.
Reply
Michael Rose said 2:15PM on 3-05-2007
Thanks guys -- both tips included in the post now.
Reply
brian said 3:15PM on 3-05-2007
Curl is the greatest thing in the world. To download a series of files:
curl -O example.org/[1-9].jpg
it's smart--it can handle leading zeroes just fine:
curl -O example.org/[01-20].jpg
You can even do a sequence of images in a series of folders, but you've got to use '-o' to name the local files--otherwise it'll download the first 10 as 1-10.jpg, then overwrite those with the next batch. So use '-o' and '#' to give unique names to the resulting files:
curl example.org/[1-5]/[1-9].jpg -o "#1-#2.jpg"
'#1' means 'the first number' and '#2' means 'the second number'
This will result in 1-1,jpg, 1-2.jpg ... 1-9.jpg, 2-1.jpg, 2-2.jpg, etc.
And since 'curl' by default just produces plain text output (if fed a plain text file) it's also handy for system administration. Want to run a shell script on a bunch of machines and you don't have ARD? You don't have to connect to a file server from each Mac to download the script, or walk around with it on a USB drive. Just post your script to a server (as 'script.sh' in this example) and then launch Terminal.app on each client and enter just one command:
curl your.ip.address.here/script.sh | sh
1,001 uses.
Reply
Jason said 12:15AM on 3-06-2007
Another simple use for curl is to use it as your download manager with Flashgot in Firefox. Running 10 simultaneous downloads in Firefox will slow your machine to a crawl, but handing them off to curl works great.
Reply
Hannes Petri said 3:06PM on 3-06-2007
Michael, I think you forgot the dash after the "-C" flag. It must be "-C - " in order to resume from the correct point.
Reply
Steve M said 12:02PM on 3-07-2007
Just want to give a shout out to developers. On a recent project I had a bunch of old SOAP code that was no longer working with a new version of Java we were upgrading to. I converted all of the C code to using the curl libraries (libcurl) in about two days. It is a really easy API to learn and use if you need to write applications that transfer information with a web application, or for simple file transfers.
Reply