Retrieve Headers with cURL
We all know the cURL is incredibly useful. We can retrieve remote content with curl, post to a remote URL, and perform hundreds of other tasks. One simple task that can be completed is simply retrieving basic response headers. To test the robot indexing prevention header I added to the Mozilla Developer Network, I used one simple cURL command to grab all headers from an address.
The Shell
The cURL command is short and sweet:
curl -I davidwalsh.name
Said command provides a list that looks similar to:
HTTP/1.1 200 OK Date: Fri, 14 Sep 2012 21:51:17 GMT Server: Apache/2.2.3 (CentOS) Last-Modified: Fri, 14 Sep 2012 21:51:00 GMT Accept-Ranges: bytes Content-Length: 10910 Cache-Control: max-age=1, private, must-revalidate Expires: Fri, 14 Sep 2012 22:51:00 GMT Vary: Accept-Encoding,Cookie X-Powered-By: W3 Total Cache/0.9.2.4 X-Pingback: https://davidwalsh.name/xmlrpc.php Pragma: public Connection: close Content-Type: text/html; charset=UTF-8
This command is helpful when ensuring a given header has been correctly set within your programming, as well as seeing where a given short URL may redirect to:
$ curl -I bit.ly/Q8f9o HTTP/1.1 301 Moved Server: nginx Date: Fri, 14 Sep 2012 21:53:14 GMT Content-Type: text/html; charset=utf-8 Connection: keep-alive Set-Cookie: _bit=5053a74a-0011d-0688d-311cf10a;domain=.bit.ly;expires=Wed Mar 13 21:53:14 2013;path=/; HttpOnly Cache-control: private; max-age=90 Location: https://davidwalsh.name/ MIME-Version: 1.0 Content-Length: 115
It's also useful to see the server name, expires information and more. I also appreciate that it's a clean list and no other information is pushed into the response. If you get some time, cURL out to different popular domains and see what headers they send -- you could be surprised!
Thanks for this great tips for cURL.
I did not know before that i can get header with cURL
I’m using Charles (on desktop), it can show both the request and response headers. Handy even when not trying to cheat for water on farmville 2 :P
I highly recommend it to other devs as it has many cool features (such as throttling, charts and request/response manipulation).
That’s a useful tip – thanks! I’ve always done
lynx -head dumpbut that’s a lot quicker and easier.One thing to be aware of is that it makes a HEAD request. I would think it normally will return the same header values as a GET or POST but still something worth thinking about depending what you are using it for.
It’s important to note that some servers are set to respond differently to HEADER requests than to GET requests. For example, a HEADER request returns a 200 OK while a GET request returns a 301 Moved Permanently… It can be quite confusing and unreliable depending on what you’re testing.
Connection header is set to closed, however when checking via Firebug or other tools such as Pingdom shows Keep-alive. Do you know why is that so?
Thank you,
Shyam