Retrieve Headers with cURL
We all know the cURL is incredibly useful. We can retrieve remote content with curl, post to a remote URL, and perform hundreds of other tasks. One simple task that can be completed is simply retrieving basic response headers. To test the robot indexing prevention header I added to the Mozilla Developer Network, I used one simple cURL command to grab all headers from an address.
The cURL command is short and sweet:
curl -I davidwalsh.name
Said command provides a list that looks similar to:
HTTP/1.1 200 OK Date: Fri, 14 Sep 2012 21:51:17 GMT Server: Apache/2.2.3 (CentOS) Last-Modified: Fri, 14 Sep 2012 21:51:00 GMT Accept-Ranges: bytes Content-Length: 10910 Cache-Control: max-age=1, private, must-revalidate Expires: Fri, 14 Sep 2012 22:51:00 GMT Vary: Accept-Encoding,Cookie X-Powered-By: W3 Total Cache/0.9.2.4 X-Pingback: https://davidwalsh.name/xmlrpc.php Pragma: public Connection: close Content-Type: text/html; charset=UTF-8
This command is helpful when ensuring a given header has been correctly set within your programming, as well as seeing where a given short URL may redirect to:
$ curl -I bit.ly/Q8f9o HTTP/1.1 301 Moved Server: nginx Date: Fri, 14 Sep 2012 21:53:14 GMT Content-Type: text/html; charset=utf-8 Connection: keep-alive Set-Cookie: _bit=5053a74a-0011d-0688d-311cf10a;domain=.bit.ly;expires=Wed Mar 13 21:53:14 2013;path=/; HttpOnly Cache-control: private; max-age=90 Location: https://davidwalsh.name/ MIME-Version: 1.0 Content-Length: 115
It's also useful to see the server name, expires information and more. I also appreciate that it's a clean list and no other information is pushed into the response. If you get some time, cURL out to different popular domains and see what headers they send -- you could be surprised!