Friday, December 11, 2009

Performing manual GET requests against both HTTP and HTTPS

Sometimes it is useful to perform manual GET requests to a particular server. You can provide any request headers or cookies, as well as view the headers sent back by the server. This is useful in situations when you need to debug your applications.

Usually you can do this with telnet or netcat, unless it is an HTTPS service. I will explain how to query both services.


1. HTTP
Here is how to query an HTTP service using telnet/netcat:
netcat google.com 80

This will open a connection to google.com using port 80. The connection will wait for you to perform your query:
GET / HTTP/1.0

After typing this, you need to press enter twice, as specified in the HTTP standard. After doing this, you should get a response from the server:

HTTP/1.0 200 OK
Date: Fri, 11 Dec 2009 21:59:04 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=..............; expires=Sun, 11-Dec-2011 21:59:04 GMT; path=/; domain=.google.com
Set-Cookie: NID=29=................; expires=Sat, 12-Jun-2010 21:59:04 GMT; path=/; domain=.google.com; HttpOnly
Server: gws
X-XSS-Protection: 0


Here, you can see all of the headers and cookies sent by the server. If you want, you can also send specific headers and cookies:

GET / HTTP/1.0
Host: http://www.google.com

2. HTTPS
Now for the fun part. OpenSSL is the library that encrypts/decrypts HTTPS (SSL) traffic, allowing you to communicate securely. This is available on Linux/Mac/Windows(cygwin). The OpenSSL installation comes with the command 'openssl' which has several modes of operation. One of them, 's_client' provides you with a simple ssl client that handles all security handshaking. Here is how to perform a GET request:
openssl s_client -connect mail.google.com:443

As soon as you run this, you will see some details on the SSL handshaking. First, you will see the details of security certificate, followed by the actual certificate:

Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
i:/C=US/O=Google Inc/CN=Google Internet Authority
1 s:/C=US/O=Google Inc/CN=Google Internet Authority
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDXDCCAsWgAwIBAgIKUmX0TgADAAANmjANBgkqhkiG9w0BAQUFADBGMQswCQYD
VQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzEiMCAGA1UEAxMZR29vZ2xlIElu
dGVybmV0IEF1dGhvcml0eTAeFw0wOTExMTIyMDM2MDBaFw0xMDExMTIyMDQ2MDBa
MGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1N
b3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRgwFgYDVQQDEw9tYWls
Lmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKW0UD+4ExS1
6W+aPjDdBuyqJm7c3p5O0p7sZCHiTqENOiIafxWlG6oRDkm0VOZqJY0Q9jHOEA5R
szygSWuuKqRjo/gBC6iahJHK92kOTzZ5JqzVAvkC/4UOXJjyOq1i+/euE5FPpzDy
z8z1aY5fgNM1KgylBBaY6O3tmfnJyv5fAgMBAAGjggEsMIIBKDAdBgNVHQ4EFgQU
i/Te/XX4P2mk9V/jRr/wkzCFtaIwHwYDVR0jBBgwFoAUv8Aw6/VDET5nup6R+/xq
2uNrEiQwWwYDVR0fBFQwUjBQoE6gTIZKaHR0cDovL3d3dy5nc3RhdGljLmNvbS9H
b29nbGVJbnRlcm5ldEF1dGhvcml0eS9Hb29nbGVJbnRlcm5ldEF1dGhvcml0eS5j
cmwwZgYIKwYBBQUHAQEEWjBYMFYGCCsGAQUFBzAChkpodHRwOi8vd3d3LmdzdGF0
aWMuY29tL0dvb2dsZUludGVybmV0QXV0aG9yaXR5L0dvb2dsZUludGVybmV0QXV0
aG9yaXR5LmNydDAhBgkrBgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMA0G
CSqGSIb3DQEBBQUAA4GBAIvbjRwrU/ssArVMt07uzh4LUjd4Zdi0Gn9ByrtBf2h1
JMItRqGh54AwGY2dmR1ffYWkP+q4IBCts3kkxUPvBf5BaqIqxuvAljN4+1yzphGr
hxYbLC8mfz0aCLnVWWclPpvP62swYPuHca6m/Og5SJeFkyf3VdTemeh0lZZO3vhf
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority


This is useful if you need to download the certificate and safe it to a certificate store. After everything is displayed, you can type your GET request as before:
GET / HTTP/1.0

Everything will be decrypted and shown to you just like netcat/telnet.