Performing manual GET requests against both HTTP and HTTPS

Sometimes it is useful to perform manual GET requests to a particular server. You can provide any request headers or cookies, as well as view the headers sent back by the server. This is useful in situations when you need to debug your applications.

Usually you can do this with telnet or netcat, unless it is an HTTPS service. I will explain how to query both services.


1. HTTP
Here is how to query an HTTP service using telnet/netcat:

netcat google.com 80

This will open a connection to google.com using port 80. The connection will wait for you to perform your query:

GET / HTTP/1.0

After typing this, you need to press enter twice, as specified in the HTTP standard. After doing this, you should get a response from the server:

HTTP/1.0 200 OK
Date: Fri, 11 Dec 2009 21:59:04 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=..............; expires=Sun, 11-Dec-2011 21:59:04 GMT; path=/; domain=.google.com
Set-Cookie: NID=29=................; expires=Sat, 12-Jun-2010 21:59:04 GMT; path=/; domain=.google.com; HttpOnly
Server: gws
X-XSS-Protection: 0

Here, you can see all of the headers and cookies sent by the server. If you want, you can also send specific headers and cookies:

GET / HTTP/1.0
Host: http://www.google.com

2. HTTPS
Now for the fun part. OpenSSL is the library that encrypts/decrypts HTTPS (SSL) traffic, allowing you to communicate securely. This is available on Linux/Mac/Windows(cygwin). The OpenSSL installation comes with the command 'openssl' which has several modes of operation. One of them, 's_client' provides you with a simple ssl client that handles all security handshaking. Here is how to perform a GET request:

openssl s_client -connect mail.google.com:443

As soon as you run this, you will see some details on the SSL handshaking. First, you will see the details of security certificate, followed by the actual certificate:

Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDXDCCAsWgAwIBAgIKUmX0TgADAAANmjANBgkqhkiG9w0BAQUFADBGMQswCQYD
VQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzEiMCAGA1UEAxMZR29vZ2xlIElu
dGVybmV0IEF1dGhvcml0eTAeFw0wOTExMTIyMDM2MDBaFw0xMDExMTIyMDQ2MDBa
MGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1N
b3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRgwFgYDVQQDEw9tYWls
Lmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKW0UD+4ExS1
6W+aPjDdBuyqJm7c3p5O0p7sZCHiTqENOiIafxWlG6oRDkm0VOZqJY0Q9jHOEA5R
szygSWuuKqRjo/gBC6iahJHK92kOTzZ5JqzVAvkC/4UOXJjyOq1i+/euE5FPpzDy
z8z1aY5fgNM1KgylBBaY6O3tmfnJyv5fAgMBAAGjggEsMIIBKDAdBgNVHQ4EFgQU
i/Te/XX4P2mk9V/jRr/wkzCFtaIwHwYDVR0jBBgwFoAUv8Aw6/VDET5nup6R+/xq
2uNrEiQwWwYDVR0fBFQwUjBQoE6gTIZKaHR0cDovL3d3dy5nc3RhdGljLmNvbS9H
b29nbGVJbnRlcm5ldEF1dGhvcml0eS9Hb29nbGVJbnRlcm5ldEF1dGhvcml0eS5j
cmwwZgYIKwYBBQUHAQEEWjBYMFYGCCsGAQUFBzAChkpodHRwOi8vd3d3LmdzdGF0
aWMuY29tL0dvb2dsZUludGVybmV0QXV0aG9yaXR5L0dvb2dsZUludGVybmV0QXV0
aG9yaXR5LmNydDAhBgkrBgEEAYI3FAIEFB4SAFcAZQBiAFMAZQByAHYAZQByMA0G
CSqGSIb3DQEBBQUAA4GBAIvbjRwrU/ssArVMt07uzh4LUjd4Zdi0Gn9ByrtBf2h1
JMItRqGh54AwGY2dmR1ffYWkP+q4IBCts3kkxUPvBf5BaqIqxuvAljN4+1yzphGr
hxYbLC8mfz0aCLnVWWclPpvP62swYPuHca6m/Og5SJeFkyf3VdTemeh0lZZO3vhf
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority

This is useful if you need to download the certificate and safe it to a certificate store. After everything is displayed, you can type your GET request as before:

GET / HTTP/1.0

Everything will be decrypted and shown to you just like netcat/telnet.

Secure your Firefox passwords!

Your passwords in Firefox are not securely encrypted by default. Your passwords are stored in a file called signons3.txt, which is encrypted using the key found in key3.db. However, the passwords can still be viewed in the preferences! It is very easy to fix this by following these steps below:



1. Choose the Security tab in preferences:







2. Enter your password:







Now your passwords will be kept save. The password file will now be encrypted with a master password, and cannot be viewed without typing in your password.

perl & java IPC

recently, i've ran into a situation where i've needed my perl cron script to communicate with java. i did not want to start up a new jvm or create database connections for each call, as this would be very sloppy. below is an example of how to do this.

in this example, perl starts the java process and communicates via standard in/out/err. right now, the java program only provides a useless 'reverse' service, but could be updated to do something more useful. if the java process has any exceptions, it will handle those and exit normally.

Main.java

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;

public class Main{
        public static void main(String args[]){
                BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
                boolean running = true;
                int processed = 0;

                while(running){
                        try{
                                String s = in.readLine();
                                if (s == null || "".equals(s)){
                                        running = false;
                                }else{
                                        System.out.println(new StringBuffer(s).reverse().toString());
                                        processed++;
                                }
                                if (processed == 100) throw new IllegalArgumentException("blah");
                        }catch(IOException e){
                                e.printStackTrace();
                        }
                }
                System.err.println("successfully processed " + processed);
        }
}

pipe.pl

#!/usr/bin/perl -w


use FileHandle;
use IPC::Open3;
use strict;

my $pid = open3(*CHILD_IN, *CHILD_OUT, *CHILD_ERR, "java Main");
CHILD_IN->autoflush();

my $out = "";
my $err = "";
my $i = 0;
for(1..1000000){
        $i++;
        print CHILD_IN "hello there $i\n";
        $out = ;
        if (defined $out){
                chomp($out);
                print("received: $out\n");
        }else{
                # error handling:
                while($err = ){
                        chomp($err);
                        print("ERROR: $err\n");
                }
                last;
        }

}

print CHILD_OUT "\n"; # send blank line to stop

# wait for process to end
waitpid($pid, 0);

print("stopping\n");

console highlighter

i've been squinting at my glassfish log tails for far too long now. i decided to write a small script that would highlight rows containing particular keywords.

this script will show error messages in red, and warning messages in yellow:

#!/usr/bin/perl

use strict;
use Term::ANSIColor;

while (<stdin>){
   if ($_ =~ m/(exception|error)/i){
       print color 'bold red';
       print $_;
       print color 'reset';
   }elsif ($_ =~ m/(warning)/i){
       print color 'bold yellow';
       print $_;
       print color 'reset';
   }else{
       print $_;
   }
}

this was pretty straight forward and can be easily modified. to use this, i put this in a script called highlighter.pl, and pipe my logs through it:

tail -F server.log | ~/bin/highlighter.pl

Show Selectors

here is a simple perl script for parsing a project for css files. it will display occurrences of css id's and classes. this is useful if you need to display unused id's/classes.

show-selectors.pl

#!/usr/bin/perl
use File::List;
use CSS;

my $cssl = new File::List(".");
my @css = @{ $cssl->find("\.css\$") };
my $cssc = @css;
my $seen;
my @final;
        
for my $cssfile (@css){
        my $css = CSS->new();
        $css->read_file($cssfile);

        for my $style ( @{ $css->{styles} }){
                for my $selector ( @{ $style->{selectors}}){
                        my $name = $selector->{name};
                        my @multiple = split(/ /, $name);

                        foreach $multiple (@multiple){
                                if ($multiple =~ /.*(\.|\#)(.*)/){
                                        my $sel = $1 . $2;
                                        $seen{$sel}++;
                                        next if $seen{$sel} > 1;
                                        push(@final, $sel);
                                }
                        }
                }
        }
}

foreach $selector (@final){
        if ($selector =~ /.(.*)/){
                my $total = 0;
                my $output = `grep -c '$1' . -R | grep -v -E '.svn|css'`;
                foreach $line (split(/\n/, $output)){
                        if ($line =~ /(.*):(\d)/){
                                my $occ = eval($2);
                                $total += $occ;
                        }
                }
                print("$selector: $total\n");
        }
}