Sunday, July 28, 2013

Fixing the client IP as perceived by Apache behind Nginx reverse proxy

In my recent post Using Nginx to reverse proxy a secure apache site that is using socket.io/node.js/websockets I found that I was only getting the proxy's IP address in all of my apache logs as well as in the application tracking. This does make sense since from Apache's perspective the proxy is the actual client and not the real remote client. I found however that it was possible to get around this. Apparently there are several Apache modules that can do this, however I did not want to build from sources if I could avoid it. So I choose the one that was already available in the EPEL repository.

I already had the EPEL repository registered on the server but for those that does not you can use the following commands to register it.
rpm -Uvh http://ftp.crc.dk/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
Now install the module
yum install mod_extract_forwarded
After installing the module you need to register an allowed forwarder (ie. the IP of your proxy). In all my logs the ip 127.0.0.1 was registered as the client of every request so adding that did the trick.
echo "MEFaccept 127.0.0.1" >> /etc/httpd/conf.d/mod_extract_forwarded.conf
service httpd restart
Recheck your webserver/application logs, this worked for me.

Thursday, July 25, 2013

Using Nginx to reverse proxy a secure apache site that is using socket.io/node.js/websockets

Challenge:
Firewalls can cause challenges, by blocking the ports that you want to use for websockets.

Many firewalls in use today are so called stateful firewalls and in short their function can be described as follows

Stateful firewalls only verify that a packet correlates to an existing, unclosed, connection. It tracks the state of the connection (opening, open, closing, closed) hence the name. When it detects a packet is part of an already open, authorized connection, it can short circuit all of the other rule checks and let the packet through.
(
http://stackoverflow.com/questions/1967943/will-html5-websockets-be-crippled-by-firewalls)
Most sites that are using node.js for websockets are also utilizing the library socket.io

A good list outlining some of the challenges when combining websockets/socket.io with various firewalls can be found at Socket.IO and firewall software.

So the question is how do you get your website running in spite of corporate firewall blocking various ports?


Possible solutions:

  1. ask the IT department to open the needed ports (in my experience hardly likely unless #2 fails)
  2. piggy-back on one of the existing established connections using http (port 80) or https (port 443) and hope the firewall is not a deep packet inspection based firewall.
 
Since I seldom have luck with suggestion number 1, I am not able to offer any guidance that is likely to help you succeed.

Suggestion number 2 however we can achieve that by letting the websockets use port 80/443 and let the website use the same port. You cannot however run 2 different services on the same port, so you cannot do this in a straightforward manner. The answer is a websocket aware reverse proxy.

There are quite a few different setups that can solve this. Below you can find 3 ways I investigated when I needed to solve this:


Setup
  • CentOS based Linux distribution
  • existing website installed as a Apache virtual host running on port 80 and 443
  • existing node.js socket.io based application running on port 8081


(setup before reverse proxy)


Please find an image of the wished for setup here below.

 
We need to do the following to achieve the reverse proxy setup
  1. Install Nginx
  2. Remove http -> https rewrite and ssl configuration from Apache as well as change Apache from listening on port 80/443 to listen on port 81
  3. Configure http -> https rewrite and reverse proxy in Nginx
  4. Change the socket.io node.js application to just use ws instead of wss (since Nginx is the SSL termination proxy now)
  5. Change the client application from using port 8081 to using 80/443

1. Install Nginx
RHEL / Centos 6: Install Nginx Using Yum Command
 


2. Remove http -> https rewrite and ssl configuration from Apache as well as change Apache from listening on port 80/443 to listen on port 81

Open the virtual hosts file for the site
vi /etc/httpd/conf.d/www_somewhere_com.conf
Change your virtual hosts file from the following 

# Catch those that are trying to access the site using http
# redirect to https
<VirtualHost *:80>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^(.*) https://www.somewhere.com [R,L]
</VirtualHost>

# Main site
<VirtualHost *:443>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common

  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/www_somewhere.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/www_somewhere.com.key
  SSLCertificateChainFile /etc/pki/tls/certs/Intermediate_Certificate.crt

  <Directory /var/www/sites/www.somewhere.com/www>
    Order Deny,Allow
    Allow from all
  </Directory>
</VirtualHost>

to the following

# Main site
<VirtualHost *:81>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common

  <Directory /var/www/sites/www.somewhere.com/www>
    Order Deny,Allow
    Allow from all
  </Directory>
</VirtualHost>

vi /etc/httpd/conf/httpd.conf
Look for the "Listen 80" in the file and change it to "Listen 81"
Save the file


3. Configure http -> https rewrite and reverse proxy in Nginx
vi /etc/nginx/conf.d/default.conf


replace the deafult content with this


Please note that in apache config you have a seperate intermidiate certificate and the site certificate. In Nginx you need to concatenate the primary certificate file (your_domain_name.crt) and the intermediate certificate file (DigiCertCA.crt) into a single pem file by running the following command:

cat DigiCertCA.crt >> your_domain_name.crt


# Listeners
# nginx         :80         responsible for redirecting http://www.somewhere.com to https://www.somewhere.com 
# nginx         :443        responsible for handling all trafic on the https://www.somewhere.com site
#                           acts as a reverse proxy for 
#                           /socket.io/     -> port 127.0.0.1:8081  (node.js websocket server)
#                           /               -> port 127.0.0.1:81    (apache  main site)
#
# node.js   :8081   node.js websocket server
# apache    :81     apache  main site


upstream www.somewhere.com {
    server 127.0.0.1:81;
    least_conn;
}

# nginx         :80         responsible for redirecting http://www.somewhere.com to https://www.somewhere.com 
server {
    listen 80;
    server_name  www.somewhere.com;
    rewrite ^(.*) https://www.somewhere.com$1 permanent;
}

# nginx         :443        responsible for handling all trafic on the https://www.somewhere.com site
#                           acts as a reverse proxy for 
#                           /socket.io/     -> port 127.0.0.1:8081  (node.js  websocket server)
#                           /               -> port 127.0.0.1:81    (apache  main site)
server {
    listen 443 ssl;
    server_name www.somewhere.com;

    ssl_certificate     /etc/pki/tls/certs/www_somewhere.com.crt;
    ssl_certificate_key /etc/pki/tls/private/www_somewhere.com.key;
    ssl_protocols       SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         RC4:HIGH:!aNULL:!MD5;

    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    # Path based websocket proxy location for nginx
    # to handle the  reverseproxying
    #
    # inital connect url: https://www.somewhere.com:8081/socket.io/1/?t=1374751734771
    # websocket url     :   wss://www.somewhere.com:8081/socket.io/1/websocket/x55Ch0B-SCACmDpW8rZd   
    location /socket.io/ {
        proxy_pass http://localhost:8081;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }   

    # Path based site proxy location for nginx
    # to handle the main apache  site
    location / {
        proxy_pass http://localhost:81;
    }   
}


4. Change the socket.io node.js application to just use ws instead of wss (since Nginx is the SSL termination proxy now)
Change you node.js socket.io application from this


var fs = require('fs');
var sslCertificate = {
    key: fs.readFileSync(config.sslCertificate.key),
    cert: fs.readFileSync(config.sslCertificate.cert),
    ca: fs.readFileSync(config.sslCertificate.ca)
};

// socket.io wss (websocket secure)
var io = require('socket.io').listen(8081, sslCertificate);
...

to the following

var fs = require('fs');
var sslCertificate = {
//    key: fs.readFileSync(config.sslCertificate.key),
//    cert: fs.readFileSync(config.sslCertificate.cert),
//    ca: fs.readFileSync(config.sslCertificate.ca)
};

// socket.io ws (websocket)
var io = require('socket.io').listen(8081, sslCertificate);
...


5. Change the client application from using port 8081 to using 80/443

Find the location in your client code where you are connecting to the websocket/socket.io application/server and change it from hardcoding the port in the url to just omitting it

So instead of (in a angular.js application / javascript single page application) doing this


var socketIO = function(data){
    // force https / wss
    var hostURL = window.location.href.replace('/'+window.location.hash, '8081').replace('http:', 'https:');

    ...
    
    try {
        var socket = io.connect(hostURL, {'sync disconnect on unload':true});
        ...
    }
    catch(e){
        console.log('Error connecting');
        ...
    }
    ...
}

do something like this
var socketIO = function(data){
    // force https / wss
    var hostURL = window.location.href.replace('/'+window.location.hash, '').replace('http:', 'https:');

    ...
    
    try {
        var socket = io.connect(hostURL, {'sync disconnect on unload':true});
        ...
    }
    catch(e){
        console.log('Error connecting');
        ...
    }
    ...
}

run the following command to check if the services are running on the correct ports

#netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      13146/nginx
tcp        0      0 0.0.0.0:8081                0.0.0.0:*                   LISTEN      13225/node
tcp        0      0 127.0.0.1:81                0.0.0.0:*                   LISTEN      13190/httpd
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      13146/nginx
...

Finally some before and after pictures from chrome
Before




After


Monday, July 01, 2013

Compressed Node.js install on CentOS 6.4

yum install -y make gcc cc gcc-c++ wget; cd /usr/src; wget http://nodejs.org/dist/v0.10.12/node-v0.10.12.tar.gz; tar zxf node-*.tar.gz; cd node-v*; ./configure; make; make install;