Unable to change network adapter UUID on Linux

I needed to reconfigure the hardware UUID of a Linux network adapter caused by cloning a virtual machine. To do this you need to either remove the /etc/udev/rules.d/70-persistent-net.rules and let the system regenerate the file on reboot, or manually edit the file.

I decided to edit the file and manually correct the UUID. I was consistently met "readonly file" so I could not save my changes.

I found using the lsattr /etc/udev/rules.d/70-persistent-net.rules that the file had the i attribute set. This i attribute means that the file is immutable. Removing this attribute using chattr -i /etc/udev/rules.d/70-persistent-net.rules, allowed me to change the file as needed. After editing the file I set the i attribute again using chattr +i /etc/udev/rules.d/70-persistent-net.rules.

Fixing the client IP as perceived by Apache behind Nginx reverse proxy

In my recent post Using Nginx to reverse proxy a secure apache site that is using socket.io/node.js/websockets I found that I was only getting the proxy's IP address in all of my apache logs as well as in the application tracking. This does make sense since from Apache's perspective the proxy is the actual client and not the real remote client. I found however that it was possible to get around this. Apparently there are several Apache modules that can do this, however I did not want to build from sources if I could avoid it. So I choose the one that was already available in the EPEL repository.

I already had the EPEL repository registered on the server but for those that does not you can use the following commands to register it.

rpm -Uvh http://ftp.crc.dk/fedora-epel/6/i386/epel-release-6-8.noarch.rpm

Now install the module

yum install mod_extract_forwarded

After installing the module you need to register an allowed forwarder (ie. the IP of your proxy). In all my logs the ip 127.0.0.1 was registered as the client of every request so adding that did the trick.

echo "MEFaccept 127.0.0.1" >> /etc/httpd/conf.d/mod_extract_forwarded.conf
service httpd restart

Recheck your webserver/application logs, this worked for me.

Using Nginx to reverse proxy a secure apache site that is using socket.io/node.js/websockets

Challenge:
Firewalls can cause challenges, by blocking the ports that you want to use for websockets.

Many firewalls in use today are so called stateful firewalls and in short their function can be described as follows

Stateful firewalls only verify that a packet correlates to an existing, unclosed, connection. It tracks the state of the connection (opening, open, closing, closed) hence the name. When it detects a packet is part of an already open, authorized connection, it can short circuit all of the other rule checks and let the packet through.
(http://stackoverflow.com/questions/1967943/will-html5-websockets-be-crippled-by-firewalls)

Most sites that are using node.js for websockets are also utilizing the library socket.io. 

A good list outlining some of the challenges when combining websockets/socket.io with various firewalls can be found at Socket.IO and firewall software.

So the question is how do you get your website running in spite of corporate firewall blocking various ports?


Possible solutions:

1. ask the IT department to open the needed ports (in my experience hardly likely unless #2 fails)
2. piggy-back on one of the existing established connections using http (port 80) or https (port 443) and hope the firewall is not a deep packet inspection based firewall.

 
Since I seldom have luck with suggestion number 1, I am not able to offer any guidance that is likely to help you succeed.

Suggestion number 2 however we can achieve that by letting the websockets use port 80/443 and let the website use the same port. You cannot however run 2 different services on the same port, so you cannot do this in a straightforward manner. The answer is a websocket aware reverse proxy.

There are quite a few different setups that can solve this. Below you can find 3 ways I investigated when I needed to solve this:

• Apache acting as a reverse proxy for node.js (this was my first choice, but I did not manage to get this working with out-of-the-box available packages for the Linux distribution I am using, but you can more than likely build your own version that will work just fine Backporting Apache support for websockets reverse proxy (aka getting GateOne to work behind Apache) and Using apache-websocket to proxy tcp connections).
• Use node.js to proxy Apache (see more here https://github.com/nodejitsu/node-http-proxy). (I never managed to get this working properly either but that is more than likely due to my slightly limited exposure to node.js, and to be honest my experience with node has been a slightly turbulent one, so I feel that Apache/Nginx are the more stable options for me, that is not to say that this is not working or an great option.
• Nginx being a reverse proxy for both apache and websockets/socket.io. This is what I will describe below.

Setup

• CentOS based Linux distribution
• existing website installed as a Apache virtual host running on port 80 and 443
• existing node.js socket.io based application running on port 8081

(setup before reverse proxy)

Please find an image of the wished for setup here below.

 
We need to do the following to achieve the reverse proxy setup

1. Install Nginx
2. Remove http -> https rewrite and ssl configuration from Apache as well as change Apache from listening on port 80/443 to listen on port 81
3. Configure http -> https rewrite and reverse proxy in Nginx
4. Change the socket.io node.js application to just use ws instead of wss (since Nginx is the SSL termination proxy now)
5. Change the client application from using port 8081 to using 80/443

1. Install Nginx
RHEL / Centos 6: Install Nginx Using Yum Command
 

2. Remove http -> https rewrite and ssl configuration from Apache as well as change Apache from listening on port 80/443 to listen on port 81

Open the virtual hosts file for the site
vi /etc/httpd/conf.d/www_somewhere_com.conf
Change your virtual hosts file from the following 

# Catch those that are trying to access the site using http
# redirect to https
<VirtualHost *:80>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^(.*) https://www.somewhere.com [R,L]
</VirtualHost>

# Main site
<VirtualHost *:443>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common

  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/www_somewhere.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/www_somewhere.com.key
  SSLCertificateChainFile /etc/pki/tls/certs/Intermediate_Certificate.crt

  <Directory /var/www/sites/www.somewhere.com/www>
    Order Deny,Allow
    Allow from all
  </Directory>
</VirtualHost>

to the following

# Main site
<VirtualHost *:81>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common

  <Directory /var/www/sites/www.somewhere.com/www>
    Order Deny,Allow
    Allow from all
  </Directory>
</VirtualHost>

vi /etc/httpd/conf/httpd.conf
Look for the "Listen 80" in the file and change it to "Listen 81"
Save the file

3. Configure http -> https rewrite and reverse proxy in Nginx
vi /etc/nginx/conf.d/default.conf

replace the deafult content with this

Please note that in apache config you have a seperate intermidiate certificate and the site certificate. In Nginx you need to concatenate the primary certificate file (your_domain_name.crt) and the intermediate certificate file (DigiCertCA.crt) into a single pem file by running the following command:

cat DigiCertCA.crt >> your_domain_name.crt

# Listeners
# nginx         :80         responsible for redirecting http://www.somewhere.com to https://www.somewhere.com 
# nginx         :443        responsible for handling all trafic on the https://www.somewhere.com site
#                           acts as a reverse proxy for 
#                           /socket.io/     -> port 127.0.0.1:8081  (node.js websocket server)
#                           /               -> port 127.0.0.1:81    (apache  main site)
#
# node.js   :8081   node.js websocket server
# apache    :81     apache  main site


upstream www.somewhere.com {
    server 127.0.0.1:81;
    least_conn;
}

# nginx         :80         responsible for redirecting http://www.somewhere.com to https://www.somewhere.com 
server {
    listen 80;
    server_name  www.somewhere.com;
    rewrite ^(.*) https://www.somewhere.com$1 permanent;
}

# nginx         :443        responsible for handling all trafic on the https://www.somewhere.com site
#                           acts as a reverse proxy for 
#                           /socket.io/     -> port 127.0.0.1:8081  (node.js  websocket server)
#                           /               -> port 127.0.0.1:81    (apache  main site)
server {
    listen 443 ssl;
    server_name www.somewhere.com;

    ssl_certificate     /etc/pki/tls/certs/www_somewhere.com.crt;
    ssl_certificate_key /etc/pki/tls/private/www_somewhere.com.key;
    ssl_protocols       SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         RC4:HIGH:!aNULL:!MD5;

    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    # Path based websocket proxy location for nginx
    # to handle the  reverseproxying
    #
    # inital connect url: https://www.somewhere.com:8081/socket.io/1/?t=1374751734771
    # websocket url     :   wss://www.somewhere.com:8081/socket.io/1/websocket/x55Ch0B-SCACmDpW8rZd   
    location /socket.io/ {
        proxy_pass http://localhost:8081;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }   

    # Path based site proxy location for nginx
    # to handle the main apache  site
    location / {
        proxy_pass http://localhost:81;
    }   
}

4. Change the socket.io node.js application to just use ws instead of wss (since Nginx is the SSL termination proxy now)
Change you node.js socket.io application from this

var fs = require('fs');
var sslCertificate = {
    key: fs.readFileSync(config.sslCertificate.key),
    cert: fs.readFileSync(config.sslCertificate.cert),
    ca: fs.readFileSync(config.sslCertificate.ca)
};

// socket.io wss (websocket secure)
var io = require('socket.io').listen(8081, sslCertificate);
...

to the following

var fs = require('fs');
var sslCertificate = {
//    key: fs.readFileSync(config.sslCertificate.key),
//    cert: fs.readFileSync(config.sslCertificate.cert),
//    ca: fs.readFileSync(config.sslCertificate.ca)
};

// socket.io ws (websocket)
var io = require('socket.io').listen(8081, sslCertificate);
...

5. Change the client application from using port 8081 to using 80/443

Find the location in your client code where you are connecting to the websocket/socket.io application/server and change it from hardcoding the port in the url to just omitting it

So instead of (in a angular.js application / javascript single page application) doing this

var socketIO = function(data){
    // force https / wss
    var hostURL = window.location.href.replace('/'+window.location.hash, '8081').replace('http:', 'https:');

    ...
    
    try {
        var socket = io.connect(hostURL, {'sync disconnect on unload':true});
        ...
    }
    catch(e){
        console.log('Error connecting');
        ...
    }
    ...
}

do something like this

var socketIO = function(data){
    // force https / wss
    var hostURL = window.location.href.replace('/'+window.location.hash, '').replace('http:', 'https:');

    ...
    
    try {
        var socket = io.connect(hostURL, {'sync disconnect on unload':true});
        ...
    }
    catch(e){
        console.log('Error connecting');
        ...
    }
    ...
}

run the following command to check if the services are running on the correct ports

#netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      13146/nginx
tcp        0      0 0.0.0.0:8081                0.0.0.0:*                   LISTEN      13225/node
tcp        0      0 127.0.0.1:81                0.0.0.0:*                   LISTEN      13190/httpd
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      13146/nginx
...

Finally some before and after pictures from chrome
Before

After

Backing up SugarCrm and MySql

When running SugarCrm in a production environment, you need to do backup. There are many opinions about how to do backup:

Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)

(Torvalds, Linus (1996-07-20). Post. linux.dev.kernel newsgroup. Google Groups. Retrieved on 2006-08-28.)

However the above might not be what your boss want, so we need to schedule backup.

Backing up SugarCrm is basically 2 separate operations.

1. zipping  up your working site directory structure
2. dumping the associated SugarCrm database

On a Linux system #1 can be done with the tar command (or any other file packer of preference).

tar -czf SITE_DIRECTORY.tar.gz SITE_DIRECTORY

For #2 it depends on the database that you are using. In the remaining part of this posting I will be using MySQL and InnoDB as storage engine.

There are numerous scopes for backing up a database:

• full backup
• incremental backup
• block level backup

Normally I prefer the full backup option combined with a block level backup plan for incremental backup.

Simple full backup can be controlled using AutoMySQLBackup. There are however additional parameters that I like to add to the mysqldump command used inside the script. If you are using any kinds of functions/stored procedures in the database, then you want to add --triggers --routines to the mysqldump command. I also like to use the --single-transaction option on InnoDB storage engine (this is also mentioned here as an upcoming feature for AutoMySQLBackup).

The above mentioned options/parameters will change the lines in the script from 

mysqldump --user=XXX -- password=YYY ... 

to

mysqldump --user=XXX -- password=YYY --triggers --routines --single-transaction ...

For incremental backup please see the ibbackup command or the mysqlbackup command.

The majority of our servers are running on Amazon EC2. We use a combination of running full backups nightly as well as Amazon Elastic Block Store (EBS) (block level backup) snapshots through the day. This requires a slightly special setup with regards to your filesystem of choice. You can read more at Creating Consistent EBS Snapshots with MySQL and XFS on EC2 or if you prefer CentOS then you can have a look at my blog posting Setting up Centos 5.6 with PHP 5.3 on Amazon EC2 with ec2-consistent-snapshot.

Setting up Centos 5.6 with PHP 5.3 on Amazon EC2 with ec2-consistent-snapshot

# Command order to get a CentOS 5.6 running with PHP 5.3 and ec2-consistent-snapshot running on AWS

# Change hostname
hostname YOUR_HOSTNAME
vi /etc/sysconfig/network
------------------------------------
HOSTNAME=YOUR_HOSTNAME
NETWORKING=yes
NETWORKING_IPV6=no
------------------------------------


#Install PHP and modules and APC
yum install php53 php53-mbstring php53-mysql php53-gd gd php53-imap php53-ldap php53-pdo php53-cli php53-pear php53-devel httpd-devel pcre-devel php-pear
pear upgrade --force Console_Getopt
pear upgrade --force pear
pear upgrade-all
pear version
pecl install apc

# Attach and mount a AWS EBS volumn (ec2-consistent-snapshot prefers xfs) 
# Make sure there is xfs support
yum install kmod-xfs xfsdump xfsprogs dmapi
mkfs.xfs  /dev/sdf
echo "/dev/sdf /vol xfs defaults,noatime 0 0" >> /etc/fstab
mkdir /vol
mount /vol

# Mysql
yum install mysql-server
mkdir /vol/mysql
chown -R mysql:mysql /vol/mysql
vi /etc/my.cnf

# Change the my.cnf file to 
------------------------------------
[mysqld]
datadir=/vol/mysql
user=mysql
log-warnings=2
log-error=/var/log/mysqld.log
default-storage-engine=innodb
skip-bdb
socket=/var/lib/mysql/mysql.sock

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
------------------------------------

service mysqld start
mysqladmin -u root password YOUR_PASSWORD
chkconfig mysqld on


# ec2-consistent-snapshot
perl -MCPAN -e shell
cpan>o conf prerequisites_policy follow
cpan>o conf commit
perl -MCPAN -e "install(q{Bundle::CPAN})"
perl -MCPAN -e "install Net::Amazon::EC2"
yum -y install perl-DBD-MySQL
yum -y install 'perl(File::Slurp)'
yum -y install 'perl(DBI)'
yum -y install 'perl(Net::SSLeay)'
yum -y install 'perl(IO::Socket::SSL)'
yum -y install 'perl(LWP::Protocol::https)'
wget http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/download/head:/ec2consistentsnapsho-20090928015038-9m9x0fc4yoy54g4j-1/ec2-consistent-snapshot
mkdir /home/ec2/bin/
mv ec2-consistent-snapshot /home/ec2/bin/
chmod 700 /home/ec2/bin/ec2-consistent-snapshot

# Create / EBS snapshot script
cd /root
vi snapshot_ebs_root.sh
------------------------------------
#!/bin/sh

PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/home/ec2/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/ho
me/ec2/bin:/root/bin

description="`hostname`-rootdrive-`date +%Y-%m-%d-%H-%M-%S`"

ec2-consistent-snapshot --aws-access-key-id=YOUR_ACCESS_KEY --aws-secret-access-key=YOUR_SECRET_ACCESS_KEY \
--description=$description --region=eu-west-1 YOUR_VOLUME_ID
------------------------------------
chmod 700 snapshot_ebs_root.sh


# Create /vol EBS snapshot script
vi ./snapshot_ebs_vol.sh
------------------------------------
#!/bin/sh

PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/home/ec2/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/ho
me/ec2/bin:/root/bin

description="`hostname`-rootdrive-`date +%Y-%m-%d-%H-%M-%S`"

ec2-consistent-snapshot --aws-access-key-id=YOUR_ACCESS_KEY --aws-secret-access-key=YOUR_SECRET_ACCESS_KEY \
--mysql-username=root --mysql-password=YOUR_DB_ROOT_PASSWORD --mysql-host=localhost --description=$description --mysql --region=eu-west-1 \
--freeze-filesystem='/vol' YOUR_VOLUME_ID
------------------------------------
chmod 700 snapshot_ebs_vol.sh


#Set the cronschedule for EBS snapshots
crontab -e
------------------------------------
54 6-23 * * * /root/snapshot_ebs_vol.sh > /dev/null 2>&1
59 22 * * * /root/snapshot_ebs_root.sh > /dev/null 2>&1
------------------------------------

# Change httpd DocumentRoot to /vol/www/default
vi /etc/httpd/conf/httpd.conf
mkdir /vol/www/default
mkdir /vol/www/sites
chown -R apache:apache /vol/www
chkconfig httpd on
service httpd start

#Set server timezone
cd /etc
ln -sf /usr/share/zoneinfo/CET localtime