Monday, September 19, 2011

SugarCrm: Continous / automated installation of packages


I needed to find a way to faster go from a bug being reported "Development Complete" to it being installed on a test installation for the testers so they could begin testing. We are not hosting the entire sugarcrm site in SVN and also deployment to the final production system is done by a third party so all we can so is send them the package zip files. So it needed to follow the Module Loader principle.

This script is run by cron at an timed interval (I am using 10 minutes).

Disclaimer
The script could be cleaned up by extracting the svn username and password into configuration variables.

The other suboptimal issue is:
When you are installing multiple packages this way and all packages are having post_install.php files defining the same functions you will get an error. The reason is that the post_install.php file currently is just a php file with named functions in it.The correct way to solve this would be to move the post_install.php functions inside a class. However that was not an option for me so I needed to handle the re-including of functions with the same name. To be able to do this I am renaming the already included functions.


#!/usr/bin/php
<?php
/**
* You must install 
* 
* yum install php-pear
* pecl install apd
* 
* and enabled in the php.ini file 
* 
* [Zend]
* zend_extension = /usr/lib/php/modules/apd.so
* 
* for this to work
*/


//make sure we dump the stuff to stderr
ini_set("error_log","php://stderr");

$svnExportDir='/mnt/';
$siteInstalledVersionStateFilesDir='/mnt/';

$svnPackages = array (
 // Define your packages here and the full SVN url
 'mypackage' => 'https://XXX/svn/mypackage/trunk',
 'mypackage2' => 'https://XXX/svn/mypackage2/trunk',
);

$preInstallFunctions = array (
 // If your package are using pre_install hooks and you are installing multiple packages and the pre_install hooks contain the same functions
 // then you need to define them here. This is hack, if the pre_install functions was defined in a class then we could just unset the class.
    "pre_install",
);

$postInstallFunctions = array (
 // If your package are using post_install hooks and you are installing multiple packages and the pre_install hooks contain the same functions
 // then you need to define them here. This is hack, if the post_install functions was defined in a class then we could just unset the class.
    "post_install",
);
 
$sugarcrmInstallationsToUpdate = array (
  0 => array(
    'name' => 'SugarCrmInstallation1 (staging)',
    'sugarInstallDir' => '/var/www/sites/staging',
    'packagesToUpdate' => array (
        0 => 'mypackage',
        1 => 'mypackage2',
    ),
  ),
);


//initialize
if(!defined('sugarEntry'))  define('sugarEntry', true);

// If you are updating more than one installation there is no need to do the svn export more than one time per
// package, just reuse that already downloaded/exported package
$packageDownloadedFromSvnRevisionLog = $siteInstalledVersionStateFilesDir . 'ci_packageDownloadedFromSvnRevision.log';
$packageVersionsDownloaded = getArrayFromFile($packageDownloadedFromSvnRevisionLog);

foreach($sugarcrmInstallationsToUpdate as $sugarCrmSite){
    chdir($sugarCrmSite['sugarInstallDir']);
    require_once('include/entryPoint.php');
    require_once('ModuleInstall/ModuleInstaller.php');
    $current_user = new User();
    $current_user->is_admin = '1';
    
    $siteInstalledVersionsStateFile = $siteInstalledVersionStateFilesDir . $sugarCrmSite['name'] . '_version';
    $revisionNumberInstalledInSite = getArrayFromFile($siteInstalledVersionsStateFile);
    
    foreach($sugarCrmSite['packagesToUpdate'] as $key=>$package){
        $packageRevisionInstalled = array_key_exists($package, $revisionNumberInstalledInSite) ?  $revisionNumberInstalledInSite[$package] : 0;

        ciLog ("Checking for newer versions in svn ...\n", $sugarCrmSite['name'],$package);
        $packageRevisionInSvn = trim(`svn info --non-interactive --username YOUR_SVN_USERNAME --password YOUR_SVN_PASSWORD $svnPackages[$package] | grep 'Last Changed Rev' | head -1 | grep -Eo "[0-9]+"`);
        
        if ($packageRevisionInstalled < $packageRevisionInSvn) {
   ciLog ("There are updates in SVN (installed: $packageRevisionInstalled, svn: $packageRevisionInSvn).\n", $sugarCrmSite['name'],$package);
            
            $packageRevisionPreviouslyDownloaded = array_key_exists($package, $packageVersionsDownloaded) ?  $packageVersionsDownloaded[$package] : 0;
            if ($packageRevisionPreviouslyDownloaded < $packageRevisionInSvn){
    ciLog ("Getting latest version from svn ...\n", $sugarCrmSite['name'],$package);

                $output = `svn export --force --non-interactive --username YOUR_SVN_USERNAME --password YOUR_SVN_PASSWORD $svnPackages[$package] $svnExportDir$package`;
                $packageVersionsDownloaded[$package] = $packageRevisionInSvn;
                saveArrayToFile($packageDownloadedFromSvnRevisionLog, $packageVersionsDownloaded);
            }
            else {
    ciLog ("(rev $packageRevisionInSvn) has previously been downloaded, using local version.\n", $sugarCrmSite['name'],$package);
            }

            //initialize the module installer
            $modInstaller = new ModuleInstaller();
            $modInstaller->silent = true;  //shuts up the javscript progress bar

            //start installation
   ciLog ("(rev $packageRevisionInSvn) Starting installation into " . $sugarCrmSite['sugarInstallDir'] . " ... \n", $sugarCrmSite['name'],$package);
   
   $preInstallFile = "$svnExportDir$package/scripts/pre_install.php";
   if(is_file($preInstallFile))
   {
    ciLog ("Including $preInstallFile.\n", $sugarCrmSite['name'],$package);
    include($preInstallFile);
    ciLog ("Executing $preInstallFile.\n", $sugarCrmSite['name'],$package);
    pre_install();
                
                // Undeclaring the pre_install functions so we can include the next pre_install file. Nasty, this is a hack. Requires PECL ADP 
                undeclareFunctions($preInstallFunctions);
   }
            $modInstaller->install($svnExportDir.$package);

   $postInstallFile = "$svnExportDir$package/scripts/post_install.php";
   if(is_file($postInstallFile))
   {
                $_REQUEST['unzip_dir'] = "$svnExportDir$package";
    ciLog ("Including $postInstallFile.\n", $sugarCrmSite['name'],$package);
    include($postInstallFile);
    ciLog ("Executing $postInstallFile.\n", $sugarCrmSite['name'],$package);
    post_install();

                // Undeclaring the post_install functions so we can include the next pre_install file. Nasty, this is a hack. Requires PECL ADP 
                undeclareFunctions($postInstallFunctions);
   }
   ciLog ("(rev $packageRevisionInSvn) Installation into " . $sugarCrmSite['sugarInstallDir'] . " is done.\n", $sugarCrmSite['name'],$package);
            
            $revisionNumberInstalledInSite[$package] = $packageRevisionInSvn;
        }
        else {
   ciLog ("No updates found (installed: $packageRevisionInstalled, svn: $packageRevisionInSvn).\n", $sugarCrmSite['name'],$package);
        }
    }
    // ciLog what package revisions that were installed
    saveArrayToFile($siteInstalledVersionsStateFile, $revisionNumberInstalledInSite);
    
    // Make sure apache owns the sugar installation
    $sugarInstallDir = $sugarCrmSite['sugarInstallDir'];
    $output = `chown -R apache:apache $sugarInstallDir`;
}

function undeclareFunctions($functions){
    foreach($functions as $function) {
       if (function_exists($function)) { rename_function($function, uniqid()); } 
    }   
}

function ciLog($message, $site, $package){
 $fullMessage = "%s %s-%s: " . $message;
 printf($fullMessage, date('m/d/Y H:i:s'), $site, $package);
}

function getArrayFromFile($filename){
    if (file_exists($filename)) {
        return unserialize(file_get_contents($filename));
    }
    else {
        return array();
    }
}

function saveArrayToFile($filename, $arrayToSave){
    $fh = fopen($filename, 'w') or die("can't open file");
    fwrite($fh, serialize($arrayToSave));
    fclose($fh);
}
?>

Wednesday, September 14, 2011

Setting up Centos 5.6 with PHP 5.3 on Amazon EC2 with ec2-consistent-snapshot

# Command order to get a CentOS 5.6 running with PHP 5.3 and ec2-consistent-snapshot running on AWS

# Change hostname
hostname YOUR_HOSTNAME
vi /etc/sysconfig/network
------------------------------------
HOSTNAME=YOUR_HOSTNAME
NETWORKING=yes
NETWORKING_IPV6=no
------------------------------------


#Install PHP and modules and APC
yum install php53 php53-mbstring php53-mysql php53-gd gd php53-imap php53-ldap php53-pdo php53-cli php53-pear php53-devel httpd-devel pcre-devel php-pear
pear upgrade --force Console_Getopt
pear upgrade --force pear
pear upgrade-all
pear version
pecl install apc

# Attach and mount a AWS EBS volumn (ec2-consistent-snapshot prefers xfs) 
# Make sure there is xfs support
yum install kmod-xfs xfsdump xfsprogs dmapi
mkfs.xfs  /dev/sdf
echo "/dev/sdf /vol xfs defaults,noatime 0 0" >> /etc/fstab
mkdir /vol
mount /vol

# Mysql
yum install mysql-server
mkdir /vol/mysql
chown -R mysql:mysql /vol/mysql
vi /etc/my.cnf

# Change the my.cnf file to 
------------------------------------
[mysqld]
datadir=/vol/mysql
user=mysql
log-warnings=2
log-error=/var/log/mysqld.log
default-storage-engine=innodb
skip-bdb
socket=/var/lib/mysql/mysql.sock

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
------------------------------------

service mysqld start
mysqladmin -u root password YOUR_PASSWORD
chkconfig mysqld on


# ec2-consistent-snapshot
perl -MCPAN -e shell
cpan>o conf prerequisites_policy follow
cpan>o conf commit
perl -MCPAN -e "install(q{Bundle::CPAN})"
perl -MCPAN -e "install Net::Amazon::EC2"
yum -y install perl-DBD-MySQL
yum -y install 'perl(File::Slurp)'
yum -y install 'perl(DBI)'
yum -y install 'perl(Net::SSLeay)'
yum -y install 'perl(IO::Socket::SSL)'
yum -y install 'perl(LWP::Protocol::https)'
wget http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/download/head:/ec2consistentsnapsho-20090928015038-9m9x0fc4yoy54g4j-1/ec2-consistent-snapshot
mkdir /home/ec2/bin/
mv ec2-consistent-snapshot /home/ec2/bin/
chmod 700 /home/ec2/bin/ec2-consistent-snapshot

# Create / EBS snapshot script
cd /root
vi snapshot_ebs_root.sh
------------------------------------
#!/bin/sh

PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/home/ec2/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/ho
me/ec2/bin:/root/bin

description="`hostname`-rootdrive-`date +%Y-%m-%d-%H-%M-%S`"

ec2-consistent-snapshot --aws-access-key-id=YOUR_ACCESS_KEY --aws-secret-access-key=YOUR_SECRET_ACCESS_KEY \
--description=$description --region=eu-west-1 YOUR_VOLUME_ID
------------------------------------
chmod 700 snapshot_ebs_root.sh


# Create /vol EBS snapshot script
vi ./snapshot_ebs_vol.sh
------------------------------------
#!/bin/sh

PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/home/ec2/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/ho
me/ec2/bin:/root/bin

description="`hostname`-rootdrive-`date +%Y-%m-%d-%H-%M-%S`"

ec2-consistent-snapshot --aws-access-key-id=YOUR_ACCESS_KEY --aws-secret-access-key=YOUR_SECRET_ACCESS_KEY \
--mysql-username=root --mysql-password=YOUR_DB_ROOT_PASSWORD --mysql-host=localhost --description=$description --mysql --region=eu-west-1 \
--freeze-filesystem='/vol' YOUR_VOLUME_ID
------------------------------------
chmod 700 snapshot_ebs_vol.sh


#Set the cronschedule for EBS snapshots
crontab -e
------------------------------------
54 6-23 * * * /root/snapshot_ebs_vol.sh > /dev/null 2>&1
59 22 * * * /root/snapshot_ebs_root.sh > /dev/null 2>&1
------------------------------------

# Change httpd DocumentRoot to /vol/www/default
vi /etc/httpd/conf/httpd.conf
mkdir /vol/www/default
mkdir /vol/www/sites
chown -R apache:apache /vol/www
chkconfig httpd on
service httpd start

#Set server timezone
cd /etc
ln -sf /usr/share/zoneinfo/CET localtime