it's a code thing

Wednesday, January 01, 2014

WebRTC app - C# / Xamarin - (C# - JNI - C/C++) - Summary and GitHub repository

This posting concludes a series of in all 4 blog posting where this being the 4th.

WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library

WebRTC app - C# / Xamarin - Part #2 - Attempt #1 - failure to using a JNI .so file directly from C# / Mono

WebRTC app - C# / Xamarin - Part #2 - Attempt #2 - success using a JNI .so file from C# / Mono

WebRTC app - C# / Xamarin - (C# - JNI - C/C++) - Summary and GitHub repository

And finally the associated GitHub repository https://github.com/kenneththorman/webrtc-app-mono

WebRTC app - C# / Xamarin - Part #2 - Attempt #2 - success using a JNI .so file from C# / Mono

This is a post in a series of postings

WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library

WebRTC app - C# / Xamarin - Part #2 - Attempt #1 - failure to using a JNI .so file directly from C# / Mono

WebRTC app - C# / Xamarin - Part #2 - Attempt #2 - success using a JNI .so file from C# / Mono

WebRTC app - C# / Xamarin - (C# - JNI - C/C++) - Summary and GitHub repository

And finally the associated GitHub repository https://github.com/kenneththorman/webrtc-app-mono

In my previous posting WebRTC app - C# / Xamarin - Part #2 - Attempt #1 - failure to using a JNI .so file directly from C# / Mono I wrote

This is pushing me in a direction that I initially hoped I could avoid (mainly due to my limited knowledge in the area), JNI.

There were a few reasons that I did not prefer to use a Java Binding Library in the solution.

I would have an even more mixed source code base (C# calling jar/Java which is wrapping C/C++), I would have preferred to keep it at C# wrapping C/C++.
The upstream build process is packaging some of the compiled java classes that I need into jar files, but not all of them so now I manually need to add a build step to the build process.
I am not familiar with JNI

If we look at the files that are generated during the build process explained in this posting WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library we will see that the jars that we need to build the test android app supplied with the project are located in

~/WebRTCDemo/trunk/webrtc/video_engine/test/android/libs/

namely

audio_device_module_java.jar
video_capture_module_java.jar
video_render_module_java.jar

For each of these jars we need to create a Java Bindings Library that is using the relevant jar as an input jar (actually 2 of the jars can go into the same Java Binding Library - video_capture_module_java.jar and video_render_module_java.jar since they share the same Java package/namespace).

(It might be possible to add them all in one Java Binding Library but I encountered problems due to Visual Studios project setting of Default Namespace, which basically was affecting the namespace of the Java classes that I needed to invoke from C#. Since each of the jars contained classes from a different Java package, the easy work around was to add a Java Binding Library for each of the jars and make sure that the Visual Studio Default namespace matched the Java package name.)

Looking at the existing Java WebRTC demo app available in the folder

~/WebRTCDemo/trunk/webrtc/video_engine/test/android/src/org/webrtc/videoengineapp

you will find

IViEAndroidCallback.java
ViEAndroidJavaAPI.java
WebRTCDemo.java

The main file is called WebRTCDemo.java and contains code like the following

...
import org.webrtc.videoengine.ViERenderer;

import java.io.File;
import java.io.IOException;
import java.net.InetAddress;
import java.net.NetworkInterface;
import java.net.SocketException;
import java.util.Enumeration;

public class WebRTCDemo extends TabActivity implements IViEAndroidCallback,
                                                       View.OnClickListener,
                                                       OnItemSelectedListener {
    private ViEAndroidJavaAPI vieAndroidAPI = null;

    // remote renderer
    private SurfaceView remoteSurfaceView = null;

    // local renderer and camera
    private SurfaceView svLocal = null;

    // channel number
    private int channel = -1;
    private int cameraId;
    private int voiceChannel = -1;
...

By looking at the code I could see there would be some problems, since the main activity is referencing 2 classes that are defined and located side by side with the main activity IViEAndroidCallback.java and ViEAndroidJavaAPI.java. In other words these are not available in a jar and especially the ViEAndroidJavaAPI.java ois the wrapper for the JNI native library so we cannot do this directly from C# according to Native library integration (posting #2).

In the beginning of this posting I outlined 3 reasons I preferred to avoid using JNI and Java Binding Libraries, one of them was

The upstream build process is packaging some of the compiled java classes that I need into jar files, but not all of them so now I manually need to add a build step to the build process.

Anyhow this was fairly easy to remedy by manually creating a new jar containing the 2 files that was needed and then adding this new jar to a new Java Binding Library.

XXX@ubuntu:~/WebRTCDemo/trunk/webrtc/video_engine/test/android/bin/classes$
jar cvf ViEAndroidJavaAPI.jar
org/webrtc/videoengineapp/IViEAndroidCallback.class
org/webrtc/videoengineapp/ViEAndroidJavaAPI.class
added manifest
adding: org/webrtc/videoengineapp/IViEAndroidCallback.class(in = 218)
(out= 173)(deflated 20%)
adding: org/webrtc/videoengineapp/ViEAndroidJavaAPI.class(in = 2845)
(out= 1303)(deflated 54%)

Now I was able to use the native library without any exceptions occurring.

WebRTC app - C# / Xamarin - Part #2 - Attempt #1 - failure to using a JNI .so file directly from C# / Mono

This is a post in a series of postings

WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library

WebRTC app - C# / Xamarin - Part #2 - Attempt #1 - failure to using a JNI .so file directly from C# / Mono

WebRTC app - C# / Xamarin - Part #2 - Attempt #2 - success using a JNI .so file from C# / Mono

WebRTC app - C# / Xamarin - (C# - JNI - C/C++) - Summary and GitHub repository

And finally the associated GitHub repository https://github.com/kenneththorman/webrtc-app-mono

In my previous posting WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library I have showed how to build a native library that we need to use to build a WebRTC app on Xamarin / Mono.Droid.

In this posting I will take you through my struggles, subsequent failure and the next posting finally success on how to actually use this JNI native library from Mono.Android.

I started a new solution in Visual Studio 2013 and added new Android Application project. Then according to Xamarin: Using Native Libraries.
I needed to add my .so file to the location
<project>\lib\armeabi-v7a\libwebrtc-video-demo-jni.so.

The next step that I tried was to start using DllImport statements.

using System;
using System.Runtime.InteropServices;
using Android.Content;
using Android.Util;
using Encoding = System.Text.Encoding;

namespace WebRtc
{
        public class ViEAndroidJavaAPI
        {

...
                // API Native

                [DllImport("libwebrtc-video-demo-jni.so")]
                private static extern bool NativeInit(Context context);

                // Video Engine API
                // Initialization and Termination functions
                [DllImport("libwebrtc-video-demo-jni.so")]
                public static extern int GetVideoEngine();

                [DllImport("libwebrtc-video-demo-jni.so")]
                public static extern int Init(bool enableTrace);

                [DllImport("libwebrtc-video-demo-jni.so")]
                public static extern int Terminate();
...

Trying to run my project yielded some EntryPointNotFoundException in the error log. After a bit of Google-ing I found that the method names as seen from Mono are not as you expect instead they contain the full package/class path.

Using the following command on the Ubuntu build machine

~/WebRTCDemo/trunk/webrtc/video_engine/test/android/libs/armeabi-v7a$ arm-linux-androideabi-nm -D libwebrtc-video-demo-jni.so

yielded the following output

00015358 T JNI_OnLoad
00015be4 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_AddRemoteRenderer
00016e3c T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_CreateChannel
00015f14 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_EnableNACK
00015f58 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_EnablePLI
00015e04 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetCameraOrientation
00015acc T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetCodecs
000153d4 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetVideoEngine
000154cc T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_Init
000153d0 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit
00015c30 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_RemoveRemoteRenderer
00015fb0 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_SetCallback
00015ea8 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_SetExternalMediaCodecDecoderRenderer
...

So changing my code to the following made the EntryPointNotFoundException go away

using System;
using System.Runtime.InteropServices;
using Android.Content;
using Android.Util;
using Encoding = System.Text.Encoding;

namespace WebRtc
{
        public class ViEAndroidJavaAPI
        {

...
                // API Native

                [DllImport("libwebrtc-video-demo-jni.so")]
                private static extern bool Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit(Context context);

                // Video Engine API
                // Initialization and Termination functions
                [DllImport("libwebrtc-video-demo-jni.so")]
                public static extern int Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetVideoEngine();

                [DllImport("libwebrtc-video-demo-jni.so")]
                public static extern int Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_Init(bool enableTrace);

                [DllImport("libwebrtc-video-demo-jni.so")]
                public static extern int Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_Terminate();

...

Now I was faced with another exception which seemed much nastier.
UNHANDLED EXCEPTION: System.Runtime.InteropServices.MarshalDirectiveException: Type Java.Lang.Object which is passed to unmanaged code must have a StructLayout attribute.

12-21 19:29:04.298 I/MonoDroid(15226): UNHANDLED EXCEPTION: System.Runtime.InteropServices.MarshalDirectiveException: Type Java.Lang.Object which is passed to unmanaged code must have a StructLayout attribute.
12-21 19:29:04.298 I/MonoDroid(15226): at (wrapper managed-to-native) WebRtc.ViEAndroidJavaAPI.Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit (Android.Content.Context) 
12-21 19:29:04.298 I/MonoDroid(15226): at WebRtc.ViEAndroidJavaAPI..ctor (Android.Content.Context) [0x00033] in XXX\WebRtc.Mono.Droid\ViEAndroidJavaAPI.cs:30
12-21 19:29:04.298 I/MonoDroid(15226): at WebRtc.Mono.Droid.WebRTCDemo.startMain () [0x0004b] in XXX\WebRtc.Mono.Droid\WebRTCDemo.cs:533
12-21 19:29:04.298 I/MonoDroid(15226): at WebRtc.Mono.Droid.WebRTCDemo.OnCreate (Android.OS.Bundle) [0x0028e] in XXX\WebRtc.Mono.Droid\WebRTCDemo.cs:313
12-21 19:29:04.298 I/MonoDroid(15226): at Android.App.Activity.n_OnCreate_Landroid_os_Bundle_ (intptr,intptr,intptr) [0x00011] in /Users/builder/data/lanes/monodroid-mlion-monodroid-4.10.1-branch/d23a19bf/source/monodroid/src/Mono.Android/platforms/android-17/src/generated/Android.App.Activity.cs:2119
12-21 19:29:04.298 I/MonoDroid(15226): at (wrapper dynamic-method) object.705dc6ba-9c58-4bcd-a8a2-f12584a9175f (intptr,intptr,intptr)

Finally after digging I found this posting Native library integration which basically state

you cannot sanely use P/Invoke to invoke the native method. You must instead use JNI to invoke the Java-side native method.

Basically because this is Java native C/C++ interface we are to invoke, you cannot do this like normal non JNI wrapped C/C++ methods.

This is pushing me in a direction that I initially hoped I could avoid (mainly due to my limited knowledge in the area), JNI.

Later: I did some (quite a bit) reading, namely I found these links useful:

Interop with Native Libraries
Java Integration Overview
Working With JNI

In the next posting in this series I manage to invoke the native methods.

WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library

This is a post in a series of postings

WebRTC app - C# / Xamarin - Part #1 - Building platform native webrtc library

WebRTC app - C# / Xamarin - Part #2 - Attempt #1 - failure to using a JNI .so file directly from C# / Mono

WebRTC app - C# / Xamarin - Part #2 - Attempt #2 - success using a JNI .so file from C# / Mono

WebRTC app - C# / Xamarin - (C# - JNI - C/C++) - Summary and GitHub repository

And finally the associated GitHub repository https://github.com/kenneththorman/webrtc-app-mono

I wish to build a cross platform app that supports WebRTC (real time communication - wikipedia article here). I would like to use Xamarin to achieve some level of code reuse between iOS and Android. There are several areas of the application that can use common code like the:

webrtc signaling
UI (ViewModel) logic if we utilize Stuart Lodge's MVVMCross.

This series of blog postings will document my attempt at implementing the Android initial app (Mono.Android) and then I will attempt to move this to iOS (monotouch). Unlike some of my other postings this is a documentation project during the attempt to reach that goal.

Lets get started.
Now building webrtc and all the associated libraries is not for the faint of heart, but luckily there are some pretty nice build tools available that does the job nicely. For both the Android and later the iOS edition we will need a C/C++ native compiled library that does all the heavy lifting with regards to audio/video rendering, decoding and encoding. So my first goal was to build a Android compatible library that I could include in my solution.

Here are some good links that got me started
http://www.webrtc.org/reference/getting-started

and then I found this little gem at Ryazantsev's blog which basically walks you through the process step by step. I already had a Ubuntu 12.04 LTS virtual machine installed and configured so the below is the console commands run on my machine. I use some slightly modified commands compared to Ryazantsev's blog. Thanks to Ryazantsev for posting this great post.

##Installing JAVA (first install default JAVA)
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update && sudo apt-get install oracle-jdk7-installer
chmod a+x jdk-6u45-linux-x64.bin
./jdk-6u45-linux-x64.bin
mkdir /usr/lib/jvm
mv jdk1.6.0_45 /usr/lib/jvm/jdk1.6.0_45

##Update alternatives of java tools
jdir=jdk1.6.0_45
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/$jdir/bin/javac 1
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/$jdir/bin/java 1
sudo update-alternatives --install /usr/bin/javaws javaws /usr/lib/jvm/$jdir/bin/javaws 1
sudo update-alternatives --install /usr/bin/jar jar /usr/lib/jvm/$jdir/bin/jar 1

##Check alternatives and java version
sudo update-alternatives --config javac
sudo update-alternatives --config java
sudo update-alternatives --config javaws
sudo update-alternatives --config jar
ls -la /etc/alternatives/{java,javac,javaws,jar}
java -version

echo 'export JAVA_HOME=/usr/lib/jvm/jdk1.6.0_45' >> ~/.bashrc
echo 'PATH="$PATH":`pwd`/depot_tools' >> ~/.bashrc
source ~/.bashrc
printenv | grep depot_tools
 
sudo apt-get install git subversion libpulse-dev g++ pkg-config gtk+-2.0 libnss3-dev
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

###Building WebRTC Trunk
cd ~
mkdir WebRTCDemo
cd WebRTCDemo/

gclient config https://webrtc.googlecode.com/svn/trunk
gclient sync
echo target_os = [\'android\', \'unix\'] >> .gclient
./trunk/build/install-build-deps.sh  --no-chromeos-fonts
./trunk/build/install-build-deps-android.sh
check  'sudo update-alternatives --config java ' for correct path
gclient sync
cd trunk
source ./build/android/envsetup.sh
gclient runhooks
GYP_GENERATORS=ninja ./build/gyp_chromium --depth=. all.gyp 
ninja -C out/Debug -j10 All

After downloading all the files and building the project it takes up about 4.6GB on my disk in the virtual machine. Most source files in the project will show to be a good reference when we need to start using this in a Xamarin project. The really relevant parts through is the java sources for the Java version of the app (written by the WebRTC authors) as well as the Android compatible .so file.

The compiled .so file we need is available at

~/WebRTCDemo/trunk/webrtc/video_engine/test/android/libs/armeabi-v7a/libwebrtc-video-demo-jni.so

The next posting will be attempt #1 in converting a Java app to a c# equivalent

Sunday, December 29, 2013

Yii Framework: Using multiple db connections in a controller method

I have a mixed database where the majority of the tables are UTF-8 encoding but there are a few tables that are required to store the data in a specific encoding that this not UTF-8.

In Yii you are defining you global database connection in the protected/config/main.php file, and this is global for the entire web application.This is causing some challenges for me since it is very important that I deliver data bitwise compatible according to the data stored in the table.

I was not able to find any solution to this in my first few Google attempts, so I had to dive into the code.

First I tried in the controller action method by changing the charset on the global CDbConnection directly using

Yii::app()->db->charset='latin1';

this did not work since the database connection is already initialized at this point.

Going through the Yii Framework source code I first found CApplication->getDB(), then searching for the function getComponent() lead me to CModule->getComponent().

My next though was "If there is a getComponent(), I wonder if there is not a setComponent() function as well?" - and behold CModule->setComponent().

Armed with this knowledge it was easy to change the connection temporarily to use a new initialized database connection with a different charset.

public function loadModel($id)
{
    // Save the original database connection for later reuse
    $originalDbConnection = Yii::app()->db;
    
    // Create a new database connection based on the original database connection
    // change the charset to latin1
    $latin1DbConnection = Yii::createComponent(array(      
        'class'=>'CDbConnection',
        'connectionString' => Yii::app()->db->connectionString,
        'emulatePrepare' => Yii::app()->db->emulatePrepare,
        'username' => Yii::app()->db->username,
        'password' => Yii::app()->db->password,
        'charset' => 'latin1',
    ));

    // Set the application wide database connection for this Apache/PHP webrequest to use this special database connection
    Yii::app()->setComponent("db",$latin1DbConnection);
    
    // Do the queries against the database
    $someModel=SomeModel::model();
    $model = $someModel->findByPk($id);
    
    // Restore the original database connection 
    // note: 
    // This is actually not needed in this simple case, due to the fact that Apache/PHP rebuilds the entire web application on each request
    // Since I am not doing additional request in this logic here after returning the result everything is torn down and discarded.
    // However I have added it as part of this code for completeness and also in case someone else out there is doing more complex logic with multiple 
    // request that needs different charsets
    Yii::app()->setComponent("db",$originalDbConnection);

    if($model===null)
        throw new CHttpException(404,'The requested page does not exist.');
    return $model;
}

Sunday, September 01, 2013

Unable to change network adapter UUID on Linux

I needed to reconfigure the hardware UUID of a Linux network adapter caused by cloning a virtual machine. To do this you need to either remove the /etc/udev/rules.d/70-persistent-net.rules and let the system regenerate the file on reboot, or manually edit the file.

I decided to edit the file and manually correct the UUID. I was consistently met "readonly file" so I could not save my changes.

I found using the lsattr /etc/udev/rules.d/70-persistent-net.rules that the file had the i attribute set. This i attribute means that the file is immutable. Removing this attribute using chattr -i /etc/udev/rules.d/70-persistent-net.rules, allowed me to change the file as needed. After editing the file I set the i attribute again using chattr +i /etc/udev/rules.d/70-persistent-net.rules.

Sunday, July 28, 2013

Fixing the client IP as perceived by Apache behind Nginx reverse proxy

In my recent post Using Nginx to reverse proxy a secure apache site that is using socket.io/node.js/websockets I found that I was only getting the proxy's IP address in all of my apache logs as well as in the application tracking. This does make sense since from Apache's perspective the proxy is the actual client and not the real remote client. I found however that it was possible to get around this. Apparently there are several Apache modules that can do this, however I did not want to build from sources if I could avoid it. So I choose the one that was already available in the EPEL repository.

I already had the EPEL repository registered on the server but for those that does not you can use the following commands to register it.

rpm -Uvh http://ftp.crc.dk/fedora-epel/6/i386/epel-release-6-8.noarch.rpm

Now install the module

yum install mod_extract_forwarded

After installing the module you need to register an allowed forwarder (ie. the IP of your proxy). In all my logs the ip 127.0.0.1 was registered as the client of every request so adding that did the trick.

echo "MEFaccept 127.0.0.1" >> /etc/httpd/conf.d/mod_extract_forwarded.conf
service httpd restart

Recheck your webserver/application logs, this worked for me.

Thursday, July 25, 2013

Using Nginx to reverse proxy a secure apache site that is using socket.io/node.js/websockets

Challenge:
Firewalls can cause challenges, by blocking the ports that you want to use for websockets.

Many firewalls in use today are so called stateful firewalls and in short their function can be described as follows

Stateful firewalls only verify that a packet correlates to an existing, unclosed, connection. It tracks the state of the connection (opening, open, closing, closed) hence the name. When it detects a packet is part of an already open, authorized connection, it can short circuit all of the other rule checks and let the packet through.
(http://stackoverflow.com/questions/1967943/will-html5-websockets-be-crippled-by-firewalls)

Most sites that are using node.js for websockets are also utilizing the library socket.io.

A good list outlining some of the challenges when combining websockets/socket.io with various firewalls can be found at Socket.IO and firewall software.

So the question is how do you get your website running in spite of corporate firewall blocking various ports?

Possible solutions:

ask the IT department to open the needed ports (in my experience hardly likely unless #2 fails)

piggy-back on one of the existing established connections using http (port 80) or https (port 443) and hope the firewall is not a deep packet inspection based firewall.

Since I seldom have luck with suggestion number 1, I am not able to offer any guidance that is likely to help you succeed.

Suggestion number 2 however we can achieve that by letting the websockets use port 80/443 and let the website use the same port. You cannot however run 2 different services on the same port, so you cannot do this in a straightforward manner. The answer is a websocket aware reverse proxy.

There are quite a few different setups that can solve this. Below you can find 3 ways I investigated when I needed to solve this:

Apache acting as a reverse proxy for node.js (this was my first choice, but I did not manage to get this working with out-of-the-box available packages for the Linux distribution I am using, but you can more than likely build your own version that will work just fine Backporting Apache support for websockets reverse proxy (aka getting GateOne to work behind Apache) and Using apache-websocket to proxy tcp connections).
Use node.js to proxy Apache (see more here https://github.com/nodejitsu/node-http-proxy). (I never managed to get this working properly either but that is more than likely due to my slightly limited exposure to node.js, and to be honest my experience with node has been a slightly turbulent one, so I feel that Apache/Nginx are the more stable options for me, that is not to say that this is not working or an great option.
Nginx being a reverse proxy for both apache and websockets/socket.io. This is what I will describe below.

Setup

CentOS based Linux distribution
existing website installed as a Apache virtual host running on port 80 and 443
existing node.js socket.io based application running on port 8081

(setup before reverse proxy)

Please find an image of the wished for setup here below.

We need to do the following to achieve the reverse proxy setup

Install Nginx

Remove http -> https rewrite and ssl configuration from Apache as well as change Apache from listening on port 80/443 to listen on port 81

Configure http -> https rewrite and reverse proxy in Nginx

Change the socket.io node.js application to just use ws instead of wss (since Nginx is the SSL termination proxy now)

Change the client application from using port 8081 to using 80/443

1. Install Nginx
RHEL / Centos 6: Install Nginx Using Yum Command

2. Remove http -> https rewrite and ssl configuration from Apache as well as change Apache from listening on port 80/443 to listen on port 81

Open the virtual hosts file for the site
vi /etc/httpd/conf.d/www_somewhere_com.conf
Change your virtual hosts file from the following

# Catch those that are trying to access the site using http
# redirect to https
<VirtualHost *:80>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^(.*) https://www.somewhere.com [R,L]
</VirtualHost>

# Main site
<VirtualHost *:443>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common

  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/www_somewhere.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/www_somewhere.com.key
  SSLCertificateChainFile /etc/pki/tls/certs/Intermediate_Certificate.crt

  <Directory /var/www/sites/www.somewhere.com/www>
    Order Deny,Allow
    Allow from all
  </Directory>
</VirtualHost>

to the following

# Main site
<VirtualHost *:81>
  ServerName "www.somewhere.com"
  DocumentRoot /var/www/sites/www.somewhere.com/www
  DirectoryIndex index.html index.php
  ErrorLog logs/www.somewhere.com-error_log
  CustomLog logs/www.somewhere.com-access_log common

  <Directory /var/www/sites/www.somewhere.com/www>
    Order Deny,Allow
    Allow from all
  </Directory>
</VirtualHost>

vi /etc/httpd/conf/httpd.conf
Look for the "Listen 80" in the file and change it to "Listen 81"
Save the file

3. Configure http -> https rewrite and reverse proxy in Nginx
vi /etc/nginx/conf.d/default.conf

replace the deafult content with this

Please note that in apache config you have a seperate intermidiate certificate and the site certificate. In Nginx you need to concatenate the primary certificate file (your_domain_name.crt) and the intermediate certificate file (DigiCertCA.crt) into a single pem file by running the following command:

cat DigiCertCA.crt >> your_domain_name.crt

# Listeners
# nginx         :80         responsible for redirecting http://www.somewhere.com to https://www.somewhere.com 
# nginx         :443        responsible for handling all trafic on the https://www.somewhere.com site
#                           acts as a reverse proxy for 
#                           /socket.io/     -> port 127.0.0.1:8081  (node.js websocket server)
#                           /               -> port 127.0.0.1:81    (apache  main site)
#
# node.js   :8081   node.js websocket server
# apache    :81     apache  main site


upstream www.somewhere.com {
    server 127.0.0.1:81;
    least_conn;
}

# nginx         :80         responsible for redirecting http://www.somewhere.com to https://www.somewhere.com 
server {
    listen 80;
    server_name  www.somewhere.com;
    rewrite ^(.*) https://www.somewhere.com$1 permanent;
}

# nginx         :443        responsible for handling all trafic on the https://www.somewhere.com site
#                           acts as a reverse proxy for 
#                           /socket.io/     -> port 127.0.0.1:8081  (node.js  websocket server)
#                           /               -> port 127.0.0.1:81    (apache  main site)
server {
    listen 443 ssl;
    server_name www.somewhere.com;

    ssl_certificate     /etc/pki/tls/certs/www_somewhere.com.crt;
    ssl_certificate_key /etc/pki/tls/private/www_somewhere.com.key;
    ssl_protocols       SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         RC4:HIGH:!aNULL:!MD5;

    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

    # Path based websocket proxy location for nginx
    # to handle the  reverseproxying
    #
    # inital connect url: https://www.somewhere.com:8081/socket.io/1/?t=1374751734771
    # websocket url     :   wss://www.somewhere.com:8081/socket.io/1/websocket/x55Ch0B-SCACmDpW8rZd   
    location /socket.io/ {
        proxy_pass http://localhost:8081;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }   

    # Path based site proxy location for nginx
    # to handle the main apache  site
    location / {
        proxy_pass http://localhost:81;
    }   
}

4. Change the socket.io node.js application to just use ws instead of wss (since Nginx is the SSL termination proxy now)
Change you node.js socket.io application from this

var fs = require('fs');
var sslCertificate = {
    key: fs.readFileSync(config.sslCertificate.key),
    cert: fs.readFileSync(config.sslCertificate.cert),
    ca: fs.readFileSync(config.sslCertificate.ca)
};

// socket.io wss (websocket secure)
var io = require('socket.io').listen(8081, sslCertificate);
...

to the following

var fs = require('fs');
var sslCertificate = {
//    key: fs.readFileSync(config.sslCertificate.key),
//    cert: fs.readFileSync(config.sslCertificate.cert),
//    ca: fs.readFileSync(config.sslCertificate.ca)
};

// socket.io ws (websocket)
var io = require('socket.io').listen(8081, sslCertificate);
...

5. Change the client application from using port 8081 to using 80/443

Find the location in your client code where you are connecting to the websocket/socket.io application/server and change it from hardcoding the port in the url to just omitting it

So instead of (in a angular.js application / javascript single page application) doing this

var socketIO = function(data){
    // force https / wss
    var hostURL = window.location.href.replace('/'+window.location.hash, '8081').replace('http:', 'https:');

    ...
    
    try {
        var socket = io.connect(hostURL, {'sync disconnect on unload':true});
        ...
    }
    catch(e){
        console.log('Error connecting');
        ...
    }
    ...
}

do something like this

var socketIO = function(data){
    // force https / wss
    var hostURL = window.location.href.replace('/'+window.location.hash, '').replace('http:', 'https:');

    ...
    
    try {
        var socket = io.connect(hostURL, {'sync disconnect on unload':true});
        ...
    }
    catch(e){
        console.log('Error connecting');
        ...
    }
    ...
}

run the following command to check if the services are running on the correct ports

#netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      13146/nginx
tcp        0      0 0.0.0.0:8081                0.0.0.0:*                   LISTEN      13225/node
tcp        0      0 127.0.0.1:81                0.0.0.0:*                   LISTEN      13190/httpd
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      13146/nginx
...

Finally some before and after pictures from chrome
Before

After

Monday, July 01, 2013

Compressed Node.js install on CentOS 6.4

yum install -y make gcc cc gcc-c++ wget; cd /usr/src; wget http://nodejs.org/dist/v0.10.12/node-v0.10.12.tar.gz; tar zxf node-*.tar.gz; cd node-v*; ./configure; make; make install;

Wednesday, June 19, 2013

Installing ejabberd2 on CentOS and configure a virtual host on Apache2

1. Enable epel yum repository

Centos 5.x

wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
sudo rpm -Uvh epel-release-5*.rpm

Centos 6.x

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -Uvh epel-release-6*.rpm

2. Install ejabberd2

yum --enablerepo=epel install ejabberd

3. Start the ejabberd2 service

service ejabberd start

4. Edit the virtual hosts configuration file and add

  ProxyPass /http-bind  http://127.0.0.1:5280/http-bind/
  ProxyPassReverse  /http-bind   http://127.0.0.1:5280/http-bind/
  Header set Access-Control-Allow-Origin "YOUR_VIRTUAL_SERVER_HOSTNAME"

(please note that using "Header set Access-Control-Allow-Origin" enables Cross-origin resource sharing and using a wildcard as origin should not normally be used)

This is generally not appropriate. The only case where this is appropriate is when a page or api response is considered completely public content and it is intended to be accessible to browsable to everyone. Including any code on any site.
(http://en.wikipedia.org/wiki/Cross-origin_resource_sharing)

5. Test configuration

service httpd configtest

6. Reload apache configuration

service httpd reload

Since ejabberd comes with http-bind enabled out of the box

{5280, ejabberd_http, [
 %%{request_handlers,
 %% [
 %%  {["pub", "archive"], mod_http_fileserver}
 %% ]},
 captcha,
 http_bind,
 http_poll,
 %%register,
 web_admin
]}

you should now be able to access the XMPP over BOSH (XEP-0206) server in your browser looking somewhat like below.

Monday, June 03, 2013

Xamarin Mono.Droid Receiving different bluetooth devices data records

When you are working with several different Bluetooth devices (from different manufacturers) you will have to be able to handle different types of data records. The official Android Bluetooth sample ported for Xamarin/Mono.Droid can be found at BluetoothChatService.cs. This was the code I used for inspiration when I built this app.
I could however not really get the code to work as I wanted and the problem was located in the following method Run().

public override void Run ()
{
 Log.Info (TAG, "BEGIN mConnectedThread");
 byte[] buffer = new byte[1024];
 int bytes;

 // Keep listening to the InputStream while connected
 while (true) {
  try {
   // Read from the InputStream
   bytes = mmInStream.Read (buffer, 0, buffer.Length);

   // Send the obtained bytes to the UI Activity
   _handler.ObtainMessage (BluetoothChat.MESSAGE_READ, bytes, -1, buffer)
    .SendToTarget ();
  } catch (Java.IO.IOException e) {
   Log.Error (TAG, "disconnected", e);
   _service.ConnectionLost ();
   break;
  }
 }
}

The problem is the following call

// Send the obtained bytes to the UI Activity
_handler.ObtainMessage (BluetoothChat.MESSAGE_READ, bytes, -1, buffer).SendToTarget ();

which might not always send back the entire device record to the handler. This is caused by the various devices implementation of how to send the data record back as a byte stream to the mobile device/phone might not finish, before the while loop is starting another iteration, as well as the multithreading and the context switches that occurs in the BluetoothChatService.

What I discovered while testing was that for some devices it was working out of the box with the default code in the Run method above. However on some more complex devices which were sending larger data records back, it never seemed to work.

After spending some time analyzing the problem - being new to the bluetooth scene and Mono.Droid development - I found that data records got chopped into pieces in the while(true) loop in the above Run() method. Assuming that a full data record was 65 bytes, sometimes the code would pass 13 bytes to the SendToTarget() method. The next iteration might yield 31 bytes and the last 21 bytes was yielded on the 3rd pass.

The number of bytes returned on each iteration was never the same and there was no specific pattern to this. To be able to handle this for different devices sending data in different ways I needed some kind of smart buffering. I came up with the following.

public override void Run ()
{
 Log.Info (TAG, "BEGIN mConnectedThread");
 byte[] buffer = new byte[1024];
 int bytes;
 
 // Keep listening to the InputStream while connected
 while (true) {
  try {
   // Read from the InputStream
   bytes = mmInStream.Read(buffer, 0, buffer.Length);

   var messagePart = new byte[bytes];
   Array.Copy(buffer,messagePart, bytes);

   // Send the obtained bytes to the UI Activity
   _handler.ObtainMessage(MESSAGE_READ, bytes, -1, messagePart)
    .SendToTarget();
  } catch (Java.IO.IOException e) {
   Log.Error (TAG, "disconnected", e);
   _service.ConnectionLost ();
   break;
  }
 }
}

The new code is only a little bit different in that I am no longer sending back the buffer but instead only the partial message (messagePart).

In the handler that handles the read bytes I changed the implementation from what can be found here BluetoothChat.cs.

case MESSAGE_READ:
 byte[] readBuf = (byte[])msg.Obj;
 // construct a string from the valid bytes in the buffer
 var readMessage = new Java.Lang.String (readBuf, 0, msg.Arg1);
 bluetoothChat.conversationArrayAdapter.Add (bluetoothChat.connectedDeviceName + ":  " + readMessage);
 break;

to the following code

case BluetoothService.MESSAGE_READ:
 byte[] readBuf = (byte[])msg.Obj;
 ParseBluetoothDeviceReading(readBuf);
 break;

and the ParseBluetoothDeviceReading() method looking like this

public void ParseBluetoothDeviceReading (byte[] readBuffer)
{
 if (readBuffer == null || readBuffer.Length <= 0)
  return;

 var deviceRecord = _zephyrResponseParser.ParseDeviceResponse(readBuffer);
 _data.LungFunction= deviceRecord.LungFunction;
 lblValue.Text = deviceRecord.LungFunction.ToString("0.00");
}

the data record buffer is implemented in the _zephyrResponseParser which is of type IDeviceResponseParser. The interface IDeviceResponseParser looks like the following

public interface IDeviceResponseParser<TDataRecord>
{
 TDataRecord ParseDeviceResponse(byte[] response);
}

and finally the actual device specific data record parser for the zephyr device.

using System.Collections.Generic;

namespace Droid.Devices.Zephyr
{
 public class ZephyrResponseParser : IDeviceResponseParser<ZephyrRecord>
 {
  public const int DeviceMessageLength = 65;
  
  private List<byte> _messageBuffer = new List<byte>();

  public ZephyrRecord ParseDeviceResponse(byte[] response)
  {
   // Since the data (byte[]) that we are getting is retreived from a BluetoothNetworkStream there are no guarantees that we will get the full message in one go.
   // For that reason we need to buffer it our selves until whe have a full message
   _messageBuffer.AddRange(response);
   if (_messageBuffer.Count < DeviceMessageLength)
   {
    // Not full message return
    return new ZephyrRecord();
   }

   // Full message, pass on to temp so we can reset the local messageBuffer for the next
   // device message
   var temp = _messageBuffer.ToArray();

   // Reset buffer
   _messageBuffer.Clear();
   
   // Now process the full message
   response = temp;
   if (response.Length >= DeviceMessageLength)
   {
    var deviceRecord = ExtractDataFromRawByteStream();
    return new ZephyrRecord { LungFunction = deviceRecord.LungFunction };
   }
   return new ZephyrRecord();
  }
 }

 ...
 
 public class ZephyrRecord
 {
  public float LungFunction { get; set; }
 }
}

This has proven to work so far, without any known problems, for several different devices that are working very differently, has different manufacturers and are measuring different medical parameters.

Sunday, June 02, 2013

Android 4.0.3 and 4.0.4 bluetooth pairing needed on every connection

While developing a cross mobile device app using Xamarin, I ran into a strange problem. Behind the cross device abstraction, that was utilized in both the Android and the iOS version of the app, the Android specific code had a problem. The problem manifested it self on one Samsung Galaxy 10.1 tablet running 4.0.3, as well as one running 4.0.4. The problem was not present on the reference "lowest common denominator tablet" (this was also a Samsung Galaxy 10.1 tablet, just running Android 3.2).

After some Google searches and not really coming up with anything specific, I found this link which was actually showing me the solution, although I did not know it at the time.

Android Programmatically Bluetooth Pairing

Our app is using a somewhat modified version of the official Android Bluetooth sample ported to monodroid, it can be found here BluetoothChat.

The problem showed out to be this line "tmp = device.CreateRfcommSocketToServiceRecord (MY_UUID);"

changing this to

tmp = device.CreateInsecureRfcommSocketToServiceRecord (MY_UUID);

solved the problem for 3 different devices for me.

Some more Googling gave me the apparent reason behind this. There seems to be a bug in Android 4.0.3 and 4.0.4+ that is causing the Bluetooth pairing to be forgotten.

This is documented amongst others on Bluetooth connection on Android ICS not possible.

I spent quite some time trying to figure out what was wrong with the code, being in the mindset "I must be doing something wrong" and being a bit reluctant to start the "blamegame".

I hope this might help someone else out there.

Below are a few links that I went through in my hunt for a solution, they are all one way or another corroborating my findings.

ICS can't pair BT devices without PIN
Bluetooth RFCOMM Server Socket no longer connects properly to embedded device on ICS 4.0.3

Wednesday, May 22, 2013

Runtime performance friendly AngularJs localization / translation using buildscript and Yii Framework + Invalidating the browser cache

Lately I have been working on a new web based single page application. The technologies and frameworks in use are:

AngularJS for creating the user interface.
Twitter Bootstrap as starting point for CSS and to achieve initial responsive design
Yii Framework for a LAMP based REST service layer
SimpleSAMLphp for federated authentication using SAML2.0

One of the design challenged were - how do we approach webpage localization?

The following requirements were given:

It should incur as small run time overhead as possible, preferably none
The localizable strings should be maintained in text files to allow for external translators, without the overhead of having a full translation system installed and configured
The translation system should support pluralization

The solution that were chosen is a deployment/build step implemented in the Yii Framework that both translates and statically renders AngularJS views. The translation engine that is used is also included in the Yii Framework.

The below image shows a module "logbook" and the views needed for some simple CRUD functionality. This image from the filesystem is from the repository (i.e. before the build script has run).

The image below shows how the filesystem looks after the buildscript has run (at the hosting server)

As we can see the build script has added a directory for each supported language with the 2 letter language abbreviation according to ISO 639-1.

Lets have a look at first the english and then the danish version of the actual view when it is presented on the browser screen.

Lets have a look at the html for the angular view.

<div class="span12" id="logbook_edit" ng-controller="logbook.EditController">
    <div id="content">
        <h1><?=LogbookModule::t('Log');?></h1>
        <div ng-include src="'assets/__REVNO__/app/views/'+language+'/logbook/logbook_edit_toolbar.htm'"></div>    
        <form id="frm_logbook_edit" data-id="{{log.id}}" class="form-horizontal">
        <p class="note"><?=Yii::t('sitewide',"msg_field_with_asterisk_are_required");?></p>
        <div class="control-group">
            <label class="control-label required" for="logdate"><?=LogbookModule::t('Date');?></label>
            <div class="controls">
                <input type="text" id="logdate" name="logdate" ng-model="log.logdate" bs-datepicker data-date-format="Yii.user.preferred_date_format.toString().toLowerCase()" />
                <span class="help-inline"></span>
            </div>
        </div>
        <div class="control-group">
            <label class="control-label required" for="title"><?=LogbookModule::t('Title');?></label>
            <div class="controls">
                <input type="text" name="title" id="title" ng-model="log.title">
                <span class="help-inline"></span>
            </div>
        </div>        
        <div class="control-group">
            <label class="control-label required" for="mood"><?=LogbookModule::t('Mood');?></label>
            <div class="controls">
                <div class="btn-group mood-button-group" ng-model="log.mood" data-toggle="buttons-radio" bs-buttons-radio>
                  <button type="button" value="happy" class="btn happy" style="background-image:url(assets/__REVNO__/app/img/happy-icon.png);"></button>
                  <button type="button" value="sad" class="btn sad" style="background-image:url(assets/__REVNO__/app/img/sad-icon.png);"></button>
                  <button type="button" value="angry" class="btn angry" style="background-image:url(assets/__REVNO__/app/img/angry-icon.png);"></button>
                  <button type="button" value="ill" class="btn ill" style="background-image:url(assets/__REVNO__/app/img/ill-icon.png);"></button>
                </div>                
                <input type="hidden" id="mood" name="mood">
                <span class="help-inline"></span>
            </div>
        </div>
        <div class="control-group">
            <label class="control-label required" for="note"><?=LogbookModule::t('Note');?></label>
            <div class="controls">
                <textarea cols="" rows="" id="note" name="note" ng-model="log.note"></textarea>
                <span class="help-inline"></span>
            </div>
        </div>
        <div ng-include src="'assets/__REVNO__/app/views/'+language+'/logbook/logbook_edit_toolbar.htm'"></div>
        </form>
    </div>
</div>

There are a few things that are special in this HTML code

"__REVNO__" placeholder which at buildscript run time is replaced by the actual revision identifier from the source code repository. This allows us to optimize the content headers, for maximum browser caching while at the same time ensuring that the end user gets the latest version when we update to a new version on the server. Controlled cache invalidation/busting
"<?=LogbookModule::t('Note');?>" is the Yii Framework translation method/function t();
the angular scope variable "language" that is part of the logged in users context, this is represented by an 2 letter language abbreviation.

Lets have a look at how English translation string file look (standard Yii Framework messages file)

<?php
return array(
    'Date'=>'Date',
    'Title'=>'Title',
    'Mood'=>'Mood',
    'Note'=>'Note',
    'Log'=>'Log',
    'Edit'=>'Edit',
    'Delete'=>'Delete',
    'Create'=>'Create',
    'List'=>'List',
    'Save'=>'Save',
    'Logbook'=>'Logbook',
    'Created_by'=>'Created by',
);

and then how the translated view ends up looking after running the buildscript

<div class="span12" id="logbook_edit" ng-controller="logbook.EditController">
    <div id="content">
        <h1>Log</h1>
        <div ng-include src="'assets/1932/app/views/'+language+'/logbook/logbook_edit_toolbar.htm'"></div>    
        <form id="frm_logbook_edit" data-id="{{log.id}}" class="form-horizontal">
        <p class="note">Fields with <span class="required">*</span> are required.</p>
        <div class="control-group">
            <label class="control-label required" for="logdate">Date</label>
            <div class="controls">
                <input type="text" id="logdate" name="logdate" ng-model="log.logdate" bs-datepicker data-date-format="Yii.user.preferred_date_format.toString().toLowerCase()" />
                <span class="help-inline"></span>
            </div>
        </div>
        <div class="control-group">
            <label class="control-label required" for="title">Title</label>
            <div class="controls">
                <input type="text" name="title" id="title" ng-model="log.title">
                <span class="help-inline"></span>
            </div>
        </div>        
        <div class="control-group">
            <label class="control-label required" for="mood">Mood</label>
            <div class="controls">
                <div class="btn-group mood-button-group" ng-model="log.mood" data-toggle="buttons-radio" bs-buttons-radio>
                  <button type="button" value="happy" class="btn happy" style="background-image:url(assets/1932/app/img/happy-icon.png);"></button>
                  <button type="button" value="sad" class="btn sad" style="background-image:url(assets/1932/app/img/sad-icon.png);"></button>
                  <button type="button" value="angry" class="btn angry" style="background-image:url(assets/1932/app/img/angry-icon.png);"></button>
                  <button type="button" value="ill" class="btn ill" style="background-image:url(assets/1932/app/img/ill-icon.png);"></button>
                </div>                
                <input type="hidden" id="mood" name="mood">
                <span class="help-inline"></span>
            </div>
        </div>
        <div class="control-group">
            <label class="control-label required" for="note">Note</label>
            <div class="controls">
                <textarea cols="" rows="" id="note" name="note" ng-model="log.note"></textarea>
                <span class="help-inline"></span>
            </div>
        </div>
        <div ng-include src="'assets/1932/app/views/'+language+'/logbook/logbook_edit_toolbar.htm'"></div>
        </form>
    </div>
</div>

Now on to the actual build script which is implemented as a Yii CConsoleCommand

<?php
class BuildReleaseCommand extends CConsoleCommand
{
    var $careSupportedLanguages = array ('EN','DA');
    var $packageRevisionInSvn;
    
    public function actionBuild($siteDirectory, $subversionRepoUrl='https://your_repo_url_here', $svnCheckout=0) 
    {
        $packageRevisionInSvn = $this->getLatestRevisionNumberInSvn($subversionRepoUrl);
        $this->packageRevisionInSvn = $packageRevisionInSvn;
        
        // Delete same revsion folder to replace with newer one
        `rm -rf $siteDirectory/assets/$packageRevisionInSvn`;
        
        // Create php file to store revision number, it will be used in all php file where revision number is required
        $revisionFileContent = '<?php $revision = '.$packageRevisionInSvn.'; ';
        file_put_contents("$siteDirectory/revision.php", $revisionFileContent);

        if($svnCheckout){
            $this->checkoutLatestVersionFromSvn($subversionRepoUrl, $siteDirectory);
        }
        $this->insertRevisionNumberInFolderNames($siteDirectory, $packageRevisionInSvn);
    
        // Delete all directory other then en& da in $siteDirectory/assets/{$this->packageRevisionInSvn}/app/views/
        `rm -rf $siteDirectory/assets/{$this->packageRevisionInSvn}/app/views/*`;
        
        foreach($this->careSupportedLanguages as $key=>$language) {
            Yii::app()->language=strtolower($language);
            $this->translateViews($siteDirectory.'/assets/__REVNO__/app/views', $siteDirectory. '/assets/'.$this->packageRevisionInSvn.'/app/views');
        } 

        $this->insertRevisionNumberInViewUrls($siteDirectory.'/assets/'.$packageRevisionInSvn, $packageRevisionInSvn);
    }

    private function getLatestRevisionNumberInSvn($packageUrl) {
        return trim(`svn info --non-interactive --username YOUR_USERNAME --password YOUR_PASSWORD $packageUrl | grep 'Last Changed Rev' | head -1 | grep -Eo "[0-9]+"`);
    }

    private function checkoutLatestVersionFromSvn($packageUrl, $svnExportDirectory){
        $output = `svn export --force --non-interactive --username YOUR_USERNAME --password YOUR_PASSWORD $packageUrl $svnExportDirectory`;
        $output = `cd $svnExportDir$package`;
    }

 /**
 * Recurse through the filessystem to process all view html files
 * Feature: Translation 
 */
    private function translateViews($srcPath, $destPath){
        $ignoreFiles = array( '.', '..' ); 
        $dh = @opendir( $srcPath ); 
        while( false !== ( $file = readdir( $dh ) ) ){ 
            if( !in_array( $file, $ignoreFiles) ){ 
                if( is_dir( "$srcPath/$file" ) ){ 
                    $this->translateViews( "$srcPath/$file", "$destPath/$file" ); 
                } else { 
                    if (preg_match('/^.*\.(htm)$/i',$file)) {
                        $translatedView = $this->translateView("$srcPath/$file");
                        $this->saveTranslatedView("$destPath/$file", $translatedView);
                    }
                } 
            } 
        } 
        closedir( $dh ); 
    } 

 /**
 * Actual translation of the html view file
 * Feature: Translation 
 */
    private function translateView($fileFullPath){
        $module = $this->getModuleName($fileFullPath);
        $controller = new CController($module,new CWebModule($module,null));
        return $controller->renderInternal($fileFullPath,null,true);
    }

 /**
 * Save the processed view file under the ISO 639-1 2 letter language code directory
 * Feature: Translation 
 */
    private function saveTranslatedView($fileFullPath, $translatedView) {
        $translatedViewFile = str_replace('views','views/'.Yii::app()->language, $fileFullPath);
        $saveDir = dirname($translatedViewFile);
        if (!file_exists($saveDir)){
            mkdir($saveDir,0755, true);
        }
        file_put_contents($translatedViewFile, $translatedView);
    }

    /**
 * Replace the __REVNO__ placeholder in urls in the Angular views so it points to the correct revision number
 * Feature: Cachebusting
 */
 private function insertRevisionNumberInViewUrls($svnExportDirectory, $packageRevisionInSvn) {
        $output = `find $svnExportDirectory \( -iname "*.htm" -or -iname "*.js" -or -iname "*.php" \) -print | xargs sed -i 's/__REVNO__/$packageRevisionInSvn/g'`;
    }

    /**
 * Copy and rename the __REVNO__ folder to the actual revision number we are parsing
 * Feature: Cachebusting
 */
 private function insertRevisionNumberInFolderNames($svnExportDirectory, $packageRevisionInSvn) {
        $output = `cp -Rf $svnExportDirectory/assets/__REVNO__  $svnExportDirectory/assets/$packageRevisionInSvn`;
    }

    public function getModuleName($viewPath)
    {
        $dirs = explode("/", $viewPath);
        $passedViews = false;
        $module = "";
        foreach ($dirs as $key=>$dir)
        {
            if ($passedViews)
            {
                $module = $dir;
                return $module;
            }
            if ($dir == "views") {
                $passedViews = true;
                continue;
            }    
        }   
    }

}
?>

The script is commented to a fair degree, suffice to say that I am using the internal view renderer in the Yii Framework to do the heavy lifting for the translation, it knows where to find the language string files and to do the translation and pluralization.

I am using the build script together with migration for auto-deployment to our test server like this:

cd /YOUR_WEBSITE_DIR; svn update --username YOUR_USERNAME --password YOUR_PASSWORD; chown -R apache:apache /YOUR_WEBSITE_DIR; protected/yiic migrate --interactive=0; protected/yiic buildrelease build --siteDirectory=/YOUR_WEBSITE_DIR/

Wednesday, February 06, 2013

Old HTC Desire phone not booting

I have an old HTC Desire phone that is not able to boot ... again. Last time this happened is about a year ago. Both back then and this time I had to spend the time all over since I could not remember what made it work last time. Thus the rationale for this posting.

What happens is that suddenly the phone just runs out of battery and then it will not boot again.It is stuck on the white HTC screen and you cannot turn it off using the power button. To turn it off you need to pull out the battery.

See more on a similar or the same problem here: Phone stuck on fastboot/hboot, buttons unresponsive

Find a available free SD card stick it into another phone and format the SDCard to FAT32 I used Linux for this since I have had problems doing this on windows for some unknown reason.

On linux you do the following

Plug in your phone USB cable and wait until your OS recognizes the device
Change to root user
type mount and locate the device name of the SD Card and the directory on which it is mounted (typically something like /dev/sdb1 and /media/SDCARD_ID)
type umount /media/SDCARD_ID
now we are going to format the SDCARD (if you have any data on the card you will loose them) type mkdosfs -I -F32 /dev/YOUR_FOUND_DEVICE_NAME_FROM_STEP_3

(please yet again note that the device name that I am refering to /dev/sdb1 might refer to a different drive on your system, when you format this drive all data will be deleted! Make sure you are pointing to the correct drive!)
Now you have a SDcard that is ready
Download the RUU file (officially release ROM from the manufacturer) I am using this one RUU_Bravo_Froyo_HTC_WWE_2.29.405.14_Radio_32.49.00.32U_5.11.05.27_release_224699_signed.exe
This is the official way to install software using a Windows OS (always make sure that you virus scan exe files downloaded from the internet, and also check the MD5 checksum to make sure that it indeed is the file that you expect it to be).
If you phone can boot then you can re-flash on Windows by just running the exe with the phone connected by the USB cable.
My phone however is missing the PB99IMG.zip file and the only way that you can reinstall the phone OS is by.
Start the exe file, and follow the following screenshots

Now do not go further, download and install the excellent tool Process Explorer
Start Process Explorer and find out where the temporary files from the RUU...exe file has been extracted.

Select the exe process that you have just started
Make sure that the lower process details pane is open
Find the directory that contains Disk1. In either this directory or one of the most recent directories in the indicated temp folder look for a file named ROM.zip
When you find it, copy it to the newly formatted SDCard
Rename the file to fit you specific image (it differs from phone model to phone model. Mine is called (please note uppercase letters) PB99IMG.zip
Eject the spare phone
Turn it off and take out the SDCard. Put it into the broken phone
Plug in the broken phone to a charger or to the PC USB slot
Turn on the broken phone while pressing volume down button and then pressing the power button until you get to the HBOOT screen
Just leave the phone and after a while it will show loading image
It will now ask you a number of times, if you want to flash the image and if you want to reboot the phone.
After rebooting you phone you should be good to go