Tuesday 17 December 2013

Unit Testing Sencha Touch (or ExtJS) applications using Karma Test Runner, Jasmine, Maven and Jenkins


I wanted to be able to run the unit tests for my Sencha Touch 2.3 application locally as well as being able to run them in my continous integration process using Jenkins.

I decides to  use "Karma - Spectacular Test Runner for JavaScript" because it nicely integrates with Jasmine and offers headless execution using PhantomJS out of the box. If PhantomJS is not enough for you, you can also specify other Browsers.

Karma runs on NodeJS. I am not going to describe how to set it up. This information can be found on the Karma site.

Run the tests locally

Here is the 'karma.config.js' I use for local testing.

I bootstrap the Sencha Touch app using the 'development.js' microloader. Because the microloader is loading all required files dynamically I need to specify a proxy element in the Karma config pointing to a local webserver hosting the development version of the application JS files as well as the Sencha Touch JS files.


module.exports = function (config) {
    config.set({
        // base path, that will be used to resolve files and exclude
        basePath: '',

        // frameworks to use
        frameworks: [ 'jasmine' ],

        // list of files / patterns to load in the browser
        // Files for Sencha Touch development microloader
        files: [
            'src/main/webapp/.sencha/app/microloader/development.js',
            'src/test/javascript/setUp.js',
            'src/test/javascript/**/*.js' 
        ],

        // list of files to exclude
        exclude: [
        ],

        proxies: {
            '/': 'http://localhost:8080/'
        },

        // test results reporter to use
        // possible values: 'dots', 'progress', 'junit', 'growl', 'coverage'
        reporters: [ 'progress'],

        // web server port
        port: 9876,

        // enable / disable colors in the output (reporters and logs)
        colors: true,

        // level of logging
        // possible values: config.LOG_DISABLE || config.LOG_ERROR ||
        // config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
        logLevel: config.LOG_DEBUG,

        // enable / disable watching file and executing tests whenever any file
        // changes
        autoWatch: true,

        // Start these browsers, currently available:
        // - Chrome
        // - ChromeCanary
        // - Firefox
        // - Opera (has to be installed with `npm install karma-opera-launcher`)
        // - Safari (only Mac; has to be installed with `npm install
        // karma-safari-launcher`)
        // - PhantomJS
        // - IE (only Windows; has to be installed with `npm install
        // karma-ie-launcher`)
        browsers: [ 'PhantomJS' ],

        // If browser does not capture in given timeout [ms], kill it
        captureTimeout: 60000,

        // Continuous Integration mode
        // if true, it capture browsers, run tests and exit
        singleRun: false
    });
};

If you do not want to use the 'development.js' microloader to bootstrap Sencha Touch one can also point to the single JS file created by a Sencha Command test or production build. Then one does not need the proxy configuration.


        // Files for Sencha Touch test build
        files: [
            'src/main/webapp/build/testing/mae/app.js',
            'src/test/javascript/setUp.js',
            'src/test/javascript/**/*.js'
        ],

The advantage of the 'development.js' microloader approach is that you can changes your JS files and test them immediately. The drawback is that you need a web server running.

The advantage of pointing to the result of a Sencha Command build is that you do not need to have a webserver running. The drawback is that you need to rebuild after every change to your JS files to be able to test them.

You have probably noticed the file 'setup.js' that I include before the real test files.

1) This creates an element in the DOM which Sencha Touch expects
2) Delays the start of the tests till the Sencha Application is ready


// We need to add an element with the ID 'appLoadingIndicator' because [app.js].launch() is expecting it and tries to remove it
var myDiv = document.createElement("div");
myDiv.setAttribute("id", "appLoadingIndicator");
document.getElementsByTagName("body")[0].appendChild(myDiv);

// Karma normally starts the tests right after all files specified in 'karma.config.js' have been loaded
// We only want the tests to start after Sencha Touch/ExtJS has bootstrapped the application.
// 1. We temporary override the '__karma__.loaded' function
// 2. When Ext is ready we call the '__karma__.loaded' function manually
var karmaLoadedFunction = window.__karma__.loaded;
window.__karma__.loaded = function () {};

Ext.onReady(function () {
    console.info("Starting Tests ...");
    window.__karma__.loaded = karmaLoadedFunction;
    window.__karma__.loaded();
});

Another nice thing about Karma is that there is integration into the Webstorm IDE, i.e. you can run and even debug you tests from within Webstorm

Run the tests on Jenkins

I am using a different 'karma.config.js' for running tests within Jenkins.

It uses a "junitReporter" and has "singleRun=true" specified. It loads the "app.js" file  created by Sencha Command containing the Sencha Touch and my application JS files.


module.exports = function(config) {
 config.set({

  // base path, that will be used to resolve files and exclude
  basePath : '',

  // frameworks to use
  frameworks : [ 'jasmine' ],

  // list of files / patterns to load in the browser
  files : [ 'javascript/app.js','javascript/setUp.js',
    'javascript/**/*.js' ],

  // list of files to exclude
  exclude : [ 
  ],

  // test results reporter to use
  // possible values: 'dots', 'progress', 'junit', 'growl', 'coverage'
  reporters : [ 'dots', 'junit' ],

  junitReporter : {
   outputFile : 'karma-test-results.xml'
  },

  // web server port
  port : 9876,

  // enable / disable colors in the output (reporters and logs)
  colors : true,

  // level of logging
  // possible values: config.LOG_DISABLE || config.LOG_ERROR ||
  // config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
  logLevel : config.LOG_INFO,

  // enable / disable watching file and executing tests whenever any file
  // changes
  autoWatch : false,

  // Start these browsers, currently available:
  // - Chrome
  // - ChromeCanary
  // - Firefox
  // - Opera (has to be installed with `npm install karma-opera-launcher`)
  // - Safari (only Mac; has to be installed with `npm install
  // karma-safari-launcher`)
  // - PhantomJS
  // - IE (only Windows; has to be installed with `npm install
  // karma-ie-launcher`)
  browsers : [ 'PhantomJS' ],

  // If browser does not capture in given timeout [ms], kill it
  captureTimeout : 60000,

  // Continuous Integration mode
  // if true, it capture browsers, run tests and exit
  singleRun : true
 });
};

I am using Maven to build my app:

1) Sencha Command builds the app.js in the 'generate-resources' phase
2) I specify where my "test resources" are so Maven can copy them to the target folder
3) In the "test" phase Karma runs the tests.

Here is a snippet from my POM:


<build>
 <!-- Specify the location of the test resources and JS files -->
 <testResources>
  <testResource>
   <directory>${basedir}/src/test/resources</directory>
  </testResource>
  <testResource>
   <directory>${basedir}/src/main/webapp/build/${senchaBuildEnvironment}/mae</directory>
   <includes>
    <include>app.js</include>
   </includes>
   <targetPath>javascript</targetPath>
  </testResource>
  <testResource>
   <directory>${basedir}/src/test/javascript</directory>
   <targetPath>javascript</targetPath>
  </testResource>
 </testResources>
 <plugins>
  <!-- Sencha Command will build our web-application -->
  <plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>exec-maven-plugin</artifactId>
   <version>1.2.1</version>
   <executions>
    <execution>
     <id>SenchaCommand</id>
     <phase>generate-resources</phase>
     <goals>
      <goal>exec</goal>
     </goals>
     <configuration>
      <workingDirectory>${basedir}/src/main/webapp</workingDirectory>
      <executable>/sencha/Cmd/4.0.0.203/sencha</executable>
      <arguments>
       <argument>app</argument>
       <argument>build</argument>
       <argument>${senchaBuildEnvironment}</argument>
      </arguments>
     </configuration>
    </execution>
  <!-- Karma runs the JS tests in the 'test' phase -->       
    <execution>
     <id>Karma Test Runner</id>
     <phase>test</phase>
     <goals>
      <goal>exec</goal>
     </goals>
     <configuration>
      <workingDirectory>${basedir}/target/test-classes</workingDirectory>
      <executable>node</executable>
      <arguments>
       <argument>${node.modules.home}/karma/bin/karma</argument>
       <argument>start</argument>
      </arguments>
     </configuration>
    </execution>
   </executions>
  </plugin>
  ...

I have different profiles for test and production builds, .e.g.
     
<profile>
 <id>production</id>
 <activation>
  <property>
   <name>env</name>
   <value>production</value>
  </property>
 </activation>
 <properties>
  <senchabuildenvironment>production</senchabuildenvironment>
 </properties>
</profile>

Now Jenkins simply calls "mvn -Denv=production clean test".

I configured Jenkings to pick up the "karma-test-results.xml" file to display the test results:

**/target/surefire-reports/*.xml, **/target/test-classes/karma-test-results.xml

Tuesday 3 December 2013

Sencha Touch 2 production builds and delta updates


A production build by Sencha Cmd (sencha app build production) will create a minfied version of your application. 

It uses a microloader (embedded in index.html) which caches the JavaScript (e.g. app.js) and CSS (e.g. app.css) files specified in “app.json” into the Browser's local storage (using a hash of the file content to identify the “version” of the file).
 
This enables the micoloader on further loads of the application to only download the app.json file, because this file contains the information which versions of the JS and CSS files need to be loaded.

If the JS and CSS files have not changed they can be read from local storage. If a file has changed it will have a different “version”-hash in app.json. The action then taken by the Microloader depends on the value of the “update” attribute associated with the respective file:
·         "update": "delta"
o   A delta-file with the changes is downloaded by the microloader and patched into the file stored in local storage
·         "update": "full"
o   The file will be loaded via Ajax call and stored in local storage


How delta-updates work



Archive Folder:
·         During a production build Sencha Cmd copies each file into the “build/archive”-folder
·         Each file goes into a folder named like the file; the file itself will be named like the hash of its content
·        Therefore the “archive”-folder contains the history of all the versions of all the files which have been built in any production build, e.g.
o   archive\app.js\     <<<  The folder named like the file
-  88c900a6530d892d21fbae08670c074e727c35ba  <<< 1st built
-  28688b22ae4da508dc6deb3e6e90473ee525d1c1 <<< 2nd built
-  727eebf10074ef70fd1584c3ced713b610197094  <<< current built
Deltas-Folder:
·         Sencha Cmd creates a “deltas”-folder for the web application (build/[appName]/production/deltas)
·         For each of the previously built files (stored in the archive folder) a delta-json-file (compared to the current file) is created, e.g.
o   deltas\
-  88c900a6530d892d21fbae08670c074e727c35ba.json
-  28688b22ae4da508dc6deb3e6e90473ee525d1c1.json
-  Each of these files contains the delta to the current file: “727eebf10074…”
·         When the Microloader loads “app.json” it sees that the version “727eebf10074…” is the current version of “app.js”
·         Depending on the version stored in local storage it loads the correct delta file, patches the stored file and stores it in local storage with the updated version number ”727eebf10074…”



Thursday 14 November 2013

M102: MongoDB for DBAs- Cheatsheet

I took the free online course "M102: MongoDB for DBAs" offered by MongoDB Inc. via https://education.mongodb.com/ and created a cheatsheet which can be download: MongoDB for DBAs-M102-Cheatsheet.pdf.

As I took the online course "M101J: MongoDB for Java Developers" first this cheat sheet only contains information not already provided in the developers course. Therefore you have to read the DBA cheat sheet together with the cheat sheet provided for the developers course

Enjoy.

Saturday 2 November 2013

Vegetarian/Vegan restaurants in Germany (Munich, Hamburg, Berlin, Cologne, ...)

If you are looking for vegetarian/vegan restaurants in Germany:

Taken from http://www.zeit.de/2013/45/deutschlandkarte-vegane-restaurants

A list of vegie friendly restaurants in Germany can also be found here - this has also links to lists of restaurants in Austria and Switzerland: http://www.peta.de/web/restaurants.4546.html 

You can also search by ZIP code or city name here: http://www.fleischlos-geniessen.de/

Vegetarian/Vegan restaurants in Istanbul

Here are a few options I found if you want to eat vegetarian/vegan in Istanbul. All of those places are in Beyoğlu on the european side of the city.

  1. Parsifal Vejetaryen Restaurant & Cafe: Kurabiye Sokak No: 9/A 
  2. lokanta helvetia: general yazgan sokak, no: 12 - very inexpensive - very good
  3. Zencefil: Kurabiye Sokak 3
Parsifal and Zencefil are basically opposite each other near Taksim square. Helvetia is on the other end of Beyoğlu's main shopping street 'Istiklal Caddesi'.




Friday 25 October 2013

Sencha Touch 2.3: Theming for multiple platforms

Sencha provides different CSS files for different platforms. In their documentation on theming they describe how you need to configure your app.json file, e.g.


"css": [
    {
        "path": "resources/css/sencha-touch.css",
        "platform": ["desktop"],
        "update": "delta"
    },
    {
        "path": "resources/css/wp.css",
        "platform": ["ie10"],
        "theme": "Windows",
        "update": "delta"
    },
    {
        "path": "resources/css/cupertino.css",
        "platform": ["ios"],
        "theme": "Cupertino",
        "update": "delta"
    },
    ...

The respective CSS-files are provided in the folder "touch\resources\css" and at first I thought that one had to copy those into the folder "resources\css" (in which by default already the CSS file "app.css" resides).

Turns out this is not how it works. The CSS file in "resources\css" are generated via Compass and Sass. Therefore one has to

1) copy the .scss files for the themes one is interested in from "touch\resources\sass" to "resources\sass", eg. copy "touch\resources\sass\cupertino.scss" to "resources\sass\cupertino.scss".

Note: You can also delete the existing "app.scss" file from "resources\sass" as you probably do not refer to it anymore from you app.json file. It really is only a copy of the default theme "sencha-touch.scss" and therefore only needed if you only work with one theme - the default one. 

2) execute "compass compile cupertino.scss" in "resources\saas" (you need to have ruby and compass installed) -> this will generate "resources\css\cupertino.css". Repeat this for all the other theme .scss files you use.

When you build using sencha cmd ("sencha app build") it will also call  "compass compile".

The "compass compile" process is controlled by the config file "resources\saas\config.rb". By default it is configured to generate a compressed CSS file (output_style = :compressed). Change this to "output_style = :expanded" if you want to get a readable version of the CSS.

Add your own styles to the themes

You can simply add your own Sass instructions or plain CSS to the copied resources\sass\[theme].scss files.

You can also externalize you Sass/CSS into a separate file and import this into the [theme].scss files (using for example @import '../mySass/myProject.scss';).

If you want to add you own custom (font) icons take a look at Bruno Tavares blog on how to do it. 

Thursday 10 October 2013

JAX-RS Joda DateTime ParamConverterProvider

I have written a JAX-RS ParamConverterProvider for the Joda DateTime class:

import java.lang.annotation.Annotation;
import java.lang.reflect.Type;

import javax.ws.rs.ext.ParamConverter;
import javax.ws.rs.ext.ParamConverterProvider;
import javax.ws.rs.ext.Provider;

import org.joda.time.DateTime;
import org.joda.time.format.ISODateTimeFormat;

@Provider
public class DateTimeParamConverterProvider implements ParamConverterProvider {

 @Override
 public  ParamConverter getConverter(Class type, Type genericType, Annotation[] annotations) {
  if (type.equals(DateTime.class)) {
   return (ParamConverter) new DateTimeParamConverter();
  } else {
   return null;
  }

 }

 private static class DateTimeParamConverter implements ParamConverter {
  @Override
  public DateTime fromString(String value) {
   try {
    return ISODateTimeFormat.dateTimeNoMillis().parseDateTime(value);
   } catch (IllegalArgumentException e) {
    return ISODateTimeFormat.dateTime().parseDateTime(value);
   }
  }

  @Override
  public String toString(DateTime value) {
   return value.toString();
  }

 }
}

Thursday 26 September 2013

Book "SQL Performance Explained"

I just read Markus Winand's wonderful book "SQL Performance Explained" which can be bought as an e-book or print edition and even be read online. The books sole topic is B-tree backed database indices.

The preface already states that "Database indexing is a development task." This is because to define an optimal index one must understand how the application queries the data.

→ This means that every developer should read this book. It will show you how to read execution plans and how to create indices for a wide range of scenarios.




Here are some key points, but you really have to read the book:

1) It is better to have a single index for all columns of a where clause than one index for each column.

2) The column order of a concatenated index has great impact on its usability so it must be chosen carefully.

A database can use a concatenated index when searching with the leading (leftmost) columns. An index with three columns can be used when searching for the first column, when searching with the first two columns together, and when searching using all columns.

→ The most important consideration when defining a concatenated index is how to choose the column order so it can support as many SQL queries as possible.

3) If you use a function (TRUNC, UPPER, ...) on a column in the where clause you must also create an index on the function

4) When searching for ranges index for equality first—then for ranges. This will keep the scanned index range as small as possible.

Example:
   WHERE date_of_birth >= TO_DATE(?, 'YYYY-MM-DD')
   AND date_of_birth <= TO_DATE(?, 'YYYY-MM-DD')
   AND subsidiary_id  = ?

→ The index should be on SUBSIDIARY_ID and then on DATE_OF_BIRTH

5) Even if you have a full copy of the production database in your development environment, the concurrent background load can still cause a query to run much slower in production.

→ Careful execution plan inspection yields more confidence than superficial benchmarks.

6) An index can be used for an order-by sort. And because databases can read indexes in both directions this can even be in the other direction. It is only important that the scanned index range is in the exact opposite order to the order by clause.

7) And a last one not only for performance but also for security: Parameterized Queries are the best way to prevent SQL injection.

Monday 23 September 2013

M101J: MongoDB for Java Developers - Cheatsheet

I took the excellent and free online course "M101J: MongoDB for Java Developers" offered by MongoDB Inc. via https://education.mongodb.com/ and created a cheatsheet which can be download: MongoDB-M101J-Cheatsheet.pdf

Enjoy.

Thursday 19 September 2013

Sencha Touch 2, CSRF & Double Submit Cookies to protect stateless REST-Services

To protect my stateless REST services against Cross Site Request Forgery attacks I use Double Submit Cookies. For more information on this see http://appsandsecurity.blogspot.de/2012/01/stateless-csrf-protection.html

On the (Java) server side I wrote a servlet filter which checks that the cookie value and the HTTP header are specified and that they match:



public class CsrfDoubleSubmitCookieFilter implements Filter {
 private static final Logger logger = LoggerFactory.getLogger(CsrfDoubleSubmitCookieFilter.class);
 private static final String ANTI_CSRF_DOUBLE_SUBMIT_COOKIE = "Anti-Csrf-Double-Submit-Cookie";

 @Override
 public void destroy() {
 }

 @Override
 public void init(FilterConfig config) throws ServletException {
 }

 @Override
 public void doFilter(ServletRequest servletRequest, ServletResponse response, FilterChain chain) throws IOException, ServletException {
  HttpServletRequest request = (HttpServletRequest) servletRequest;
  String antiCsrfHeader = request.getHeader(ANTI_CSRF_DOUBLE_SUBMIT_COOKIE);
  String antiCsrfCookie = getCookie(request.getCookies(), ANTI_CSRF_DOUBLE_SUBMIT_COOKIE);

  logger.debug("Checking CSRF cookie & header for: {}{}", request.getServletPath(), request.getPathInfo());
  checkCrsfToken(antiCsrfHeader, antiCsrfCookie);

  chain.doFilter(request, response);
 }

 /**
  * Checks that the antiCsrfHeader and the antiCsrfCookie are not blank and do match
  */
 private void checkCrsfToken(String antiCsrfHeader, String antiCsrfCookie) {
  if (StringUtils.isBlank(antiCsrfHeader) || StringUtils.isBlank(antiCsrfCookie) || !antiCsrfHeader.equals(antiCsrfCookie)) {
   logger.error("POSSIBLE CSRF (Cross-Site-Request-Forgery) ATTACK !!! -> Anti-Csrf-Header {} and Anti-Csrf-Cookie {} do not match",
     antiCsrfHeader, antiCsrfCookie);
   throw new WebSecurityException("");
  }
 }

 private String getCookie(Cookie[] cookies, String name) {
  String retVal = null;
  for (int i = 0; i < cookies.length; i++) {
   if (name.endsWith(cookies[i].getName())) {
    retVal = cookies[i].getValue();
    break;
   }
  }
  return retVal;
 }
}



On the Senche Touch side I added a 'beforerequest' listener function to Ext.Ajax in the launch-method of app.js. This will add a new CSRF token to the Anti-CSRF Cookie and Header for each Ajax request



  launch: function () {
        "use strict";
        // Destroy the #appLoadingIndicator element
        Ext.fly('appLoadingIndicator').destroy();
        //Add double submit anti-CSRF cookie and header to each Ajax-request

        Ext.Ajax.addListener( 'beforerequest', function(conn, options, eOpts) {  
          "use strict";
          var csrfToken = Math.random();
          document.cookie='Anti-Csrf-Double-Submit-Cookie=' + csrfToken + ';path=/';
          options.headers['Anti-Csrf-Double-Submit-Cookie'] =csrfToken;
        }

    },


Note: When you keep state on the server side it is obviously more secure to let the server generate a new CSRF token for each user session and put it in the cookie. The client has to read the cookie value and set it into the header for each request. The server can than compare the CSRF token value from the server side session with the header value provided by the client.