Google Wave Tips: catching robot exceptions

Almost trivial code, which I found very useful debugging my robots

public void processEvents(RobotMessageBundle bundle) {
   try {
      processEventsInternal(bundle);
   }
   catch (Throwable t) {
      final StringWriter stringWriter = new StringWriter();
      t.printStackTrace(new PrintWriter(stringWriter));
      bundle.getWavelet().getRootBlip().getDocument().append(stringWriter.getBuffer().toString());
   }
}

Groovy+Scala+Java

Almost to years ago I’ve invented and then wrote joint compiler for Groovy and Java. The idea was pretty simple – we do compilation in 3 steps
1) Generate Java stubs from Groovy sources
2) Compile all Java code with generated stubs
3) Compile Groovy code with already compiled Java classes

First implementation was done as part of IntelliJ and later another one was written to contribute in Groovy compiler infrastructure. Thanks a lot to Jochen and other guys who improved and incorporated it in to groovyc and ant tasks. It became probably last cornerstone for 100% interoperability between Groovy and Java and now understood by everybody as natural as if it was in Groovy from day one.

But now my arsenal includes Scala as well. Obviously I want to have Groovy and Scala and of course Java together. Isn’t it natural wish?

Fortunately scala compiler is smart as well and knows how to do joint compilation with Java. Scala does it a bit differently than Groovy – by parsing Java source during Java compilation. So the process looks like following
1) Compile Scala code using Java sources on the way
2) Compile Java sources using already compiled scala classes

Unfortunally (but not surprisingly) scala has no clue about Groovy.

Today I realized that the small modification of my original algorythm + scala compiler can create joint compiler for all three languages together. The complete algorithm is following
1) Generate Java stubs from Groovy sources
2) Compile Scala code using Java sources and generated stubs on the way
3) Compile all Java code with generated stubs and compiled Scala classes
4) Compile Groovy code with already compiled Java classes

I believe you will not be surprised that when I knew what I want to do it was pretty easy to implement. In fact it was mostly refactoring of what we did before. So now in my working copy I can compile Groovy+Java+Scala together. Of course all references in all three languages are resolved correctly in all possible direction.

I think it opens many amazing opportunities of Groovy becoming orchestration language for people who needs both static and dynamic typing plus functional programming. Of course I would like to see it as part of Groovy Core.

I would like to learn what other people think about it. Please let me know.

GroovyGrid DSL poll: what is right way to implement branching control structures in Groovy

It is well-known that you can’t implement if/else type control structure in Java. Another useful example of such control structure is continuations. It is much less know that you can implement such kind of structure in Groovy. In this short article I will show several options how it can be done. Our use case will be the same: check point definition for GroovyGrid. The main quesion of the post is which option is better from readability point of view. I will appreciate any comments from Groovy user, which way do they prefer.

Usecase

GroovyGrid is Groovy DSK for GridGain framework. I develop GroovyGrid with idea to simplify coding of grid applications using Groovy instead of Java. Our use case for this article will be checkpoint definition.

So what is checkpoint? Checkpoint is possibility for extensive computing jobs can save their intermidiate state, so if job failed and was started by GridGain again it can continue not from scratch but using previously save state. In general logic of such job looks like following pseudo-code

Check if saved state exist
If it doesn't
     Execute first part of the job
     Save result of first part on checkpoint
Execute second part of job using either loaded or calculated result of first part

Now we are redy to start with implementation options

Option I: Closure after closure

We can use following syntax

checkPoint ("checkpoint") {
     // first part calculation
}
{ state ->
    // second part calculation
}

As you can see we put two closures one after another without any syntactical indication of connection between closures. Fortunately for us Groovy parser understand such construction as call of ‘checkpoint’ method with one parameter of type String and two parameters of type closure. So our implementation of ‘checkPoint’ method is pretty trivial (avoiding of course exception handling and such)

def checkPoint(String name, Closure before, Closure after) {
    Serializable state = gridTaskSession.loadCheckpoint(name)
    if (state == null) {
       state = before ()
       gridTaskSession.saveCheckpoint("checkpoint", state)
    }
    after(state)
}

What I like in this option is that it is extremly simple in implementation and very straight forward. Important to note here that if we want to use several checkpoints it is very easy to nest it.

checkPoint("state2") {
    checkPoint("state1") {
        // step1
     }
     { state1 ->
       // step2
     }
}
{ state2 ->
   // step3
}

Option II: Control object

What I don’t like in Option I is the fact that closure are not ‘visually’ connected in the code. So here are improved syntax

checkPoint ("checkpoint") {
    // first part calculation
}.andContinue { state ->
    // second part calculation
}

What we gain is ‘andContinue’, which probably improves readability. What we complicate is implementation – now ‘checkPoint’ method returns not value but control object to be called to complete the calculation

def checkPoint(String name, Closure before) {
    [
      andContinue : { Closure after ->
          Serializable state = gridTaskSession.loadCheckpoint(name)
          if (state == null) {
              state = before ()
              gridTaskSession.saveCheckpoint("checkpoint", state)
          }

          after (state)
      }
    ]
}

There is nice trick we can use to variate the same idea – use ‘rightShift’ of ‘rightShiftUnsigned’ name instead of ‘andContinue’ It allows us to write code like this (‘rightShiftUnsigned’ case)

checkPoint ("checkpoint") {
    // first part calculation
}  >>> { state ->
    // second part calculation
}

Again nesting is pretty simple

checkPoint("state2") {
    checkPoint("state1") {
        // step1
     } >>> { state1 ->
       // step2
     }
} >>> { state2 ->
   // step3
}

Option III: No after-brunch at all

Careful reader might notice for our particular use case special syntax for ‘after brunch’ is not necessary because we will always execute it. So we can simply write

def state = checkPoint ("checkpoint") {
    // first part calculation
}
// second part calculation

There main reason I consider other options here is because the article is more about methodology and not about particular use case. For example, in case if we want to execute first part asynchronously on grid or thread pool and and continue with second part only after first one completed (of course without waiting) the current option will not work at all.

Anyway for completeness here is implementation

def checkPoint(String name, Closure before) {
    Serializable state = gridTaskSession.loadCheckpoint(name)
    if (state == null) {
       state = before ()
       gridTaskSession.saveCheckpoint("checkpoint", state)
    }
    state
}

Main questions

1. Which option do you prefer?
2. What are other options?

Please let me know and Enjoy Groovy!

GroovyGrid: grid computing with Groovy and GridGain

Grid computing is complicated field. Even with perfect GridGain framework, which helps a lot in developing and testing distributed applications, it still require writing some auxilary Java code, which mostly serves framework itself and not application logic (like defining classes for tasks and jobs). Today I want to talk a bit about small Groovy DSL for GridGain, which I develop with purpose to simplify the story.

It is important to notice that GroovyGrid is still to be experimental project and one of the main purposes of this post is to collect opinions if it can be useful for other people.

The task we gonna solve is pretty standard – given directory (let us assume on shared file system) and we want to calculate how many times each character occurs in all Groovy and Java source files recursively. Solution of this problem should help us to see GroovyGrid in action.

For purpose of this post I wrote trivial Groovy class, which will help us to accumulate statistics. It is a bit lengthy but in reality everything it does is definition of several ‘leftShift’ methods, which allow us write something like statistic << data, where data can be list of files, or file, or String or char or even another statistic accumulator. You can easily skip following code.

class CharStatisticAccumulator extends HashMap {
    private CharStatisticAccumulator add (char ch, counter = 1) {
        if (!this[ch])
          this [ch]  = counter
        else
          this [ch] += counter
        this
    }

    CharStatisticAccumulator leftShift (char ch) {
        add ch
    }

    CharStatisticAccumulator leftShift (Map.Entry e) {
        add e.key, e.value
    }

    CharStatisticAccumulator leftShift (Collection coll) {
        coll.each {
            this << it
        }
        this
    }

    CharStatisticAccumulator leftShift (String line) {
        this << Arrays.asList(line.chars)
    }

    CharStatisticAccumulator leftShift (File file) {
        file.eachLine {
            this << it
        }
        this
    }

    CharStatisticAccumulator leftShift (CharStatisticAccumulator statistic) {
        this << statistic.entrySet ()
    }
}

Now we are back to our main task of calculation statistic. Standard map/reduce approach nicely implemented by GridGain advice us how to do it:
1) split files by groups

2) assign calculation of statistic for each group as job to different grid nodes

3) combine results calculated by all jobs in to final statistic.

Here is final code for calculation of our statistic. It is richly commented and I hope self-explaining. I am not trying to be lazy here but found that commented code probably the best way to demonstrate what’s going on.

    /**
     * Recursively calculates number of character usage in
     * .java and .groovy files of given directory
     */
    static def calculateCharStatistic (File dir) {

        // In GridGain each task consist of one or several jobs.
        // Two main operations of task
        // - map - which split task to jobs and assigned jobs to grid nodes
        // - reduce - which combains result of jobs to result of task
        //
        // In GroovyGrid body of closure solves both task.
        // It will be called in 'map' method  with both purposes
        // of creating jobs and define useful callbacks
        // like 'reduce', 'onJobResult' etc.
        GroovyGrid.mapReduce {

            // recursively collect all .java and .groovy files
            // it is usual Groovy and has nothing to do with grid computing
            def files = []
            dir.eachFileRecurse { File innerFile ->
                if (!innerFile.directory
                 && ( innerFile.name.endsWith(".java")
                   || innerFile.name.endsWith(".groovy"))) {
                    files << innerFile
                }
            }

            // group files in to groups by 64 files
            // again usual Groovy iteration/grouping
            int groupIndex = 0
            Map groups = files.groupBy { (groupIndex++) >> 6 }

            
            // iterate groups
            groups.each { groupId, groupFiles ->
                // define grid job for each group
                // It is important to note that closure below will be
                // called not immediately but later 
                // on some (possibly different) grid node.
                def job = job {
                    println "Job $groupId started"
                    CharStatisticAccumulator jobResult
                        = new CharStatisticAccumulator()
                    groupFiles.each { File jobFile ->
                        jobResult << jobFile
                        println "processed $jobFile.absolutePath"
                    }
                    println "Job $groupId ended"

                    // result of job execution
                    jobResult
                }

                // Usually it is not necessary to map jobs to grid nodes
                // manually but if needed it is possible to do
                // by assigning of 'gridNode' property of job
                // 'gridGrid' property of type Grid and
                // 'gridLoadBalancer' of type GridLoadBalancer
                // are injected in each task.
                //
                // In fact, this demostration is only purpose of 'job' variable -
                // you usually don't do it in normal life.
                if (groupId % 5 == 0)
                    job.gridNode = gridGrid.localNode
                else
                    job.gridNode = gridLoadBalancer.getBalancedNode (job)
            }

            // here we will collect statistic
            def taskResult = new CharStatisticAccumulator()

            // defines callback to be called when result
            // of a job becomes available
            onJobResult { GridJobResult jobResult
                      /*, List<GroovyJobResult> receivedBefore */ ->
                // Interesting to notice that we are able
                // to access 'groupId' of the job here
                println "Result of job $jobResult.job.groupId obtained"

                // normally GridGain process all partial results in 'reduce' method
                // It is possible with GroovyGrid as well but here
                // for demonstration purposes we do differently -
                // partial result added immediately in to result statistic
                taskResult << jobResult.data

                // as we don't need result of the job anymore we can forget it
                jobResult.data = null
            }

            // defines callback to be called to calculate task result
            // after results of all jobs became available
            reduce { List<GridJobResult> results ->
                // Let us print IDs of all jobs to make sure
                // that we've calculated everything
                // Of course, here we are also able to access 'groupId' of the job
                println results*.job*.groupId.sort ()

                // result of task calculation
                taskResult
            }

            // defines callback to be called when task finished
            onTaskFinished { GridTaskFuture future ->
                println "Task completed"
            }
        }
    }

I will appreciate comments and advices from everybody – especially from people familiar with both Groovy and GridGain.

Groovy dynamic stateful mixins

My last post was dedicated to Groovy dynamic stateless mixins. Today I want to talk about stateful ones, which are now presented in Groovy trunk.
The main difference is obvious – additionally to providing new method stateful mixins can also keep state.

Here is non-trivial sample. First of all let us use EMC to add convinient method to JDK class ReentrantLock.

ReentrantLock.metaClass {
  withLock { Closure operation ->
     lock ()
     try {
        operation ()
     }
     finally {
       unlock ()
     }
  }
}

Now we are able to implement Groovy ConcurrentQueue like this

class ConcurrentQueue {
   private LinkedList list = new LinkedList ()
   private ReentrantLock lock = new ReentrantLock ()

   def get () {
      lock.withLock () {
        list.removeFirst ()
      }
   }

   void put (def obj) {
      lock.withLock () {
         list.addLast (obj)
      }
   }
}

So far, we didn’t use any mixins. Let us rewrite last sample using ones

class ConcurrentQueue {
   static {
     ConcurrentQueue.metaClass.mixin LinkedList, ReentrantLock
   }

   def get () {
      withLock {
        removeFirst ()
      }
   }

   void put (def obj) {
      withLock {
         addLast (obj)
      }
   }
}

What we did is we replace private fields with mixins, which we assign in static initialization of the class. Of course, this sample is a little bit artificial because what we do is emulating compile time mixins with static initializer. I will discuss compile time mixins separately but now let us modify our sample to be more dynamic.

        def queue = new Object ()
        queue.metaClass {
            mixin LinkedList, ReentrantLock

            get { ->
               withLock {
                 removeFirst ()
               }
            }

            put { obj ->
               withLock {
                  addLast (obj)
               }
            }
        }

Instead of creating special class for our concurrent queue we use combination of mixin and per instance meta class to create what we need. Isn’t it lovely?

Now you may wish to ask me several interesting question, which I try to answer below.

How to access instance of mixed-in object if needed? Well, it is very easy. Let us add one more method to implementation of our queue, which will demonstrate how to do that.

        queue.metaClass {
            duplicateEachElement {
               withLock {
                  LinkedList newList = new LinkedList ()
                  mixedIn[LinkedList].each {
                     newList << it
                     newList << it
                  }
                  mixedIn[LinkedList] = newList
               }
            }
        }

How to access owner object from mixed-in instance if needed? This is a little bit more tricky. To understand why is it so we need to understand how mixins are implemented. Obviously, where is no magic here – mixed in references kept separately from objects in weakly referenced map. It means that if mixed in instance keeps reference to owner, then owner has zero chances to be collected. So in the perfect world mixed in instance should never keep reference for owner. But in pragmatic world it might make sense to do it time to time. Here is example how to do that

class NoDuplicateCollection {
    void put (def obj) {
        def clone = (mixinOwner as Collection).find {
            it == obj
        }

        if (!clone)
          mixinOwner.add obj
    }
}

def list = new Object ()
list.metaClass.mixin NoDuplicateCollection, LinkedList

list.put 1
list.put 1
list.put 2
list.put 2
list.put 3
list.put 3

assertEquals (3, list.size ())
assertEquals 1, list [0]
assertEquals 2, list [1]
assertEquals 3, list [2]

NoDuplicateCollection class designed to be used with any JDK Collection as owner. In example above we use it with LinkedList but any other Set or List will work as well. Property ‘mixinOwner’ is magically (via delegating per instance meta class) inserted in to instance of NoDuplicateCollection and… Voila! Of course, internally mixedOwner is week reference so it will not prevent owner from being collected.

One more interesting thing to notice is ‘as operator’ used to cast our object to Collection. Yes, it is true – we cast instance of Object to Collection. Again, there is no magic here – if our object can’t be casted itself we try mixed-ins. Simple as that.

Enjoy it in the Groovy 1.6 trunk

Groovy dynamic stateless mixins

Last week I wrote about per-instance meta-classes and ExpandoMetaClass DSL. At that point it was just prototyped code in my working copy and I was not even sure if it will come to Groovy 1.6 trunk. Now it is there. If you didn’t read this previous post I recommend to have quick look on it as it might help to understand current one.

Today I want to talk about another part of the same commit, which is kind of side-effect of ability to have per-instance meta classes. Now we able to have per instance dynamic stateless mixins.

But let me start we explanation of what is mixin for me.

Mixin is either static or dynamic way to extend class or instance behavior by adding methods and/or state defined in another class.

Static here means defined at compile time and dynamic means defined on runtime. This two types of mixins are totally different.

• Static mixins applied to class but dynamic one to meta class (either class one or instance one)
• Static mixins always effect hierarchy, dynamic ones may or may not effect it depending on nature of meta class under effect (if meta class is the one associated with the class it does, if not it does not
• Static mixins always effect type information, dynamic ones never does that.

Stateless or stateless mixins means ability of mixins to extend object behavior not only with methods but with state as well. Stateful compile time mixins can be understand as kind of multiple inheritance.

Groovy has no compile time mixins yet. I think Groovy has to have something similar to Scala traits but it is subject to different discussion.

Groovy has no stateful mixins yet as well and below I will talk a bit about my idea how they should look like. So main subject for today is dynamic stateless mixins

Let me show simple example. First of all we define usual category class, which will be used to mix in.

class DeepFlattenToCategory {
    static Set flattenTo(element) {
        LinkedHashSet set = new LinkedHashSet()
        element.flattenTo(set)
        return set
    }
    // Object - put to result set
    static void flattenTo(element, Set addTo) {
        addTo <
           element.flattenTo(set);
        }
    }
    // Map - flatten each value
    static void flattenTo(Map elements, Set addTo) {
       elements.values().flattenTo(addTo);
    }
    // Array - flatten each element
    static void flattenTo(Object [] elements, Set addTo) {
        elements.each { element ->
           element.flattenTo(set);
        }
    }
}

This category does pretty obvious thing – deep flattening of complex data structure as demonstrated by following code snippet.

        // mixin Category to meta class
        Object.metaClass.mixin DeepFlattenToCategory
        // and here we are
        assertEquals ([8,9,3,2,1,4], [[8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

Careful reader may notice probably that we modify only meta class for Object but methods for Collection, Object [] and Map becomes available as well. Such behavior is usual for categories but a bit uncommon for ExpandMetaClass – in the past he had to call ExpandoMetaClass.enableGlobally () to ask meta classes to look for missed methods. Such approach had several downsides – to name a few: unnecessary memory footprint for creation ExpandoMetaClass for each class (both modified and non-modified) and lack of “good place” where to call ExpandoMetaClass.enableGlobally (). It means that if you developed library, which requires it – you should call it somewhere, because nobody can guarantee that user application will do that. Fortunately, now ExpandoMetaClass can define methods for sub classes and if sub class missed some method he will try to find it in hierarchy. Obviously such behavior is must for mixins – otherwise our sample will look much less nicer(which is still to be valid code but creating four ExpandoMetaClasses instead of one)

        // mixin Category to meta classes
        Object.metaClass.mixin DeepFlattenToCategory
        Collection.metaClass.mixin DeepFlattenToCategory
        Object[].metaClass.mixin DeepFlattenToCategory
        Map.metaClass.mixin DeepFlattenToCategory
        // and here we are
        assertEquals ([8,9,3,2,1,4], [[8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

        Object.metaClass.foo(ArrayList){ -> .... }

BTW, ExpandoMetaClass has new syntax to define methods for subclasses

        Object.metaClass.foo(ArrayList){ -> .... }

OR

        Object.metaClass {
           foo(ArrayList){ -> .... {
        }

OR

        Object.metaClass {
           define(ArrayList){
              foo{ -> .... }
           }
        }

OK, back to mixins. So far, it seems very similar to normal categories, so let us make our sample more interesting

        class NoFlattenArrayListCategory {
           // ArrayList - put to result set
           static void flattenTo(ArrayList element, Set addTo) {
              addTo << element;
           }
        }

        def x = [2,3]
        x.metaClass.mixin NoFlattenArrayListCategory
        assertEquals ([x, 8,9,3,2,1,4], [x, [8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

What we do here is create category, which does not flatten array lists and mix it in instance meta class. Voila!
Of course the same can be achieved with ExpandoMetaClass without mixing in category

        def x = [2,3]
        x.metaClass.flattenTo = { Set addTo -> addTo << delegate }
        assertEquals ([x, 8,9,3,2,1,4], [x, [8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

but please remember that category can be easily implemented in Java and provide much better performance compare to closure-based meta methods. And now we can use it as mixins.

The last thing I want to talk about is some features, which is missing and which I would love to see in Groovy. All below is just imagination and I don’t know exact implementation details and underwater stones but here are problems to be addressed.

• Stateful mixins
• Mixing in objects
• Better syntax to define categories in Groovy
• Compile time mixins

Stateful mixins

Imagine that category class defines not only static methods but also some fields and these fields magically becomes available as properties when we mix these category in. Probably if object already has such properties the one defined by category should be ignored, but if not… we have extremely powerful tool.

The main technical problem here is that we don’t want to define these new properties via ExpandoMetaClass, which will be very slow, but if we created instance of category class to associated it with original object… well, we have no good place to store this association. For Groovy compiled objects we of course can store refernence(s) directly in the object and for Java one in WeakHashMap. But what will happen if this mixed in instance keeps strong reference for original Java object… It will never be collected. Believe me, it will happen very often. So far I don’t know any good solution for that.

Mixing in objects

If we had solution for stateful mixin most probably it will lead us to ability to mix in not only category class but any object. I can imagine fantastic possibilities for that.

Better syntax to define categories in Groovy

Well static methods with additional parameter ‘self’ are so ugly. We definitely need something better. So far we have @Category annotation, which solves only part of the problem.

Compile time mixin

Having stateful mixins with good syntax to define category methods will lead us to ability mixin category classes on runtime. As result we will have probably the strongest mixin story between all dynamic languages and as powerful as Scala traits in the world of staticly compiled ones.

So it goes.

Groovy per instance meta classes and ExpandoMetaClass DSL

So far Groovy allow per instance meta classes only for objects, which implements GroovyObject interface (like any class defined in Groovy). It can be illustrated by following simple test case:

   // for java.lang.String
   "test string".metaClass.toString = { "modified string" }
   assertEquals( "modified string", "test string".toString() )
   // but any other string will have the same behavior
   assertEquals( "modified string", "any other string".toString() )

At the same time

   // for GroovyObject
   class MyBean {
      String toString () { "bean" }
   }

   def bean = new MyBean ()
   def emc = new ExpandoMetaClass(MyBean,false,true)
   emc.toString = { -> "modified bean" }
   bean.metaClass = emc
   assertEquals("modified bean", bean.toString())
   // but any other bean will keep default behavior
   assertEquals("bean", new MyBean().toString())
   // if we want to modify behavior of all MyBean objects
   MyBean.metaClass = emc
   assertEquals("modified bean", new MyBean().toString())

Notice that we use different technique to add methods to Java and Groovy objects. As getMetaClass () for Java object always returns the same meta class we can use .metaClass notation, each GroovyObject has it is own meta class (of course, it is the same object by default) so we act differently depending on what we want to modify (whole class or just one Object)

Now I have patch which solves all these problems.

First of all we can assigned per instance meta class for any object. Of course, for Java objects it kept in WeakHashMap which involves some performance penalties. But now our first sample looks much more natural

   // for java.lang.String
   "test string".metaClass.toString = { "modified string" }
   assertEquals( "modified string", "test string".toString() )
   // but any other string will have natural behavior
   assertEquals( "any other string", "any other string".toString() )

Groovy objects also has benefits – we don’t need to create ExpandoMetaClass manually.

   def bean = new MyBean ()
   bean.metaClass.toString = { -> "modified bean" }
   assertEquals("modified bean", bean.toString())
   // but any other bean will keep default behavior
   assertEquals("bean", new MyBean().toString())
   // if we want to modify behavior of all MyBean objects
   MyBean.metaClass = bean.metaClass
   assertEquals("modified bean", new MyBean().toString())

But now, when we have unified behavior for Java and Groovy objects we can use simply and powerful ExpandoMetaClass DSL. Here is example

        // add behavior to meta class for Integer
        Integer.metaClass {
           // block of static methods
           'static' {
                fib { Number n ->
                    n.fib ()
                }
                // we can use both = or << if needed
                unusedStatic << { -> }
            }

            // one more syntax for static methods
            static.unusedStatic2 = { -> }

            // property definition
            ZERO = 0

            // method definition
            fib { ->
                def n = delegate
                if (n == ZERO)
                  return 1;
                if (n == 1)
                  return 1
                else
                  return fib(n-1) + fib(n-2)
            }

            // of course we can use both = or << if needed as well
            unusedMethod << { -> }
        }
        // and now we have it
        assertEquals( 3, 3.fib())

        // but we can also modify meta class for particular number
        4.metaClass {
            fib { ->
                10
            }
        }

        // and it works!
        assertEquals( 13, Integer.fib(5))

You know what… I like it. And thanks a lot for Graeme for inspirations and advices while I worked on that. Hope it will be accepted well and commited to the trunk.

Groovy testing: 3 lessons from 1 test case

As Groovy Core developer involved in compiler and language runtime I am always worried for broken tests in the trunk. Especially when it happened after my commit. But even if it is not a case I have big problem with broken trunk because to test any of my changes in the local copy I need to re-run whole Groovy test suite.

So when after update this morning I found that one test is broken I decided to have a look what’s going on. Especially strange was the fact that test was broken on Windows build server but passed well on Linux one. We had such problem recently, which was probably caused by different default stack size on different VMs and forced me to improve compiler a bit. So I start to look on the test.

It is relatively simple

class Groovy1759_Bug extends GroovyTestCase {
   void testInterception() {
      def benchmarkInterceptor = new BenchmarkInterceptor()
      def proxy = ProxyMetaClass.getInstance(A.class)
      proxy.setInterceptor(benchmarkInterceptor)
      proxy.use {
         def a = new A()
         a.a()
         a.b()
      }

      def actual = benchmarkInterceptor.statistic()
      def expected = [['b', 2, 0],['ctor', 1, 0],['a', 1, 0]]
      assert expected == actual
   }
}

class A{
   void a(){ b() }
   void b(){}
}

We test BenchmarkInterceptor, which records all calls to different methods and record number of executions and accumulated time spent inside methods. Then we compare actual results with our expectations.

Here is piece of code, which actually does the job:

def actual = benchmarkInterceptor.statistic()
def expected = [['b', 2, 0],['ctor', 1, 0],['a', 1, 0]]
assert expected == actual


But it was failing with following not very helpful message:
java.lang.AssertionError: Expression: (expected == actual). Values: expected = [[b, 2, 0], [ctor, 1, 0], [a, 1, 0]], actual = [[Ljava.lang.Object;@cc06e0, [Ljava.lang.Object;@d7643a, [Ljava.lang.Object;@f52a1d]

Lesson 1: If you provide diagnostic try to provide useful one.

I had quick look to compiler and immediately found that instead of call to stringBuffer.append(obj) during generation of assert statement it would be much better better to use stringBuffer.append(InvokerHelper.toString(obj)). Fast check and … voila! It works and I have much better error message:

java.lang.AssertionError: Expression: (expected == actual). Values: expected = [[b, 2, 0], [ctor, 1, 0], [a, 1, 0]], actual = [[ctor, 1, 0], [a, 1, 0], [b, 2, 0]]

OK, content of expected and actual lists are the same but the order of elements is wrong. Fortunately, I had this lesson many times before

Lesson 2: Never expect any particular order of elements in a hash map. Or either sort or use linked one if you need an order.

What hash map one can ask? Really, I had no idea but I could imagine that BenchmarkInterceptor use internal one to record calls. And it is was exactly the case – method names as keys and list of current time before and after the call as values.
So the fix was almost trivial – replace HashMap by LinkedHashMap in BenchmarkInterceptor and put expected values in order how calls happen


def expected = [['ctor', 1, 0],['a', 1, 0],['b', 2, 0]]
assert expected == actual

At that point I run clean build, checked that all tests passed and commited my changes. In fact before commit I also run remote build on TeamCity, where test failure was reported from. In two minutes I had beautiful surprise from build server. Test was failing!

It took me another 5 or 6 local runs tio reproduce failure on my environment. Some time execution time was 1 instead of 0.

Lesson 3: Never expect particular execution time of piece of code

Do I need to explain? Probably not.

Thanks to Groovy here is final code snippet.

      def actual = benchmarkInterceptor.statistic().collect{ [ it[0], it[1] ] }
      def expected = [['ctor', 1],['a', 1],['b', 2]]
      assert expected == actual

And it works much better.

Bonus Lesson: Analize your tests we have a lot to learn from our failures

Grails Trainings in Scandinavia

We were asked about that very often and now it happened.

G2One, Inc., company behind both Grails and Groovy, and Callista Enterprise AB announce today series of public trainings in Scandinavia.

Göteborg 2008-09-08
Stockholm 2008-09-15
Malmö 2008-09-22

This 3 day course covers both Groovy language and Grails framework. After an introduction to the basics of the Groovy language, Students are taken through a step-by-step lab-driven experience on how to build a solid Grails application learning from the people who created the technology. We will start from basic but will go very deeply.

Full description of course can be found here

BTW, thanks to our friends from JetBrains each student will receive free 12 month license for IntelliJ IDEA

If you are interested to participate in one of these courses please register here or email to us on training at g2one dot com. Group discounts and discounts for students coming from non-Scandinavian countries are available, so we have what to talk about 🙂

And I totally forgot – many other European and North American locations of trainings will be announced very soon.

TeamCity saved my day

I had crazy problem this morning – Groovy Core build was failing with StackOverflowException on some Windows configurations but not on Linux or Mac. Unfortunately, it happen after my commit several days ago, so I was somehow responsible for that.

I have only Mac. No Linux. No Windows. So my first idea was to ask someelse to take care for the bug. But you know what… It is somehow not very professional.

Fortunately, I found very simple solution. TeamCity Remote Build. 45 minutes and 15 builds and problem was located. I even was able to create test case, which also failing on my Mac. Really cool.

Now I have more serious problem – to understand how to fix it. But it is kind of more regular work.