Groovy testing: 3 lessons from 1 test case

May 25, 2008

As Groovy Core developer involved in compiler and language runtime I am always worried for broken tests in the trunk. Especially when it happened after my commit. But even if it is not a case I have big problem with broken trunk because to test any of my changes in the local copy I need to re-run whole Groovy test suite.

So when after update this morning I found that one test is broken I decided to have a look what’s going on. Especially strange was the fact that test was broken on Windows build server but passed well on Linux one. We had such problem recently, which was probably caused by different default stack size on different VMs and forced me to improve compiler a bit. So I start to look on the test.

It is relatively simple

class Groovy1759_Bug extends GroovyTestCase {
   void testInterception() {
      def benchmarkInterceptor = new BenchmarkInterceptor()
      def proxy = ProxyMetaClass.getInstance(A.class)
      proxy.setInterceptor(benchmarkInterceptor)
      proxy.use {
         def a = new A()
         a.a()
         a.b()
      }

      def actual = benchmarkInterceptor.statistic()
      def expected = [['b', 2, 0],['ctor', 1, 0],['a', 1, 0]]
      assert expected == actual
   }
}

class A{
   void a(){ b() }
   void b(){}
}

We test BenchmarkInterceptor, which records all calls to different methods and record number of executions and accumulated time spent inside methods. Then we compare actual results with our expectations.

Here is piece of code, which actually does the job:

def actual = benchmarkInterceptor.statistic()
def expected = [['b', 2, 0],['ctor', 1, 0],['a', 1, 0]]
assert expected == actual

But it was failing with following not very helpful message:
java.lang.AssertionError: Expression: (expected == actual). Values: expected = [[b, 2, 0], [ctor, 1, 0], [a, 1, 0]], actual = [[Ljava.lang.Object;@cc06e0, [Ljava.lang.Object;@d7643a, [Ljava.lang.Object;@f52a1d]

Lesson 1: If you provide diagnostic try to provide useful one.

I had quick look to compiler and immediately found that instead of call to stringBuffer.append(obj) during generation of assert statement it would be much better better to use stringBuffer.append(InvokerHelper.toString(obj)). Fast check and … voila! It works and I have much better error message:

java.lang.AssertionError: Expression: (expected == actual). Values: expected = [[b, 2, 0], [ctor, 1, 0], [a, 1, 0]], actual = [[ctor, 1, 0], [a, 1, 0], [b, 2, 0]]

OK, content of expected and actual lists are the same but the order of elements is wrong. Fortunately, I had this lesson many times before

Lesson 2: Never expect any particular order of elements in a hash map. Or either sort or use linked one if you need an order.

What hash map one can ask? Really, I had no idea but I could imagine that BenchmarkInterceptor use internal one to record calls. And it is was exactly the case – method names as keys and list of current time before and after the call as values.
So the fix was almost trivial – replace HashMap by LinkedHashMap in BenchmarkInterceptor and put expected values in order how calls happen


def expected = [['ctor', 1, 0],['a', 1, 0],['b', 2, 0]]
assert expected == actual

At that point I run clean build, checked that all tests passed and commited my changes. In fact before commit I also run remote build on TeamCity, where test failure was reported from. In two minutes I had beautiful surprise from build server. Test was failing!

It took me another 5 or 6 local runs tio reproduce failure on my environment. Some time execution time was 1 instead of 0.

Lesson 3: Never expect particular execution time of piece of code

Do I need to explain? Probably not.

Thanks to Groovy here is final code snippet.

      def actual = benchmarkInterceptor.statistic().collect{ [ it[0], it[1] ] }
      def expected = [['ctor', 1],['a', 1],['b', 2]]
      assert expected == actual

And it works much better.

Bonus Lesson: Analize your tests we have a lot to learn from our failures


Grails Trainings in Scandinavia

May 21, 2008

We were asked about that very often and now it happened.

G2One, Inc., company behind both Grails and Groovy, and Callista Enterprise AB announce today series of public trainings in Scandinavia.

Göteborg 2008-09-08
Stockholm 2008-09-15
Malmö 2008-09-22

This 3 day course covers both Groovy language and Grails framework. After an introduction to the basics of the Groovy language, Students are taken through a step-by-step lab-driven experience on how to build a solid Grails application learning from the people who created the technology. We will start from basic but will go very deeply.

Full description of course can be found here

BTW, thanks to our friends from JetBrains each student will receive free 12 month license for IntelliJ IDEA

If you are interested to participate in one of these courses please register here or email to us on training at g2one dot com. Group discounts and discounts for students coming from non-Scandinavian countries are available, so we have what to talk about 🙂

And I totally forgot – many other European and North American locations of trainings will be announced very soon.


TeamCity saved my day

May 17, 2008

I had crazy problem this morning – Groovy Core build was failing with StackOverflowException on some Windows configurations but not on Linux or Mac. Unfortunately, it happen after my commit several days ago, so I was somehow responsible for that.

I have only Mac. No Linux. No Windows. So my first idea was to ask someelse to take care for the bug. But you know what… It is somehow not very professional.

Fortunately, I found very simple solution. TeamCity Remote Build. 45 minutes and 15 builds and problem was located. I even was able to create test case, which also failing on my Mac. Really cool.

Now I have more serious problem – to understand how to fix it. But it is kind of more regular work.


invokedynamic: Is It What We Really Need?

May 9, 2008

John Rose wrote interesting post regarding dynamic invocation in the VM. Here are my thoughts as implementor of Groovy language.

DECLAIMER: It is not official opinion of Groovy Development Team or G2One Inc. but only my personal opinion.

First of all I think John and JSR 292 team finally start to do very good job. After long period of uncertainty we have a lot of communication last months about what’s going on and ideas under discussion. It is really helpful.

Finally we have very clear explanation of method handles idea, which sounds very promising. To make long story short it is about avoiding expensive calls to Reflection API, which is one of the biggest problems each dynamic runtime meet. And now we have more or less clear picture (or suggestion) for invokedynamic.

Recent significant progress in JRuby and Groovy runtime performance proved clearly that dynamic languages has a lot of space for improvements in terms of performance. It is also more clear now what kind of support is required from JVM for dynamic language runtime.

In fact, the main challenge for implementor of dynamic language sounds pretty trivial: you have call to the method and in general can assume nothing about type of receiver and types of arguments. So execution of a call consist <i>logically</i> of two important steps:

  • method selection
  • method invocation

IMPORTANT NOTE: Current Groovy MOP doesn’t separate these steps, which also leads to some difficulties. Of course, implicitly (logically) both steps are presented. For Groovy MOP 2.0 we consider possibility to separate them explicitly.

Method invocation in Groovy is expensive. We have to use Reflection API, which does a lot of checks, requires allocating of array for result, boxing/unboxing of arguments and return value and several nested calls before we come to execution of real method. As I said before there is some hope that methods handle will help here. What is especially beautiful in method handles is the fact that they don’t require new bytecode but can be handled directly by JVM.

Method selection in Groovy is even more expensive. Following diagram gives very simplified overview of what’s going on. Let me repeat this: it is simplified overview and real process even more complicated (categories, ExpandoMetaClass etc.)

Fortunately there is solution for the method selection problem: if we store result of previous method selection there is very good chance that next time our choice will be the same and check of this fact is much cheaper compare to real selection process. This technique is implemented in Groovy 1.6 and proved to be very efficient (of course some memory overhead is involved)

Here is pseudo-code of API used by Groovy 1.6

public abstract class CallSite {
    // index of call site
    protected final int index;
    
    // name of method
    public final String name;
    
    // array of all call sites in compiled class
    protected final CallSiteArray array;
   
   // calls to this methods appear in bytecode
   public Object call (Object receiver, Object [] args) {
      retutn acceptCall(receiver,args).invoke(receiver,args);
   }
  
   // check if receiver, arguments, categories in use, etc. fits to memorized choice
   // if so, we can continue to use same call site
   // if not, new call site created to replace current one
   protected CallSite acceptCall (Object receiver, Object [] args) {
      if (checkCall(receiver,args))
         return this;
      else
         return createCallSite(this,receiver,args);
   }  

   // check if receiver, arguments, categories in use, etc. fits to memorized choice
   // depends on type of call site
   protected abstract boolean checkCall (Object receiver, Object [] args);

   // real invocation
   // depends on type of call site
   protected abstract Object invoke (Object receiver, Object [] args);

   // creates new call site, which will replace existing one
   protected final CallSite createCallSite(CallSite base, Object receiver, Object [] args) {
      ....
   }
}

In bytecode receiver.method (arg1, arg2) becomes

callSiteArray[callSiteIndex].call (receiver, new Object[] {arg1, arg2} );

callSiteArray is synthetic local variable filled in the beginning of each method from static soft referenced field. Initially array is filled by dummy call sites, which needed to keep name of target method and index of call site, because we don’t want to send two additional parameters each time and we also don’t want to do null check for call site on each invocation.

So what happen when we visit call site first time is creation of real call site and invocation. On second visit we check if it still to be valid and either recreate or use it.

Truely speaking, API above is not real API. There are special paths reflecting the fact that in Groovy there is several different types of method call but it is out of scope of this note and I hope the idea is clear. Another thing, which makes API much more complex is special paths for binary operations and in the future for calls with small number of parameters, which allow to avoid creating of argument arrays.

So how all that is related to invokedynamic? Very simply. Bootstrap method will call our createCallSite. Target handle will be CallSite.call doing check and either invoking another method handle (really selected method) or setting new target. And voila!

The only thing, which really worries me is do we need special bytecode or Java API (of course specially recognized and optimized by JVM) is enough.

The real beauty of API is that you can use it (backported versions) for old JVMs. Even now a lot of people stay with 1.4 and I am not sure how many already switched to 1.6. For us, as language developers, very important to support old platforms. For sure, to support 1.5. And I would prefer not to have two versions of compiler for platform supporting invokedynamic and for one, which doesn’t.

The only really good side of invokedynamic compare to fixed API is ability to make calls with absolutely any signature. I don’t know how important is that for other languages, for Groovy I think it will be enough to have rich enough fixed API for small number of arguments and Reflection-like general API.

So if I was member of EG, which I am not, I would prefer not to introduce new bytecode.

Dixi 🙂


Joint compilation of Groovy and Java – III

July 4, 2007

Believe you or not but it happened. Now you can compile Groovy and Java together.

Read Jochen Theodorou article on implementation details. And hey… it is based on my patch 🙂


This is the house what Jack built

June 8, 2007

Today I realized what we (or may be just I) live in very strange world. Let me describe why.

I work now on problem of making GORM domain classes accessible in GWT client-side code. It is an interesting task by itself and I plan  to dedicate  several separated posts to it but in general it is very simple. I want to define my server data objects in Groovy and be able to request them in from client-side code without coding too much.

The  solution is almost trivial. GWT provides great idea of generators which can create Java code to be integrated in to compilation of client-side JavaScript, so all what I need is to implement generator, which read groovy classes and generate all additional Java code I need. It is not too compilcated especially with Groovy in your hands.

What I realized debugging my code is how many different compilers, generators and even instrumenters involved in run of my one single test.

  1. Javac. I use it to compile Java part of my generator and the test to be run by JUnit.
  2. Groovyc. Most interesting part of my generator is written on Groovy
  3. GWT compiler from Java to JavaScript. It calls my generator and compile my test and generated Java code. If you curious, it contains built-in parser from Eclipse compiler
  4. BTW, compilation of unit tests in GWT involves generation of client-side part of the test. It’s a long story, which I will probably need to blog separately later. For now we just notice one more generation step.
  5. My generator use Groovy compiler to parse Groovy sources of domain classes. Main task for the generator is  to create client-side twins for domain classes and make them serializable.
  6. Of course, serialization may not happen for free. Fortunately GWT provide default generator, which takes care for generation such code for java source, generated by me.
  7. Again, just for record we should notice that my generator not only written on Groovy but also depends on Groovy runtime, which does some code instrumentation in magical GroovyClassLoader.
  8. There is one more thing, which I am afraid to speak about. GORM uses Hibernate and Spring. How many code generation and instrumentation used inside? God knows.
  9. I know only that even GORM iteself uses some to access Hibernate and Spring.

That’t it. But something is wrong in this world, isn’t it?


Loopy with Groovy

June 7, 2007

There’s been a lot of people asking for a Java loop syntax in groovy. A lot if not almost all usages however are cleaner without it, once you know some idiomatic Groovy.

Marc Palmer’s post blog post demonstrates it almost obviously.