Groovy dynamic stateful mixins

July 9, 2008

My last post was dedicated to Groovy dynamic stateless mixins. Today I want to talk about stateful ones, which are now presented in Groovy trunk.
The main difference is obvious - additionally to providing new method stateful mixins can also keep state.

Here is non-trivial sample. First of all let us use EMC to add convinient method to JDK class ReentrantLock.

ReentrantLock.metaClass {
  withLock { Closure operation ->
     lock ()
     try {
        operation ()
     }
     finally {
       unlock ()
     }
  }
}

Now we are able to implement Groovy ConcurrentQueue like this


class ConcurrentQueue {
   private LinkedList list = new LinkedList ()
   private ReentrantLock lock = new ReentrantLock ()

   def get () {
      lock.withLock () {
        list.removeFirst ()
      }
   }

   void put (def obj) {
      lock.withLock () {
         list.addLast (obj)
      }
   }
}

So far, we didn’t use any mixins. Let us rewrite last sample using ones


class ConcurrentQueue {
   static {
     ConcurrentQueue.metaClass.mixin LinkedList, ReentrantLock
   }

   def get () {
      withLock {
        removeFirst ()
      }
   }

   void put (def obj) {
      withLock {
         addLast (obj)
      }
   }
}

What we did is we replace private fields with mixins, which we assign in static initialization of the class. Of course, this sample is a little bit artificial because what we do is emulating compile time mixins with static initializer. I will discuss compile time mixins separately but now let us modify our sample to be more dynamic.


        def queue = new Object ()
        queue.metaClass {
            mixin LinkedList, ReentrantLock

            get { ->
               withLock {
                 removeFirst ()
               }
            }

            put { obj ->
               withLock {
                  addLast (obj)
               }
            }
        }

Instead of creating special class for our concurrent queue we use combination of mixin and per instance meta class to create what we need. Isn’t it lovely?

Now you may wish to ask me several interesting question, which I try to answer below.

How to access instance of mixed-in object if needed? Well, it is very easy. Let us add one more method to implementation of our queue, which will demonstrate how to do that.


        queue.metaClass {
            duplicateEachElement {
               withLock {
                  LinkedList newList = new LinkedList ()
                  mixedIn[LinkedList].each {
                     newList << it
                     newList << it
                  }
                  mixedIn[LinkedList] = newList
               }
            }
        }

How to access owner object from mixed-in instance if needed? This is a little bit more tricky. To understand why is it so we need to understand how mixins are implemented. Obviously, where is no magic here - mixed in references kept separately from objects in weakly referenced map. It means that if mixed in instance keeps reference to owner, then owner has zero chances to be collected. So in the perfect world mixed in instance should never keep reference for owner. But in pragmatic world it might make sense to do it time to time. Here is example how to do that


class NoDuplicateCollection {
    void put (def obj) {
        def clone = (mixinOwner as Collection).find {
            it == obj
        }

        if (!clone)
          mixinOwner.add obj
    }
}

def list = new Object ()
list.metaClass.mixin NoDuplicateCollection, LinkedList

list.put 1
list.put 1
list.put 2
list.put 2
list.put 3
list.put 3

assertEquals (3, list.size ())
assertEquals 1, list [0]
assertEquals 2, list [1]
assertEquals 3, list [2]

NoDuplicateCollection class designed to be used with any JDK Collection as owner. In example above we use it with LinkedList but any other Set or List will work as well. Property ‘mixinOwner’ is magically (via delegating per instance meta class) inserted in to instance of NoDuplicateCollection and… Voila! Of course, internally mixedOwner is week reference so it will not prevent owner from being collected.

One more interesting thing to notice is ‘as operator’ used to cast our object to Collection. Yes, it is true - we cast instance of Object to Collection. Again, there is no magic here - if our object can’t be casted itself we try mixed-ins. Simple as that.

Enjoy it in the Groovy 1.6 trunk


Groovy dynamic stateless mixins

June 7, 2008

Last week I wrote about per-instance meta-classes and ExpandoMetaClass DSL. At that point it was just prototyped code in my working copy and I was not even sure if it will come to Groovy 1.6 trunk. Now it is there. If you didn’t read this previous post I recommend to have quick look on it as it might help to understand current one.

Today I want to talk about another part of the same commit, which is kind of side-effect of ability to have per-instance meta classes. Now we able to have per instance dynamic stateless mixins.

But let me start we explanation of what is mixin for me.

Mixin is either static or dynamic way to extend class or instance behavior by adding methods and/or state defined in another class.

Static here means defined at compile time and dynamic means defined on runtime. This two types of mixins are totally different.

  • Static mixins applied to class but dynamic one to meta class (either class one or instance one)
  • Static mixins always effect hierarchy, dynamic ones may or may not effect it depending on nature of meta class under effect (if meta class is the one associated with the class it does, if not it does not
  • Static mixins always effect type information, dynamic ones never does that.

Stateless or stateless mixins means ability of mixins to extend object behavior not only with methods but with state as well. Stateful compile time mixins can be understand as kind of multiple inheritance.

Groovy has no compile time mixins yet. I think Groovy has to have something similar to Scala traits but it is subject to different discussion.

Groovy has no stateful mixins yet as well and below I will talk a bit about my idea how they should look like. So main subject for today is dynamic stateless mixins

Let me show simple example. First of all we define usual category class, which will be used to mix in.

class DeepFlattenToCategory {
    static Set flattenTo(element) {
        LinkedHashSet set = new LinkedHashSet()
        element.flattenTo(set)
        return set
    }
    // Object - put to result set
    static void flattenTo(element, Set addTo) {
        addTo <
           element.flattenTo(set);
        }
    }
    // Map - flatten each value
    static void flattenTo(Map elements, Set addTo) {
       elements.values().flattenTo(addTo);
    }
    // Array - flatten each element
    static void flattenTo(Object [] elements, Set addTo) {
        elements.each { element ->
           element.flattenTo(set);
        }
    }
}

This category does pretty obvious thing - deep flattening of complex data structure as demonstrated by following code snippet.


        // mixin Category to meta class
        Object.metaClass.mixin DeepFlattenToCategory
        // and here we are
        assertEquals ([8,9,3,2,1,4], [[8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

Careful reader may notice probably that we modify only meta class for Object but methods for Collection, Object [] and Map becomes available as well. Such behavior is usual for categories but a bit uncommon for ExpandMetaClass - in the past he had to call ExpandoMetaClass.enableGlobally () to ask meta classes to look for missed methods. Such approach had several downsides - to name a few: unnecessary memory footprint for creation ExpandoMetaClass for each class (both modified and non-modified) and lack of “good place” where to call ExpandoMetaClass.enableGlobally (). It means that if you developed library, which requires it - you should call it somewhere, because nobody can guarantee that user application will do that. Fortunately, now ExpandoMetaClass can define methods for sub classes and if sub class missed some method he will try to find it in hierarchy. Obviously such behavior is must for mixins - otherwise our sample will look much less nicer(which is still to be valid code but creating four ExpandoMetaClasses instead of one)


        // mixin Category to meta classes
        Object.metaClass.mixin DeepFlattenToCategory
        Collection.metaClass.mixin DeepFlattenToCategory
        Object[].metaClass.mixin DeepFlattenToCategory
        Map.metaClass.mixin DeepFlattenToCategory
        // and here we are
        assertEquals ([8,9,3,2,1,4], [[8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)


        Object.metaClass.foo(ArrayList){ -> .... }

BTW, ExpandoMetaClass has new syntax to define methods for subclasses


        Object.metaClass.foo(ArrayList){ -> .... }

OR


        Object.metaClass {
           foo(ArrayList){ -> .... {
        }

OR


        Object.metaClass {
           define(ArrayList){
              foo{ -> .... }
           }
        }

OK, back to mixins. So far, it seems very similar to normal categories, so let us make our sample more interesting


        class NoFlattenArrayListCategory {
           // ArrayList - put to result set
           static void flattenTo(ArrayList element, Set addTo) {
              addTo << element;
           }
        }

        def x = [2,3]
        x.metaClass.mixin NoFlattenArrayListCategory
        assertEquals ([x, 8,9,3,2,1,4], [x, [8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

What we do here is create category, which does not flatten array lists and mix it in instance meta class. Voila!
Of course the same can be achieved with ExpandoMetaClass without mixing in category


        def x = [2,3]
        x.metaClass.flattenTo = { Set addTo -> addTo << delegate }
        assertEquals ([x, 8,9,3,2,1,4], [x, [8,9] as Object [], [3,2,[2:1,3:4]],[2,3]].flattenTo () as List)

but please remember that category can be easily implemented in Java and provide much better performance compare to closure-based meta methods. And now we can use it as mixins.

The last thing I want to talk about is some features, which is missing and which I would love to see in Groovy. All below is just imagination and I don’t know exact implementation details and underwater stones but here are problems to be addressed.

  • Stateful mixins
  • Mixing in objects
  • Better syntax to define categories in Groovy
  • Compile time mixins

Stateful mixins

Imagine that category class defines not only static methods but also some fields and these fields magically becomes available as properties when we mix these category in. Probably if object already has such properties the one defined by category should be ignored, but if not… we have extremely powerful tool.

The main technical problem here is that we don’t want to define these new properties via ExpandoMetaClass, which will be very slow, but if we created instance of category class to associated it with original object… well, we have no good place to store this association. For Groovy compiled objects we of course can store refernence(s) directly in the object and for Java one in WeakHashMap. But what will happen if this mixed in instance keeps strong reference for original Java object… It will never be collected. Believe me, it will happen very often. So far I don’t know any good solution for that.

Mixing in objects

If we had solution for stateful mixin most probably it will lead us to ability to mix in not only category class but any object. I can imagine fantastic possibilities for that.

Better syntax to define categories in Groovy

Well static methods with additional parameter ’self’ are so ugly. We definitely need something better. So far we have @Category annotation, which solves only part of the problem.

Compile time mixin

Having stateful mixins with good syntax to define category methods will lead us to ability mixin category classes on runtime. As result we will have probably the strongest mixin story between all dynamic languages and as powerful as Scala traits in the world of staticly compiled ones.

So it goes.


Groovy per instance meta classes and ExpandoMetaClass DSL

June 2, 2008

So far Groovy allow per instance meta classes only for objects, which implements GroovyObject interface (like any class defined in Groovy). It can be illustrated by following simple test case:

   // for java.lang.String
   "test string".metaClass.toString = { "modified string" }
   assertEquals( "modified string", "test string".toString() )
   // but any other string will have the same behavior
   assertEquals( "modified string", "any other string".toString() )

At the same time

   // for GroovyObject
   class MyBean {
      String toString () { "bean" }
   }

   def bean = new MyBean ()
   def emc = new ExpandoMetaClass(MyBean,false,true)
   emc.toString = { -> "modified bean" }
   bean.metaClass = emc
   assertEquals("modified bean", bean.toString())
   // but any other bean will keep default behavior
   assertEquals("bean", new MyBean().toString())
   // if we want to modify behavior of all MyBean objects
   MyBean.metaClass = emc
   assertEquals("modified bean", new MyBean().toString())

Notice that we use different technique to add methods to Java and Groovy objects. As getMetaClass () for Java object always returns the same meta class we can use .metaClass notation, each GroovyObject has it is own meta class (of course, it is the same object by default) so we act differently depending on what we want to modify (whole class or just one Object)

Now I have patch which solves all these problems.

First of all we can assigned per instance meta class for any object. Of course, for Java objects it kept in WeakHashMap which involves some performance penalties. But now our first sample looks much more natural

   // for java.lang.String
   "test string".metaClass.toString = { "modified string" }
   assertEquals( "modified string", "test string".toString() )
   // but any other string will have natural behavior
   assertEquals( "any other string", "any other string".toString() )

Groovy objects also has benefits - we don’t need to create ExpandoMetaClass manually.


   def bean = new MyBean ()
   bean.metaClass.toString = { -> "modified bean" }
   assertEquals("modified bean", bean.toString())
   // but any other bean will keep default behavior
   assertEquals("bean", new MyBean().toString())
   // if we want to modify behavior of all MyBean objects
   MyBean.metaClass = bean.metaClass
   assertEquals("modified bean", new MyBean().toString())

But now, when we have unified behavior for Java and Groovy objects we can use simply and powerful ExpandoMetaClass DSL. Here is example


        // add behavior to meta class for Integer
        Integer.metaClass {
           // block of static methods
           'static' {
                fib { Number n ->
                    n.fib ()
                }
                // we can use both = or << if needed
                unusedStatic << { -> }
            }

            // one more syntax for static methods
            static.unusedStatic2 = { -> }

            // property definition
            ZERO = 0

            // method definition
            fib { ->
                def n = delegate
                if (n == ZERO)
                  return 1;
                if (n == 1)
                  return 1
                else
                  return fib(n-1) + fib(n-2)
            }

            // of course we can use both = or << if needed as well
            unusedMethod << { -> }
        }
        // and now we have it
        assertEquals( 3, 3.fib())

        // but we can also modify meta class for particular number
        4.metaClass {
            fib { ->
                10
            }
        }

        // and it works!
        assertEquals( 13, Integer.fib(5))

You know what… I like it. And thanks a lot for Graeme for inspirations and advices while I worked on that. Hope it will be accepted well and commited to the trunk.


Joint compilation of Groovy and Java - III

July 4, 2007

Believe you or not but it happened. Now you can compile Groovy and Java together.

Read Jochen Theodorou article on implementation details. And hey… it is based on my patch :)


Joint compilation of Groovy and Java-II : The Quest for Solution

June 6, 2007

In my previous post I promised to describe how to compile Groovy and Java together without writing complicated ant scripts or limiting yourself in design decisions.

Let us analyze the problem and probably it will lead us to a solution:

  1. To compile Java code, which refers some Groovy code, we have to compile Groovy first because the only way for javac to know about Groovy is .class file
  2. We can’t compile Groovy code if it has to know about Java one because again the only way for groovyc to know about Java code is .class file form.
  3. But do we really need as much? Obviously, not. We don’t really need compiled Groovy classes during Java compilation. All we need to know are “exported interfaces” of the classes. What are their names, methods, fields, implemented interfaces, superclass etc. We don’t care about real implementation at all. Let me repeat it is once more. We don’t need it at all.
  4. So if we are able to supply javac with .class files with the same “exported interfaces” as our Groovy classes we are done. After that we are able to compile Java code and provide compiled code to groovyc. And our circular dependencies will be resolved.
  5. It is cool, isn’t it? All we need to do is to write simple compiler which will take Groovy code and create .class file with the same “exported interface” as our Groovy code but ignoring all implementation logic, possible errors etc.
  6. But how does it help us with “ant script problem”. Unfortunately, it doesn’t because we need to call this special compiler.
  7. Wait a minute. Why do we need additional compiler? We can build it in groovyc itself. It will be one task to call for joint compilation of Groovy and Java code, which will do three passes internally. First pass will generate dummy .class files, usually such things called stubs. The second one will call regular Java compiler, for example javac or eclipse compiler if we like. And the final step will do usual Groovy compilation. And it is exactly what we need, isn’t it? Yes, it is exactly what we were looking for.
  8. So the only problem we have is to patch groovyc. We know how to parse and even compile Groovy sources. We know how to call external compiler. We have great ASM library, which will help us to generate stubs. We just need to put it all together. Not too complicated. Right?
  9. OK, there is one more problem, which still worry me. I know that ASM is great library but I have no too much experience with it and it may take long time to code and especially debug this code. Sounds like a problem, right? Really not at all! We don’t need compiled classes. What we need is to provide javac with information about “exported interfaces” of our Groovy code. It can be .class files or … Java source. Let us just generate Java sources with exactly the same “exported interfaces” as our Groovy code. It is really easy task if you have parser for Groovy, which we obviously have, and what can be easier than such generator? “Hell, World!” probably.
  10. We have last thing to improve. Currently we plan to parse Groovy code twice - once for stubs generation and once more during Groovy compilation. It is not dramatically bad but unpleasant. Fortunately, it is easy to resolve. What we need is to incorporate our patch in the middle of Groovy compilation process. Parse Groovy, generate stubs, compile java and stubs and continue Groovy compilation. What is even more fortunately is that Groovy compiler is modular enough to allow it.

What’s it.  We found solution for all our problems and our quest completed. There is interesting story about different technical details and difficulties, which I met implementing this approach, but I will describe it in the later posts.

And wait a second. The patch gonna be incorporated in to Groovy trunk in nearest days.


Joint compilation of Groovy and Java - I

June 5, 2007

The first thing I’ve learned about Groovy was that you have to compile it separately from Java. OK, it is understandable because the languages are different.

But the next news was even worse. There is no way to compile it correctly if there are dependencies between Groovy and Java code. Again, of course we are software engineers, everything is possible(c) and there is a such possibility but…

Let me give you a sample. Imagine we have

  • Java interface JInterface
  • Groovy class GClass implementing JInterface
  • Java class JClass, which has to create instance of GClass (probably not thinking too much is it Groovy or Java class) and use it as instance of JInterface

I believe  everybody will agree that such design may be very logical in some situations. But how can we compile this code? Well, we do it in three steps of ant script

  • javac for JInterface
  • groovyc for GClass
  • javac for JClass

And here we are. Our problem resolved. May be the solution is not so elegant (carefully written ant script instead of IDE compilation) but it works. And it is what Groovy coders do every day.

But does the solution above really work? No, unfortunately it doesn’t!

Imagine that by some reasons JInterface has to know about GClass. We can argue long hours is it good design or bad one, so let just think it has to. Now we have circular dependencies between Groovy and Java code. To compile Java we need to compile Groovy first but to compile Groovy we need to compile Groovy. Ooops! We have dead lock.

Obviously, it is not new problem. Have a look on issue http://jira.codehaus.org/browse/GROOVY-1735 The problem is nobody knows how to solve it.

Again, we can refactor our design. Probably introduce some additional class. May be add additional step to our carefully written ant script and finally being able to compile our simple code. But does it make us more productive? I am not sure.

Fortunately I have some good news. There is a solution for joint compilation of Groovy and Java. Some nice trick was invented by guys from JetBrains And I can say even more - the’ve submitted patch, which probably will be accepted before Groovy 1.1 Beta 2 is out.

But I will describe this story tomorrow. Stay tuned :)


A groovy use case

June 5, 2007

Luc Dewavrin recently discovered Spring’s ability to declare beans that are script-based. He describes why this is great and helpful to to speed-up development. Sample declarations and scripts are also provided.

The article ends with interesting opinion “…, I like both Groovy and JRuby but now tend to prefer Groovy because it leverages my knowledge of Java API. With Ruby i spend too much time finding the reference web site how to achieve things that i do almost naturally in Java. OTH, Ruby has its core API use closures. Closures are tightly integrated and tend to favor loose coupling.”

Read full story…


Using Spring Factories with Groovy

June 5, 2007

Spring is an open-source framework created to address the complexity of Java enterprise application development. One of Spring’s goals is to help developers write simple, testable and loosely coupled systems while reducing the amount of scaffolding code required. In this respect, Groovy has a common goal. So, for simple systems, Groovy alone may be sufficient for your needs. However, as your system grows in size and complexity, and especially in hybrid Java/Groovy environments, you might find Spring’s facilities provide great value to your Groovy system development.

Read more in Groovy documentation…