Beware of this: play framework’s cascaded save() only works on loaded collections

I am currently developing a web application using the really excellent play framework. If you are a web developer with some Java background I strongly encourage you to give it a try.

However, play has taken a little different or let’s better call it an additional approach to control object synchronizations with the underlying database. Whereas JPA manages a loaded object graph and automatically persists any changes upon transaction completion, play has changed this behavior (“Explicit save”) to give the beginner more control over what is going on with the objects. This extension turns the implicit save mechanism into an explicit one where the developer is in control to call save() on the changed objects. This save() will in turn cascade (if for instance the CascadeType.ALL annotation option is present) to the reachable objects.

This might in the first place seem clever because you get control back but I have mixed feelings about it because there are cases where it does just not work as easily as expected.

As an example, imagine a 3-level parent-child object model.

@Entity
public class Item extends Model {

 @OneToMany(cascade = CascadeType.ALL, orphanRemoval=true, mappedBy="item")
       @OrderBy("createdAt")
 public List<Position> positions = new ArrayList<Position>();
}

@Entity
public class Position extends Model {

 public Date createdAt = new Date();
 public int quantity;

 @ManyToOne public Item item;

 @OneToMany(cascade = CascadeType.ALL, orphanRemoval=true, mappedBy="position")
 @OrderBy("createdAt")
 public List<Order> orders = new ArrayList<Order>();
}

@Entity @Table(name="Ordr")
public class Order extends Model { 
 public Date createdAt = new Date();
 public int quantity;

 @ManyToOne public Position position;
}

So far nothing complicated. But now imagine you are loading (for performance reasons) objects from the lowest level (Order) and manipulate the parent object (i.e., the Position).

public void dostuff(Item item) {

 // load orders according to some important criteria
 List<Order> orders = Order.find("position.item = ? order by createdAt", item).fetch();

 // manipulate the parent of the order (the Position)
 Order order = extractImportantOne(orders);
 order.position.quantity += 200;

 item.save();
}

Intuitively I would assume that the manipulated Position object is updated in the database through the Item.positions cascade settings. But this is not the case here! The reason is that the collection Item.positions has not been populated from the database. Thus, play sees an unintialized PersistentCollection and will simply ignore to cascade.

One solution to the problem would be to touch the cascading collection:

 item.positions.size();
 item.save();

This will make sure the ‘positions’ collection will be considered during the following call to save(). With pure JPA on the other hand, such a call to do the trick would never be necessary and also the call to save() could be avoided, because JPA knows all loaded and changed objects and automatically updates the database.

This is why I have very mixed feelings about play’s extended persistence API. It works well in simple cases but for more complex models it might be better to switch back to plain JPA mode.

Advertisements

Why I completely dropped Qt and QtWebKit from my Hybrid App

In my recent articles I described certain aspects related to the use of QtWebKit in a hybrid C++ application. In the meantime, it turned out that Qt and QtWebKit are not such a good bet as I thought they would be at the beginning. Here’s why:

Apple

First and foremost, the most interesting question in a hybrid app written in C++ is: where do you want it to run? In my case, it is on Mac OS X in the first place, then on Windows and very likely also on Apple’s iPhone or iPad. However, this puts me in a situation where I have to decide which subset of these platforms I support primarily and which ones secondarily as there is no such thing as a common layer to access all of them from a single code base. It is either a combination of Mac OS X and Windows via the Qt port or a combination of Mac OS X and iPad/iPhone that can go as the primary target.

For a certain time, I concentrated on the first combination as shown above while hoping that time works for me and somebody ports Qt to the iPhone, and there already is a project for that at http://www.qt-iphone.com/Introduction.html. But, in the meantime, Apple introduced iPhone OS 4.0 and *bam* there will never be such a thing, because http://daringfireball.net/2010/04/iphone_agreement_bans_flash_compiler “and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited)”, a rule introduced by Apple which is actually targeted against Adobe and its Flash technology but will also affect Qt, especially because that comes from Nokia, another one of Apple’s (future arch) rivals. This means that Qt can never be the common layer on all of my targeted platforms. Given that, and the fact I target Mac OS X first, Windows second, and that I also tend to shift Windows one step further down and promote the iPad version instead which completely changes the porting picture:

This looks much cleaner now as it provides unified, access to all Apple platforms. For Windows and Android Google Chromium is a good choice.

But what about Qt in general?

Qt / Nokia

The first part was the rational part, here comes the emotional one:

In the recent weeks I had enough time to dive into the details of Qt in all aspects. Its visual quality, its API quality, its implementation quality and the quality and structure of their development process. My conclusion from that is that it is the wrong way of doing things as evaluated from a broader perspective.

What is very important in a visual app is visual quality. Qt does a good job in replicating functionality of controls on each addressed platform. But, it can never, by its nature, take care about the special controls certain platforms offer because then it wouldn’t be cross-platform any more. Qt certainly has to focus on the lowest common denominator across all supported platforms. By driving the GUI via Qt controls, an application loses the capability to provide the eye-candy to the platform user like he is used to. The most prominent example of this is Google Maps which is based on Qt and which totally lacks platform L&F on the Mac up to a degree where it certainly loses style. In Google Maps this is not so important as interaction with the Earth requires a special user interface anyway and the rest of the app is unimportant, but it shows how difficult it is even for Google to provide attractive visual quality on the platform. Gaining access to the platform specialities is difficult as everything is hidden by Qt in private implementations. You have no chance to do “that special thing”.

Qt has a comprehensive “fat” API that consumes everything from all platforms and which has to take care about the specialities on all platforms, which is of course its nature but which is also very difficult to maintain and to improve over years. Development of Qt is distributed across Europe with Nokia focussing on minimizing cost not maximizing quality. Their development centers are mostly in cheaper parts of Europe and they try to cooperate with educational institutions to bring the costs down. This might be a legacy from Trolltech but is still in place and shows that quality is not their first concern.

So far I have touched Qt releases 4.5, 4.6 and 4.7 and came across older API documentations. From there I can tell that many things they have done are not very well and strategically thought out. They introduced too many APIs doing similar things and deprecating previous ones (QML, QtQuick, Scripting with QtScript and in QtWebKit, Qt3 vs. Qt4, QGraphicsScene, QItemView, QtDesigner, QtCreator). To me this is too much feature driven and dictated by the progress of other market players. Then I saw these super new technologies contrasted by very basic demo videos in YouTube showing excitement over very basic features which is not very professional – you can check out their channel on http://www.youtube.com/user/QtStudios to get an idea. What I want as a consumer of platform technology building a commercial grade application is that the selected platform is very stable and well thought out. But there is just too much noise coming from Qt and to make matters worse, none of the new features really feels final, it feels more like work in progress and you can never know how long something survives. Last not least, one day down the road Nokia might become a rival again to Apple just as relevant as Google now is. Economically speaking, Nokia already is, but Apple sees Google as its main competitor, not Microsoft and not Nokia. They are just not in the same technology league any more. Apple and Google have the brains now.

Don’t get me wrong, the Qt guys are doing a good job and they surely do it with a lot of enthusiasm, all I want to say is that Qt is just not the best bet in a commercial hybrid application. That is why I gave up on it.

Accessing the Original WebKit API in QtWebKit Hybrid Apps

Everybody wraps WebKit

For each and every port of WebKit, be it Safari, Chrome, QtWebkit, there is always a wrapper around the WebKit object model. These wrappers function as language mappers, e.g., in case of Safari C++ API is mapped to Objective-C, and as a simplification layer around the complex WebKit API. The first case makes total sense, as no developer familiar with Cocoa and Objective-C will like to take an excursion (back) to C++. On the other hand, the Chrome port as well as the Qt port expose a C++ API towards their developers and completely hide the WebKit API despite that this is also already written in C++. Within the C++ universe, I do not understand why the original WebKit API is such a bad bet and must be hidden under all circumstances (I derive this from the fact that everybody hides it)? Is it really that evil?

WebKit API is very stable as it is a reflection of HTML and XML and all the things used in the browser for years and which do rarely change, so a dependency should be acceptable. The only benefit is that the Qt developer works solely with the Qt API or the Chromium developer with a Chromium API. This is ok in case of very simple integration with WebKit, like displaying some external web content in an otherwise traditional application or when implementing a browser. But not in a hybrid application! In a hybrid application I want much more control over the internals of the browser. I do not consider it evil. I consider it as a very nice work horse doing most of the rendering and layout work. I want to reference DOM elements from my application and I want the DOM to interact with my application. I want to replace the excessive use of JavaScript found in Web 2.0 apps with excessive C++ – I do not want JavaScript in a hybrid app, I want C++ only!

In case of QtWebKit, how hard is to get access to the original WebKit API? It is very hard, as the necessary headers are all eliminated from the QtWebKit API and all WebKit symbols are local in the compiled DLL or OS X framework However, I have taken the pain and hacked me into it. The steps required to expose the WebCore API via QtWebKit are:

Get the source tree from Qt git

You will need to build from source. So, get it from http://qt.gitorious.org/qt and switch to the desired branch or tag.

Configure Qt just as normal

configure it but do not run make now!

Adjust the compiler flags for WebCore

run the following commands in the source tree

sed -i -e 's/-fvisibility=hidden//g' src/3rdparty/webkit/{Web,JavaScript}Core/Makefile.*
sed -i -e 's/-fvisibility-inlines-hidden//g' src/3rdparty/webkit/{Web,JavaScript}Core/Makefile.*

(or change these the Makfiles manually) This will compile WebCore in way to expose all internal symbols later in the DLL or framework.

Gain access to the WebKit API

In src/3rdparty/webkit/WebKit/qt/Api/qwebelement.h make QWebElement::m_element public

This is just one way to do it (see later for an alternative)

Build Qt

Now run the normal make; make install

Add WebKit header files and defines to your .pro file

The application code needs to include the WebCore header files.

DEFINES += QT_SHARED ...
release:DEFINES += NDEBUG
INCLUDEPATH += /Users/andre/src/qt/git/src/3rdparty/webkit/WebCore/bridge/qt \
    /Users/andre/src/qt/git/src/3rdparty/webkit/WebCore/page/qt \
    ...

This is a very shortened version to save space here. See the full set of defines in this file: myapp.pri

The order is important. Replace /Users/andre/src/qt/git with the home of your Qt source tree. Some defines are very important and some might be just optional, I just took all of them from the effective compile command when Qt was built.

Include WebCore headers in application code

#include <WebCore/html/HTMLElement.h>
#include <WebCore/platform/text/PlatformString.h>
#include <WebCore/platform/text/CString.h>
#include <WebCore/svg/SVGElement.h>
...

Alternative access to QWebElement::m_element

Instead of changing the visibility of the “m_element” member from private to public, it would also be possible to use a fake subclass in the application code with a public member m_element like here:

class HackWebElement: public QWebElement {
public:
WebCore::HTMLElement *m_element;
};

and then gain access to it by down-casting a QWebElement to a HackWebElement:

QWebElement webElement = ...;
HackWebElement *hwe = (HackWebElement*)&webElement;
// now we can access hwe->m_element

You can also use your preferred method to access a private member in C++ 😉

Using the WebKit API

Now I can do fancy stuff with the WebKit API. Here is an example where I create an SVG element programmatically, i.e., without the need to have WebKit parse some HTML (or XML in this case):

WebCore::QualifiedName svg("svg", "svg", "http://www.w3.org/2000/svg");
RefPtr<WebCore::SVGElement> nel = WebCore::SVGElement::create(svg, hel->document());
WebCore::String s = nel->tagName();
debug(string(s.utf8().data()));
if(nel) {
    debug("it's a WebCore::SVGElement!!!");
}

Conclusion

So far, I have only gained initial access to the internal browser DOM as a first step. This way I can create DOM elements programmatically. In the future, I will experiment with event handlers on DOM objects written in C++ in order handle all UI events in the C++ part of my application. This will reduce the browser part to act as a pure layout and rendering engine which I think should be its sole role in a hybrid application.

Regarding the little “hacky” approach I can say that there is only one place on the whole application code where access to WebKit is gained. This can be in the document, in the frame, or like here in the general element. From then on, no further hacks are required, it is just used and QtWebKit is not needed any more.

I would also wish that in ports like QtWebKit the WebCore API is preserved and exposed to the application developer. I think there are 2 aspects in a WebKit port: One is the physical rendering and display and the other one is the kind how WebKit embeds into the application and is access. Both concerns should be considered separately and I want to be free to consume only the first one and live with the original in the latter one.

Advanced Object Lifecycle in OO Systems applied to implementations in C++

C++ certainly provides sophisticated mechanisms to create and destroy objects via it’s constructors and destructors. There are, however, certain aspects to these which make them hardly usable in advanced OO systems. First and foremost, a C++ constructor implementation lacks one very important feature: polymorphic calls! In a constructor, a call to a virtual function of the object constructed, is not a polymorphic call but a call to the method as overridden at the level of the class currently constructing the object! This makes it impossible to give the more abstract constructor information from the more concrete class. Imagine you have a UI element that need size information during construction, but that size information can only be provided by it’s specific subclasses, then you will need to implement a 2-phased approach with construction first and using (displaying in this case) second.

C++ constructors also lack in contrast for instance to Java the possibility to call another constructor from the same class. You can only call constructors from super classes. This makes it harder to initialize a large number of members because the initializers have to be implemented redundantly in every constructor. Without a code generator this is very error prone.

There is another problem which is the rather weird logic of which members get initialized and which do not. If you don’t have a constructor which initializes pointers, then these pointers are not initialized to 0. However, if a class uses some struct as a member and this struct has a pointer as member, then this pointer is initialized to 0.

So, a general approach to get over all these problems would be to not use constructors at all and delegate all responsibilities to some generic virtual methods. Let’s first look at the requirements to an OO lifecycle in general:

Creation

  • Create all mandatory child objects upon creation of the parent
  • If an instance requiring a parent element is created, the parent is already present

Transfer Ownership

  • Take a partial tree out of an object tree and connect it to a different object tree (reparenting)

Deletion

  • Isolate a subtree and destroy it

These requirements can be translated into a general lifecycle model in OO systems:

Objects are, once allocated via operator new() in an intermediate state “allocated”, where they can be attached to a parent object, i.e., the parent becomes known to the child and can be used in the subsequent init() call which initializes the object structure in it’s depth. Instead of deleting an object via operator delete() it is destroyed by the destroy() method. In the destroy method, all child objects get destroyed and all destroyed objects are returned to the object pool as isolated instances. Only the pool deletes these instances if needed. Instances are never created via operator new() directly, they are always obtained from the pool via the factory method instance() of the class. This factory method creates a new instance if none is found to be recycled from the pool. Both the default constructor and the destructor are declared as private to ensure the sole usage of the factory method.

This would further lead to the 2 virtual methods init() and destroy() to be implemented on each class level. These methods would have the following behavioral characteristics:

init()

  • is called only after business parent has been set
  • is the constructor equivalent
  • init() calls super::init() first, just like a constructor
  • init() initializes members of the class
  • init() can be called repeatedly
  • memory allocation is a separated concern
  • init() is virtual
  • init() can call virtual methods

destroy()

  • is called before business parent is removed
  • is the destructor equivalent
  • destroy() first resets own members
  • destroy() calls super::destroy() second like a destructor
  • destroy() can be called repeatedly
  • memory deallocation is a separated concern
  • destroy() is virtual
  • destroy() can call virtual methods

Conclusion

  • Constructors/destructors are not needed any more, except for memory allocation
  • Instances are created in the pool only (i.e., in the class)
  • Objects have either a business parent or are referenced by the pool during destroy()
  • objects are returned to the pool
  • if objects do not have a business parent, then a parent is also not needed for initialization

These feature improve the handling of complex object structures in a meta-driven OO environment applied to C++. In fact, they are so generic that they can be easily adapted in a completely different programming and runtime environment. There are, of course, some drawbacks which is basically the more complex protocol for initialization and the implementation of the object pool.

Embedding 2D Drawing Objects in QWebView

One question not so easily answered is how to display high-frequently updated dynamic content in a QWebView. QWebView is a great tool to build hybrid web applications in C++. But because it is basically only a port of WebKit and Qt tries to wrap and encapsulate as much as possible, it is also only a plain web browser, just capable of displaying content that comes in HTML form. What if you want to display content within a web page like a stock chart or even a video…?

The options WebKit browsers offer are basically:

  • <object>, , <applet>: they each require you to implement a specific API and in case of applet even a different programming language
  • html 5.0 canvas: this is interesting, however, it requires JavaScript to program against it which might be to slow due to frequent parsing; it would be much more interesting if a QWebElement style wrapper would be available for drawing directly from C++
  • http://code.google.com/p/nativeclient/ is very interesting in normal browsers, but not in a hybrid application, as it is just another form of the <object> tag

This is why I have been thinking about a different approach, which is closer to the machine’s (i.e., Qt’s) drawing and came up with this idea:

Here we place a special, empty <div> in the document which can later be accessed by it’s ID using a QWebElement. We overwrite the QWebView::render() method and check if the region overlaps our dynamic content as specified by the QWebElement::geometry() data. The actual dynamic content can for instance be pre-rendered into an internal buffer, which we can use later as the source of a pixel copy operation towards the QWebKit paint device. Whenever our dynamic content changes, we can force a repaint via QWebView::repaint() exactly on the region that has changed.

Architecture-wise this just means that the main presentation layer of the hybrid app is not accessed in an outbound fashion only. I have no problem with that, as long as the dynamic content can be displayed and the module doing that can later be replaced with some less “hacked” approach…

03/05/2010 Update: HTML 5.0 Canvas is working in QWebView. I don’t know to what exact extent but some initial drawing tests completed successfully.

Alternative Hybrid Application Architecture

In my recent article (Not So Easy) Hybrid Applications with Qt, WebKit and JavaScriptCore I reflected my opinion on the problems associated with the current Qt scripting APIs QtScript and WebKit Script. I’d like to depict the differences between both again:

QtScript and QtWebKit both use the same JavaScriptCore but from different code copies and by wrapping them both with completely different APIs. In the meantime, I have also crosscheck current status at Qt and Nokia’s scripting plans. There are two separate issue tracking databases, one at Nokia for Qt development in general and one at webkit.org tracking the Qt WebKit port progress. Here are the 2 relevant entries:

I left my comment on the first one as I am still afraid that the Qt developers might come up with even a 3rd API. If that 3rd API is a unified API which can be used for QtSCript as well as in QWebKit scripting then it’d be fine. Given the comment I received from Kent on the first one, I hope for the best.

Alternatives
So far on that, but I want to focus now on the alternatives to the QWebKit based approach to building hybrid – i.e., Web enabled – applications in C++. First of all, WebKit in itself is an outstanding HTML rendering engine, both in performance and portability, both of which features I wouldn’t want to lose. That’s why I seek a solution based on WebKit, which could look like in this diagram:

The first and most important change is that whilst WebKit is still in it, it is not wrapped by Qt any more. Using the WebKit API directly, I would have access to the complete publicly available functionality in WebKit via e.g. HTMLElement and HTMLFrame classes. This would be to a great advantage. On the con side, I would lose Qt’s additional features from the wrapper classes, but I think they weren’t that many so that should be no big deal. Most of the customization would have been done using overridden event handling functions of the QWebView anyway and not via signals/slots as there aren’t many of them available (yet).

However, there are 2 big question marks associated with this architectorial picture:

  1. How can WebKit be used in a standalone way?
  2. Could I get rid of Qt completely then?

Q1 is not easily answered. If you look at the WebKit project structure you will find that it’s very much interwoven with the different ports like Safari or Chrome at the source code level. Port specific functions are partially included in the kernel and the same or more applies to the build process. WebKit is not a library that you can just consume. All ports are centrally built on WebKit’s environment and good builds are consumed by the browser vendors as a whole including the vendor specific extensions to whatever extend they want those to have public. You can get an impression here: WebKit Buildbot: Waterfall Display To make it worse, the whole WebCore engine is nearly completely undocumented in the source code. Only some public introductory documents are available in their Wiki or perhaps something more advanced on the Chromium Homepage. But nonetheless, a port should be possible, though with quite some effort, in a way to render to the underlying OS’s platform Grahics2D subsystem.

Although I am miles aways from making any achievement on this I have a better picture on the answers to question #2:

Qt’s feature rich functionality provides an experience to the C++ application developer which is at about the same level as the Java runtime library is to the Java developer. Which means: it will be hard to replace.

In the picture above I have created two possible scenarios for the replacement of certain aspects of Qt my application is making use of. It will – though not effortlessly – be possible to continue living without Qt either by consuming some functional equivalent from boost or by implementing a proprietary replacement. I have mixed feelings about boost, even though I consider it the right approach to utilize templates to be as syntactically clean as possible in C++, this excessive use of templates does not really lead to intuitively readable code which makes me reluctant and I tend, given the meta-driven approach I am implementing anyway, to replace of the Qt functionality by proprietary code making use of my meta level elements.

Rationale for HTML in native applications

In my last article (Not So Easy) Hybrid Applications with Qt, WebKit and JavaScriptCore, I explained my thoughts on the limitations of the scripting capabilities built into Qt’s WebKit port. One of the options to build a nice web based UI within a native C++ application would be to use WebKit directly. Before I elaborate on that in an upcoming article, I would like to explain why this would be worth considering at all.

Here’s a picture of how I see current state of the art in UI development:

Over the course of the last 15-20 years software engineers focussed on the provisioning of higher level windowing toolkits on top of the very low level UI libraries like X11 and WINAPI. These higher level toolkits tried to provide some consistent look & feel to the end user, sometimes across platforms. They all lack, however, one essential feature: style. This is a direct result of providing a prepackaged look which can hardly be changed by the application developer. And this lack of style is a result of lack of flexibility.

On the other hand, people love style which is the reason why they prefer the web, i.e., HTML based UIs over native UIs in many cases. Engineers cannot provide style because they mostly love their particular toolset and fight with other engineers whose one’s is best.

The question is: Who can provide style? One possible answer is: Graphical designers can! With HTML, they can not only create good looking user interfaces, but they can also create different UIs for different target audiences, which makes HTML so attractive across all industries. The required flexibility is only achievable with HTML or with special interfaces based on very low level drawing capabilities like OpenGL in computer games and it is hardly imaginable to create each and every user interface from scratch like computer games. This leaves HTML as the most attractive solution.

Given that, my #1 goal is to provide as much as HTML in a native application as possible, because only then I can consume stylish designs created by graphical designers in a quality which only graphical designers are able to produce.