Why I completely dropped Qt and QtWebKit from my Hybrid App

In my recent articles I described certain aspects related to the use of QtWebKit in a hybrid C++ application. In the meantime, it turned out that Qt and QtWebKit are not such a good bet as I thought they would be at the beginning. Here’s why:

Apple

First and foremost, the most interesting question in a hybrid app written in C++ is: where do you want it to run? In my case, it is on Mac OS X in the first place, then on Windows and very likely also on Apple’s iPhone or iPad. However, this puts me in a situation where I have to decide which subset of these platforms I support primarily and which ones secondarily as there is no such thing as a common layer to access all of them from a single code base. It is either a combination of Mac OS X and Windows via the Qt port or a combination of Mac OS X and iPad/iPhone that can go as the primary target.

For a certain time, I concentrated on the first combination as shown above while hoping that time works for me and somebody ports Qt to the iPhone, and there already is a project for that at http://www.qt-iphone.com/Introduction.html. But, in the meantime, Apple introduced iPhone OS 4.0 and *bam* there will never be such a thing, because http://daringfireball.net/2010/04/iphone_agreement_bans_flash_compiler “and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited)”, a rule introduced by Apple which is actually targeted against Adobe and its Flash technology but will also affect Qt, especially because that comes from Nokia, another one of Apple’s (future arch) rivals. This means that Qt can never be the common layer on all of my targeted platforms. Given that, and the fact I target Mac OS X first, Windows second, and that I also tend to shift Windows one step further down and promote the iPad version instead which completely changes the porting picture:

This looks much cleaner now as it provides unified, access to all Apple platforms. For Windows and Android Google Chromium is a good choice.

But what about Qt in general?

Qt / Nokia

The first part was the rational part, here comes the emotional one:

In the recent weeks I had enough time to dive into the details of Qt in all aspects. Its visual quality, its API quality, its implementation quality and the quality and structure of their development process. My conclusion from that is that it is the wrong way of doing things as evaluated from a broader perspective.

What is very important in a visual app is visual quality. Qt does a good job in replicating functionality of controls on each addressed platform. But, it can never, by its nature, take care about the special controls certain platforms offer because then it wouldn’t be cross-platform any more. Qt certainly has to focus on the lowest common denominator across all supported platforms. By driving the GUI via Qt controls, an application loses the capability to provide the eye-candy to the platform user like he is used to. The most prominent example of this is Google Maps which is based on Qt and which totally lacks platform L&F on the Mac up to a degree where it certainly loses style. In Google Maps this is not so important as interaction with the Earth requires a special user interface anyway and the rest of the app is unimportant, but it shows how difficult it is even for Google to provide attractive visual quality on the platform. Gaining access to the platform specialities is difficult as everything is hidden by Qt in private implementations. You have no chance to do “that special thing”.

Qt has a comprehensive “fat” API that consumes everything from all platforms and which has to take care about the specialities on all platforms, which is of course its nature but which is also very difficult to maintain and to improve over years. Development of Qt is distributed across Europe with Nokia focussing on minimizing cost not maximizing quality. Their development centers are mostly in cheaper parts of Europe and they try to cooperate with educational institutions to bring the costs down. This might be a legacy from Trolltech but is still in place and shows that quality is not their first concern.

So far I have touched Qt releases 4.5, 4.6 and 4.7 and came across older API documentations. From there I can tell that many things they have done are not very well and strategically thought out. They introduced too many APIs doing similar things and deprecating previous ones (QML, QtQuick, Scripting with QtScript and in QtWebKit, Qt3 vs. Qt4, QGraphicsScene, QItemView, QtDesigner, QtCreator). To me this is too much feature driven and dictated by the progress of other market players. Then I saw these super new technologies contrasted by very basic demo videos in YouTube showing excitement over very basic features which is not very professional – you can check out their channel on http://www.youtube.com/user/QtStudios to get an idea. What I want as a consumer of platform technology building a commercial grade application is that the selected platform is very stable and well thought out. But there is just too much noise coming from Qt and to make matters worse, none of the new features really feels final, it feels more like work in progress and you can never know how long something survives. Last not least, one day down the road Nokia might become a rival again to Apple just as relevant as Google now is. Economically speaking, Nokia already is, but Apple sees Google as its main competitor, not Microsoft and not Nokia. They are just not in the same technology league any more. Apple and Google have the brains now.

Don’t get me wrong, the Qt guys are doing a good job and they surely do it with a lot of enthusiasm, all I want to say is that Qt is just not the best bet in a commercial hybrid application. That is why I gave up on it.

Accessing the Original WebKit API in QtWebKit Hybrid Apps

Everybody wraps WebKit

For each and every port of WebKit, be it Safari, Chrome, QtWebkit, there is always a wrapper around the WebKit object model. These wrappers function as language mappers, e.g., in case of Safari C++ API is mapped to Objective-C, and as a simplification layer around the complex WebKit API. The first case makes total sense, as no developer familiar with Cocoa and Objective-C will like to take an excursion (back) to C++. On the other hand, the Chrome port as well as the Qt port expose a C++ API towards their developers and completely hide the WebKit API despite that this is also already written in C++. Within the C++ universe, I do not understand why the original WebKit API is such a bad bet and must be hidden under all circumstances (I derive this from the fact that everybody hides it)? Is it really that evil?

WebKit API is very stable as it is a reflection of HTML and XML and all the things used in the browser for years and which do rarely change, so a dependency should be acceptable. The only benefit is that the Qt developer works solely with the Qt API or the Chromium developer with a Chromium API. This is ok in case of very simple integration with WebKit, like displaying some external web content in an otherwise traditional application or when implementing a browser. But not in a hybrid application! In a hybrid application I want much more control over the internals of the browser. I do not consider it evil. I consider it as a very nice work horse doing most of the rendering and layout work. I want to reference DOM elements from my application and I want the DOM to interact with my application. I want to replace the excessive use of JavaScript found in Web 2.0 apps with excessive C++ – I do not want JavaScript in a hybrid app, I want C++ only!

In case of QtWebKit, how hard is to get access to the original WebKit API? It is very hard, as the necessary headers are all eliminated from the QtWebKit API and all WebKit symbols are local in the compiled DLL or OS X framework However, I have taken the pain and hacked me into it. The steps required to expose the WebCore API via QtWebKit are:

Get the source tree from Qt git

You will need to build from source. So, get it from http://qt.gitorious.org/qt and switch to the desired branch or tag.

Configure Qt just as normal

configure it but do not run make now!

Adjust the compiler flags for WebCore

run the following commands in the source tree

sed -i -e 's/-fvisibility=hidden//g' src/3rdparty/webkit/{Web,JavaScript}Core/Makefile.*
sed -i -e 's/-fvisibility-inlines-hidden//g' src/3rdparty/webkit/{Web,JavaScript}Core/Makefile.*

(or change these the Makfiles manually) This will compile WebCore in way to expose all internal symbols later in the DLL or framework.

Gain access to the WebKit API

In src/3rdparty/webkit/WebKit/qt/Api/qwebelement.h make QWebElement::m_element public

This is just one way to do it (see later for an alternative)

Build Qt

Now run the normal make; make install

Add WebKit header files and defines to your .pro file

The application code needs to include the WebCore header files.

DEFINES += QT_SHARED ...
release:DEFINES += NDEBUG
INCLUDEPATH += /Users/andre/src/qt/git/src/3rdparty/webkit/WebCore/bridge/qt \
    /Users/andre/src/qt/git/src/3rdparty/webkit/WebCore/page/qt \
    ...

This is a very shortened version to save space here. See the full set of defines in this file: myapp.pri

The order is important. Replace /Users/andre/src/qt/git with the home of your Qt source tree. Some defines are very important and some might be just optional, I just took all of them from the effective compile command when Qt was built.

Include WebCore headers in application code

#include <WebCore/html/HTMLElement.h>
#include <WebCore/platform/text/PlatformString.h>
#include <WebCore/platform/text/CString.h>
#include <WebCore/svg/SVGElement.h>
...

Alternative access to QWebElement::m_element

Instead of changing the visibility of the “m_element” member from private to public, it would also be possible to use a fake subclass in the application code with a public member m_element like here:

class HackWebElement: public QWebElement {
public:
WebCore::HTMLElement *m_element;
};

and then gain access to it by down-casting a QWebElement to a HackWebElement:

QWebElement webElement = ...;
HackWebElement *hwe = (HackWebElement*)&webElement;
// now we can access hwe->m_element

You can also use your preferred method to access a private member in C++ 😉

Using the WebKit API

Now I can do fancy stuff with the WebKit API. Here is an example where I create an SVG element programmatically, i.e., without the need to have WebKit parse some HTML (or XML in this case):

WebCore::QualifiedName svg("svg", "svg", "http://www.w3.org/2000/svg");
RefPtr<WebCore::SVGElement> nel = WebCore::SVGElement::create(svg, hel->document());
WebCore::String s = nel->tagName();
debug(string(s.utf8().data()));
if(nel) {
    debug("it's a WebCore::SVGElement!!!");
}

Conclusion

So far, I have only gained initial access to the internal browser DOM as a first step. This way I can create DOM elements programmatically. In the future, I will experiment with event handlers on DOM objects written in C++ in order handle all UI events in the C++ part of my application. This will reduce the browser part to act as a pure layout and rendering engine which I think should be its sole role in a hybrid application.

Regarding the little “hacky” approach I can say that there is only one place on the whole application code where access to WebKit is gained. This can be in the document, in the frame, or like here in the general element. From then on, no further hacks are required, it is just used and QtWebKit is not needed any more.

I would also wish that in ports like QtWebKit the WebCore API is preserved and exposed to the application developer. I think there are 2 aspects in a WebKit port: One is the physical rendering and display and the other one is the kind how WebKit embeds into the application and is access. Both concerns should be considered separately and I want to be free to consume only the first one and live with the original in the latter one.

Embedding 2D Drawing Objects in QWebView

One question not so easily answered is how to display high-frequently updated dynamic content in a QWebView. QWebView is a great tool to build hybrid web applications in C++. But because it is basically only a port of WebKit and Qt tries to wrap and encapsulate as much as possible, it is also only a plain web browser, just capable of displaying content that comes in HTML form. What if you want to display content within a web page like a stock chart or even a video…?

The options WebKit browsers offer are basically:

  • <object>, , <applet>: they each require you to implement a specific API and in case of applet even a different programming language
  • html 5.0 canvas: this is interesting, however, it requires JavaScript to program against it which might be to slow due to frequent parsing; it would be much more interesting if a QWebElement style wrapper would be available for drawing directly from C++
  • http://code.google.com/p/nativeclient/ is very interesting in normal browsers, but not in a hybrid application, as it is just another form of the <object> tag

This is why I have been thinking about a different approach, which is closer to the machine’s (i.e., Qt’s) drawing and came up with this idea:

Here we place a special, empty <div> in the document which can later be accessed by it’s ID using a QWebElement. We overwrite the QWebView::render() method and check if the region overlaps our dynamic content as specified by the QWebElement::geometry() data. The actual dynamic content can for instance be pre-rendered into an internal buffer, which we can use later as the source of a pixel copy operation towards the QWebKit paint device. Whenever our dynamic content changes, we can force a repaint via QWebView::repaint() exactly on the region that has changed.

Architecture-wise this just means that the main presentation layer of the hybrid app is not accessed in an outbound fashion only. I have no problem with that, as long as the dynamic content can be displayed and the module doing that can later be replaced with some less “hacked” approach…

03/05/2010 Update: HTML 5.0 Canvas is working in QWebView. I don’t know to what exact extent but some initial drawing tests completed successfully.

Alternative Hybrid Application Architecture

In my recent article (Not So Easy) Hybrid Applications with Qt, WebKit and JavaScriptCore I reflected my opinion on the problems associated with the current Qt scripting APIs QtScript and WebKit Script. I’d like to depict the differences between both again:

QtScript and QtWebKit both use the same JavaScriptCore but from different code copies and by wrapping them both with completely different APIs. In the meantime, I have also crosscheck current status at Qt and Nokia’s scripting plans. There are two separate issue tracking databases, one at Nokia for Qt development in general and one at webkit.org tracking the Qt WebKit port progress. Here are the 2 relevant entries:

I left my comment on the first one as I am still afraid that the Qt developers might come up with even a 3rd API. If that 3rd API is a unified API which can be used for QtSCript as well as in QWebKit scripting then it’d be fine. Given the comment I received from Kent on the first one, I hope for the best.

Alternatives
So far on that, but I want to focus now on the alternatives to the QWebKit based approach to building hybrid – i.e., Web enabled – applications in C++. First of all, WebKit in itself is an outstanding HTML rendering engine, both in performance and portability, both of which features I wouldn’t want to lose. That’s why I seek a solution based on WebKit, which could look like in this diagram:

The first and most important change is that whilst WebKit is still in it, it is not wrapped by Qt any more. Using the WebKit API directly, I would have access to the complete publicly available functionality in WebKit via e.g. HTMLElement and HTMLFrame classes. This would be to a great advantage. On the con side, I would lose Qt’s additional features from the wrapper classes, but I think they weren’t that many so that should be no big deal. Most of the customization would have been done using overridden event handling functions of the QWebView anyway and not via signals/slots as there aren’t many of them available (yet).

However, there are 2 big question marks associated with this architectorial picture:

  1. How can WebKit be used in a standalone way?
  2. Could I get rid of Qt completely then?

Q1 is not easily answered. If you look at the WebKit project structure you will find that it’s very much interwoven with the different ports like Safari or Chrome at the source code level. Port specific functions are partially included in the kernel and the same or more applies to the build process. WebKit is not a library that you can just consume. All ports are centrally built on WebKit’s environment and good builds are consumed by the browser vendors as a whole including the vendor specific extensions to whatever extend they want those to have public. You can get an impression here: WebKit Buildbot: Waterfall Display To make it worse, the whole WebCore engine is nearly completely undocumented in the source code. Only some public introductory documents are available in their Wiki or perhaps something more advanced on the Chromium Homepage. But nonetheless, a port should be possible, though with quite some effort, in a way to render to the underlying OS’s platform Grahics2D subsystem.

Although I am miles aways from making any achievement on this I have a better picture on the answers to question #2:

Qt’s feature rich functionality provides an experience to the C++ application developer which is at about the same level as the Java runtime library is to the Java developer. Which means: it will be hard to replace.

In the picture above I have created two possible scenarios for the replacement of certain aspects of Qt my application is making use of. It will – though not effortlessly – be possible to continue living without Qt either by consuming some functional equivalent from boost or by implementing a proprietary replacement. I have mixed feelings about boost, even though I consider it the right approach to utilize templates to be as syntactically clean as possible in C++, this excessive use of templates does not really lead to intuitively readable code which makes me reluctant and I tend, given the meta-driven approach I am implementing anyway, to replace of the Qt functionality by proprietary code making use of my meta level elements.

(Not So Easy) Hybrid Applications with Qt, WebKit and JavaScriptCore

Recently I discovered a set of slides from the Nokia’s Head of Technology Development, Pekka Kosonen, here: http://www.slideshare.net/pkosonen/next-generation-hybrid-applications-with-qt-presentation-for-see-2009 In this presentation, Pekka gives an overview about what Nokia’s Qt team calls “Hybrid Applications”. What this basically means is an application which makes use of web technology within traditional programming environments like C++. I am myself working on an application based on such an architecture, though I didn’t have a name for such an approach yet, so my thanks goes to Nokia for providing me with another buzzword for my professional life as a software consultant 😉

In this article I want to focus on my current approach, the current state of the API implementations at Nokia and their limitations.

The picture shows the basic structure of my application architecture, as it was design at the very beginning. At its core, there is the model object layer. This layer carries all business objects as well as business functionalities and will be implemented as QObjects while making use of some essential features of Qt like its datatypes and some of the hardware abstraction APIs. One particular aspect important to mention is the QtScript API, which shall be used by the final application after deployment to provide functional customizability to the customer of the application.

QtScript is a scripting environment that is as of Qt 4.5 based on the JavaScriptCore framework provided by WebKit, i.e., a highly optimized scripting engine, it even comes with a JIT. Using the QtScript APIs it is possible to integrate the complete model objects layer into the JavaScript engine.

The presentation layer in my application consists of 2 parts: the presentation of structured text, tables, forms etc. and the presentation of 2-dimensional drawings. In the hybrid application, the display of structured text will be accomplished using QWebViews, which delegate internally to a WebKit engine (the same WebKit as found in Apple’s Safari or Google’s Chrome web browsers). The complexity of the WebKit engine is completely encapsulated by QWebView wrapper classes like QWebElement and QWebFrame. For the 2D drawing part of the application Qt’s QGraphicsScene will be used. Alternatively, HTML 5.0 JavaScript Canvas could be used if performance is good enough.

Now back to the presentation I mentioned above: In the API of a QWebFrame there are 2 interesting methods:

void addToJavaScriptWindowObject ( const QString & name, QObject * object )
void addToJavaScriptWindowObject ( const QString & name, QObject * object, QScriptEngine::ValueOwnership own )

which can be used to provide the UI scripting environment with direct access to the model object layer and vice versa. However, these two functions are the only access point towards the browser internal scripting engine, which makes it impossible to add serious scripting support into the UI scripting part. Having access from the WebKit JavaScript to the C++ model object would offer the possibility to implement very lean presentation in JavaScript, for instance to react on a signal in JavaScript like something has been added to some model element could then lead to a new <TR> being added to a <TABLE>, which coulld all be handled via JavaScript!

This however, will not be possible as long as the API to access the WebKit’s JavaScriptCore is solely based on these two methods only and as there is no way to enrich the WebKit engine with the necessary configuration to tell it about the object model features like all its associations or constructors.

All of this is possible, however, in the standalone QtScript engine, which is essentially based on the same JavaScriptCore but which unfortunately is completely separate from the browser’s JavaScriptCore even though they both do the same. Both JavaScriptCore even have different source code copies in the Qt source code. I can only assume that Nokia separated both because they target different execution environments and therefore they must be adapted differently, but even if, I would like to ask why?

My conclusion so far is that I either have to wait for Qt to integrate both engines in a homogenous style (or do it myself) or think elsewhere, perhaps by integrating the WebKit directly and dropping QWebView completely.

QWebView QWebElement Experiment

1. Teil

In diesem Experiment mit dem QWebView und dessen DOM-Elementen, den QWebElement’s (aus der Qt beta 4.6) untersuche ich die Performance im Falle hochfrequenter Updates auf dem DOM. Ziel ist es herauszufinden, ob sich herkömmliche GUI-Entwicklungen (z.B. mit QtGui) durch eingebettete browserbasierte Ansätze effizient ersetzen lassen, um dadurch in den Genuss der fortgeschrittenen Layoutingfähigkeiten moderner Webbrowser zu gelangen.

Video in Originalauflösung: http://www.pareis.com/images/stories/video/QtWebView-Experiment.mov

Die Testapplikation verwalten im Bereich des C++ Modells Referenzen auf die angeschlossenen View-Elemente im DOM des eingebetteten Browserwidgets. Bei Änderungen an den Modelldaten werden die formatierten Zahleninformationen in die DOM-Elemente geschrieben und anschliessend je nach Änderungstyp die CSS class der Zelle (table cell, td) gesetzt.

void Browser::updateCells() {
    for(int i=0; i<batch; i++) {
        int r = rand() % rows;
        int c = rand() % columns;
        double orig = values[r][c];
        double &now = values[r][c];
        now = 150.0 * rand()/((double)RAND_MAX + 1) - 20.0;
        QString s = QString("%1").arg(now, 0, 'f', 2);
        cell[r][c].setPlainText(s);
        if(now  orig) {
            cell[r][c].removeClass("down");
            cell[r][c].removeClass("norm");
            cell[r][c].addClass("up");
        } else {
            cell[r][c].removeClass("down");
            cell[r][c].removeClass("up");
            cell[r][c].addClass("norm");
        }
        coord cc = {r,c};
        queue.enqueue(cc);
        if(queue.size()>1000/interval/2*batch) {
            cc = queue.dequeue();
            cell[cc.r][cc.c].removeClass("up");
            cell[cc.r][cc.c].removeClass("down");
            cell[cc.r][cc.c].addClass("norm");
        }
        nupdates++;
    }
    elapsed += t.elapsed();
}

Die Ergebnisse können sich sehen lassen. Das eingebettete Browser-Widget bewältigt mehr als 1000 TD Änderungen pro Sekunde. Den limitierenden Faktor stellt der QTimer dar bzw. dessen Signals. Werden aber viele View-Änderungen innerhalb eines einzelnen Qt-Events durchgeführt, bleibt die CPU-Last im sinnvollen Rahmen. Ein sehr positiver Aspekt ist die problemlose Bedienbarkeit der Oberfläche selbst im Bereich der Sättigung.

2. Teil

In einem 2. Versuch habe ich einen direkten Vergleich der Performance zwischen dem embedded QWebView Browser und einer rein Qt-basierten Anzeige vorgenommen. Ausserdem habe ich die Ereignisproduktion in einen separaten Thread ausgelagert.

In der 2. Applikation arbeiten folgende Codebestandteile:

Das Modell emittiert Changes:

void Changer::run() {
    forever {
        if(stopped) return;
        msleep(1000/interval);
        for(int i=0; i<batch; i++) {
            int r = rand() % rows;
            int c = rand() % columns;
            double orig = values[r][c];
            emit changed(r, c, orig,
                values[r][c] = 150.0 * rand()/((double)RAND_MAX + 1) - 20.0);
        }
    }
}

Im embedded QWebView die gleiche Verarbeitung wie oben, leicht modifiziert:

void Browser::updateCell(int r, int c, double orig, double now) {

    QString s = QString("%1").arg(now, 0, 'f', 2);

    QWebElement &ccc = cell[r][c];
    ccc.setPlainText(s);

    if(color->isChecked()) {
        if(now < orig) {
            ccc.setAttribute("class", "down");
        } else if(now > orig) {
            ccc.setAttribute("class", "up");
        } else {
            ccc.setAttribute("class", "norm");
        }
    }
}

Im Qt Widgets basierten Ansatz sieht es fast identisch aus, ausser dass hier keine class-Attribute gesetzt werden (was in Qt auch ginge, jedoch aus Performancegründen nicht in Betracht gezogen wurde) sondern Farbvorgaben gemacht werden, was selbstredend nicht so sehr flexibel ist wie HTML:

void QtBrowser::updateCell(int r, int c, double orig, double now) {

    QString s = QString("%1").arg(now, 0, 'f', 2);

    QLabel *ccc = cell[r][c];
    ccc->setText(s);

    if(color->isChecked()) {
        if(now < orig) {
            ccc->setPalette(red);
            ccc->setAutoFillBackground(true);
        } else if(now > orig) {
            ccc->setPalette(green);
            ccc->setAutoFillBackground(true);
        } else {
            ccc->setAutoFillBackground(false);
        }
    }
}

Threadanalyse

QWebView-threads

Threaddump eines laufenden QWebView

Im browserbasierten Ansatz werden ca. 61% der Zeit mit Repaints, ca. 3.5% mit dem Eventhandling und interessanterweise ca. 30% mit einem asynchronen (timergesteuerten) Layouting zugebracht.

QtWidgets-threads

Threadanalyse Qt Widgets basierte Anzeige

In Fall der Qt Widgets basierten Implementierung werden ca. 76% der Zeit im Repaint und ca. 15% im Eventhandling verbracht.

Fazit

Ich komme nach diesem Experiment also zu dem Schluss, dass der embedded Browser gegenüber dem Qt-basierten Ansatz zunächst erst einmal keine signifikanten Performancenachteile besitzt und sich daher als Basis für die Implementierung von Rich-Client-Applikationen mit C++ anbietet, bei denen es auf sehr gute Grafikfähigkeiten gepaart mit einem hochperformanten Kern ankommt.