buscatcher: Never miss another tram

Opening public data is a hot topic in Finland at the moment. As a small experiment with the data that is available I wrote buscatcher, a simple N900 app that displays Helsinki trams (and some buses) moving on a map in real time. This makes it easy to determine when your next tram is coming to the stop, or where it is stuck.

So far I'm keeping this application away from Extras until HSL gets scalability issues solved with their dataset. In the meanwhile you can grab and run the application from the GitHub repo.

My interview at dot KDE

Jos Poortvliet did an interview with me for dot KDE in this summer's aKademy and it has been online for a while now. In it we discuss things like Midgard as a storage engine for desktop applications, and Maemo's open QA process for Downloads applications. Some excepts:

At maemo.org we have an appstore for FOSS applications on the Maemo platform. This appstore is enabled by default on all Nokia N900s so we wanted to have some quality control. We had to create our own appstore approval process, compatible with the FOSS philosophy. Now any developer can submit an app, and anyone can test and vote. The whole process is completely transparent, auditable and visible. And it also provides a feedback channel from testers and users to the developers!

...

Midgard is a data storage service. Whether you write desktop or web applications, instead of coming up with your own file format, you just use Midgard. You can work more easily and object-based. Users have many different devices these days, so Midgard has strong replication features to synchronize between different systems. Midgard is built on top of GObject; we provide bindings to a bunch of different languages so developers can choose the tools they like - PHP, Python, Javascript. Currently (as in now, while we're talking) Qt bindings are being developed here at Akademy.

Read the whole interview.

Aloha and the art of semantic web content

To bring CMS editing to the next level, the IKS project is working on a semantic HTML5 editor. This week we had a hackathon in Helsinki focusing on implementing our ideas with the Aloha Editor. In addition to enjoying the hot summer weather here, we accomplished quite a bit and in the end were able to present the whole pipeline of:

• Loading content from Midgard CMS to Aloha Editor
• Annotating our content with Google-compatible Person RDFa elements
• Saving the content back to Midgard
• ...and finally analysing the content with FISE to find more semantic information

The hackathon participants included developers from Nemein, Gentics, Infigo, Salzburg Research and the German Research Center for Artificial Intelligence. Some screenshots:

Editing content with Aloha in Midgard

Annotating persons with the Aloha RDFa plugin

RDFa annotation created with the semantic editor

Additional semantic information suggested by FISE

All the relevant code can be found from GitHub (see also the FISE Midgard integration).

Frankencamera aims to make cameras open and programmable

Frankencamera, or fCam, the open source computational photography platform from Stanford's Camera 2.0 project was unleashed for the Nokia N900 this Wednesday. PhysOrg has a story outlining the significance of this:

Computational photography refers to the ways computers can extend the capabilities of digital imaging by combining multiple photographs taken with different camera settings to create an image that could not be taken in a single shot, or with an ordinary camera.

Some of these new ways of combining images can be done in Photoshop or another such program, but until now they could not be done inside the camera, Levoy said. That's because commercial cameras are closed to development by all but their manufacturers. Frankencamera, on the other hand, brings computational photography directly to the camera, by making the camera a programmable platform.

I installed fCamera and the HDR photo assistant from Maemo extras-devel yesterday, and the results (taking .DNG RAW images, automatically generating HDR pictures) seem quite impressive. Here is a quick example from our office. Sun is shining outside and the office is not lit:

For comparison, here is the same setting with the regular N900 camera application:

It will be interesting to see what developers will come up with, now that all these camera capabilities are available through an open API!

Recent performance improvements for Midgard 8.09

Midgard 8.09 is an industrial CMS that is now in Long-Term Supported stage, with the community maintaining it until 2013. As we all know, performance is a feature, and with a CMS framework that has lived through many changes including transitions from PHP4 to 5.2 and from Classic Midgard era to the modern APIs, there is a lot to do.

For the next 8.09.10 release we decided to put quite a bit of efforts into performance tuning, with some excellent work done by Content Control to simplify ACL handling and cache navigation information. As you can see, the result is quite impressive:

What is left to be done is some work with the multilingual content database queries. After that we should be good to go with what is probably the fastest Midgard1 ever.

Zeitgeist does location: what did I do while in Brussels?

Zeitgeist, the desktop activity logging engine is now becoming geo-aware. From Seif Lotfy's blog:

It allows you to ask Zeitgeist stuff like

• “Get me the recent files I edited at university”
• “Who do I contact most when I am at School?”
• “Which pictures did I take in Brazil?”
• “Where was I when an Email came in?”
• “What files did I open during the conference?”

As I've been advocating since 2006, location is important for making applications smarter. While you might not remember where you stored some file, you probably remember where you were when working on it. Then Zeitgeist's location features, powered by GeoClue, will be able to get it for you.

This is especially cool since Zeitgeist is coming for Maemo as well. My laptop is quite mobile, but the N900 is even more so.

My GeoClue talk from aKademy 2010

aKademy 2010 was hosted in the sunny city of Tampere by the Finnish Centre for Open Source Solutions, an organization that I'm a steering group member of. In addition to helping a bit with the arrangements and organizing the Midgard Gathering there, I also gave a talk about GeoClue, the positioning framework for Linux desktops.

• Presentation slides (PDF)
• Video of my talk

We initially started the push for location-aware desktops around 2006, and now the efforts are finally starting to bear fruit. Both Zeitgeist and Nepomuk are looking at indexing documents based on where you accessed them, Telepathy can share your location with your friends, and hopefully soon also your desktop clock will switch timezones when you travel.

It is very cool that this development seems to be happening on both GNOME and KDE at a reasonably similar pace. GeoClue is also a service in MeeGo and I've been told another major mobile phone manufacturer uses it. Maybe soon Mac OS X will not be the only platform with location APIs built in?

Photo by Alexey Zakhlestin.

Meet Midgard and GeoClue in aKademy 2010

We tried to get the combined GUADEC and aKademy conferences to Tampere in 2009, but a warmer place unfortunately won. However, we will be hosting this year's aKademy so at least KDE and Qt fans will get to enjoy this beautiful northern industrial city.

The main conference will be held at the Tampere University over the weekend, and then the remaining hackweek will be in the nice Demola facility in the Finlayson district. Expect great connectivity and close proximity to all Tampere nightlife.

I'm involved with two aKademy activities:

• GeoClue presentation on the location service that can be used on any Linux desktop and is used in MeeGo. I'll also talk about what desktops can do when location is available
• Midgard Gathering during Mon-Wed of the hackweek. Midgard provides a content repository for applications that should be available also for Qt apps through GObject Introspection. With some proven performance benefits and replication support, Midgard should be interesting for any desktop or mobile app that deals with structured data

In addition there will be a Maemo / MeeGo meetup in the Plevna brewery on Friday evening. See you there!

Midgard Runtime brings our web framework to the desktop

Midgard2 10.05.1 was released yesterday, bringing a long-waited feature finally to the Midgard installation packages: the Midgard Runtime.

Midgard Runtime is an application that consists of a simple Qt WebKit viewer that, when run, starts a local Midgard web server on the background and connects to it. This means that you'll have the full Midgard MVC stack available on your own desktop, in a way that is easy to install and easy to run. Get it for your favorite Linux distribution from OBS!

As a whole, the Midgard Runtime stack is quite interesting:

• SQLite is used as the default storage through the libgda4 abstraction layer
• Midgard2 is used as the content repository, brought to PHP through an extension
• Midgard MVC provides the web development framework
• Alexey Zakhlestin's AppServer is used as the pure-PHP web server
• The Qt application starts server processes on the background and displays the site in a WebKit view
• Runtime's bundle setup tool uses libraries from the Pake build system

The AppServer is particularly something that other PHP projects might find useful. Since the server itself is written in PHP you don't have any additional dependencies besides php-cli. It also means your application can use a lot more efficient caching as files have to be loaded only once, and things can be kept in memory between requests. Of course the downside is that if your PHP script dies, then the whole server is down. But that is easy to deal with by a bit of defensive programming.

At this point the Runtime is targeted at web developers interested in trying out Midgard, but eventually we'll be using the bundle system also for distributing full Midgard-powered web applications to the desktop. The progress on this can be followed on GitHub.

Compared to Nokia's Web Runtime, the Midgard Runtime should be more familiar to developers as you'll be able to write also server-side PHP code, not just JavaScript. In addition to PHP, the whole system is also accessible via any language that can handle GObject Introspection.

So, what can I do with it?

• Install a Midgard environment with $ midgard2-runtime-bundle-simple-install
• Start the runtime with $ midgard2-runtime-bundle-simple-run
• Midgard MVC and components are located in ~/.midgard2/simple-bundle/. This is where you can also place your own code

Open Source? Free Software? What we need is Open Projects

Both companies and public administration are starting to understand the benefits of free software: reducing vendor lock-in, possibility to continue development of a project after a vendor has gone out of business or lost interest, and in general enjoying the four freedoms. But unfortunately much of this understanding has been limited to the context of licenses.

In reality, licenses are only a small part of a project being truly open. They are just a layer of insurance comparable to traditional source code escrow.

What we really need is understanding of a bit more wholesome project openness. The actual goals of openness that the license should derive from. Here are some aspects to consider:

Project transparency

If a project aims to have outside users or contributors, they need to be able to see the history of changes in the software, decisions that have been made, and the open list of bugs or enhancements being worked on.

A released software package answers these questions poorly regardless of a license. Instead, what is needed is the project being developed out in the open, preferably using one of the common project hosting environments like Gitorious, GitHub, SourceForge, Launchpad or GNU Savannah. You can also host the project yourself using something like Trac or GForge but this limits access and visibility to the project.

The project must actually use the service, not just by code dumps at release time, but with constant development activity visible as code commits and active issue tracking. Depending on business goals it is also good to have future plans for the project visible to the public.

All of this is mandatory for others to gauge the viability of a software package to their needs. Josh Berkus presented a good list of things you shouldn't do to create a community around your project.

Contribution policy

Potential users and developers need to know how they can make their changes available to a package. Is it possible at all, are copyright assignments or some contributor agreements necessary, is there a documented process for submitting changes or even becoming an acknowledged developer in the project? Or is the project being developed behind closed curtains of a company?

Requirements and software stack

Another area some projects fail at is communicating how the software can be built and installed. If the only practical way to run the software is from released binary packages, or through buying consulting, is it truly open? Does the project require additional closed software or specific hardware to run with?

Specialized licensing concerns

Depending on the type of software other concerns may be being able to provide it as part of a Software as a Service offering, or being able to deploy it on some constrained or closed hardware.

Some software licenses address these questions clearly, like EUPL requiring contributions to be opened also when the software is offered in SaaS manner, or GPLv3 forbidding device manufacturers from locking down or 'Tivoizing' their hardware products.

Wrapping up

Most of these questions are well understood within the free software community itself. But we generally communicate it poorly by focusing the discussion on license technicalities. I guess this is because we're so used to working in this open manner that we take the it as a given. But users, especially in the public administration only see the licensing side of things because that is the only aspect we talk about and have definitions for.

A good exception for this is the Apache Software Foundation that has a well-defined set of rules that projects must follow before they can be adopted under the ASF umbrella. Maybe FSF and OSI should also publish some understandable guidelines and definitions for project openness?

Staff meeting in the park

At Nemein we have a monthly staff meeting to go through all project backlogs, new initiatives and happenings in the R&D side of things. Today the weather was nice, so we decided to keep the meeting in the nearby Sinebrychoff Park, armed with croissants, cake and the company waterpipe. Quite pleasant change from routine!

Thanks to Aslan for the cake!

First year of IKS for Midgard

Last year we became a partner in the European Commission -funded Interactive Knowledge Systems project aiming to increase semantic capabilities in Open Source CMSs and vendors.

The first year of IKS has focused mostly on research and requirements gathering. As a project participant we at the Midgard project have taken this time for making our own preparations as well. Here are some initial results:

• grokking the semantic web and linked data ideas and terminology obviously required some amounts of studying
• we did a comparison of features and concepts between three content repositories together with Day Software. The similarities between Midgard, JCR and CouchDB are remarkable
• while others in the IKS sphere are focused on Java, we've started collaborating with the developers of Tracker, a GObject-based RDF triple store. Tracker would fit the Midgard architecture much better and we would be able to avoid big external dependencies
• we chose Wymeditor as the new rich text editor in Midgard CMS. Wymeditor is fast, jQuery-based and relatively sparse of features, allowing us to build the functionality we need in an integrated manner. Wymeditor is also capable of dealing with RDFa
• MgdSchema, the XML syntax for describing Midgard content types was expanded to allow us to link objects and properties to appropriate RDF ontologies
• we participated in several IKS meetings around Europe: Salzburg, Rome, Salzburg, Rome and now Paderborn
• preparations were made to migrate Midgard's default HTML templates from Microformats to RDFa

All of this aims for making Midgard compatible with the designs of the IKS project, but building as much of it on top of the pure GNOME + PHP stack as possible.

Midgard2 has moved to GitHub

Now that Midgard2 is at Long-Term Supported stage it was time to finally make the jump and migrate our development efforts to happen on top of Git, the fast version control system. To maximize project visibility and enable easy tool access we chose GitHub as the Git hosting provider.

While migrating to Git we also decided to implement the Distributed Version Control model where the MidgardProject account contains "blessed" repositories of various modules of the Midgard2 ecosystem, with development happening in personal clones. Maintainers of Midgard modules will be responsible for merging changes from developers back upstream, based on pull requests. More information can be found from the developer list thread.

You can get started by following the MidgardProject account on GitHub! See also Midgard2 and Midgard MVC on Ohloh.

Midgard1 Ragnaroek maintenance and development will still happen in our old SVN environment and the Trac tool.

Privacy: how Qaiku is doing it

Facebook is facing a backlash on their constant erosion of privacy. They have a privacy policy longer than US constitution, and a track record that has even sparked a federal complaint. While I'm a big believer in a transparent society I still believe users should be in control of who can see their information and how.

I think Qaiku, the conversational microblogging service, is doing this quite well. In your profile you have a simple setting:

Privacy of individual conversations comes from the settings of the person who initiated that thread. The setting is clearly shown in the sidebar. Some examples:

This way you can know who will be able to see your comments and make the decision before posting them.

The Terms of Service are not too tricky either, containing items like Be Nice and What is yours is yours.

Notes from my first Ubuntu Developer Summit

Last week was the Ubuntu Developer Summit targeted at planning how the next iteration of the operating system, Maverick Meerkat, targeted at a October 10th 2010 release, would look like. The event was held in a spa resort off in the countryside near Brussels. A place where the developers were comfortably separated from the busy towns by forests and country roads.

I went to the UDS with two goals in my mind: to discuss making the Ubuntu desktop location-aware, and to figure out what sort of role Midgard, our content repository could play in the Ubuntu space. Normally such things would be better approached by talking to upstream projects and distributions, but as Ubuntu is taking an increasingly active role in developing itself to be a comprehensive and unique operating system, it would be a better place to accomplish such wide-ranging changes.

Location-awareness was a area that many were interested in. Ubuntu already had considered GeoClue, our location service, to be a component of the distribution in Lucid Lynx, but that plan had been abandoned due to the low-maintained status of the software. However, now that GeoClue has become the geolocation framework for MeeGo things are about to change, and so there is a significant possibility to get GeoClue into Ubuntu as well.

GeoClue by itself will obviously not do much, and so making applications use it is the important thing. Some applications already support location information and could be changed to include those features by default. This includes Empathy sharing location information with instant messaging buddies, and Getting Things GNOME TODO manager being able to tag action items with location. Zeitgeist is intending to start tagging and searching activity history with location, and Gwibber also wants to include location sharing to interface with social web services like Brightkite, Qaiku and Google Latitude. Another reasonably low-hanging fruit would be to make the Ubuntu panel clock automatically change time zones as you travel.

Midgard in Ubuntu is a bit trickier question. A lot of Ubuntu has now been developed to utilize the CouchDB document database, which provides quite similar storage services as Midgard does. CouchDB's cool map/reduce algorithms and replication capabilities have enabled Ubuntu developers to quickly integrate a content repository into many applications and to provide a cloud storage and synchronization service to users of the operating system.

But at the same time CouchDB introduces some problems:

CouchDB can't seem to handle the load of Gwibber's messages, leading to excessive CPU consumption and poor performance in certain cases. For example, the overhead of computing the views causes lag when the user switches streams after Gwibber refreshes. The cost of pulling the account configuration data out of the database can also sometimes cause a noticeable lag that lasts up to four or five seconds when opening Gwibber's account manager.

CouchDB has been designed with web servers in mind, and in many situations having a constantly-running Erlang process and having to do HTTP roundtrips to access data are undesirable on more constrained computers. We ported CouchDB to Maemo a few years ago and the performance was terrible. I'd imagine the same issues on netbooks and other small hardware that Ubuntu's Light initiative targets.

On such environments Midgard might be a better option. Midgard's recent LTS release uses regular relational databases for fast performance and low footprint, utilizes important pieces of free desktop architecture like D-Bus signalling and GObject Introspection, and has been proven to work well even on low-end devices like the Nokia N900. And yet it provides full content repository APIs comparable to CouchDB, and can even handle replication with Couch. One way to make Midgard possible, then, would be to provide Python DesktopCouch APIs to it.

Another interesting technology discussed much in UDS was Puppet, the  systems administration tool. It enables developers to describe their setups in a object-oriented declarative language, and then just let Puppet to make it so. I attended two Puppet sessions, and picked up a Kindle edition of the Puppet book which I read on the train ride to the Netherlands. As my company manages lots of Midgard web servers for our customers, automating system setup and maintenance on top of Puppet and Ubuntu Server might save us lots of time and effort.

All in all, quite an enlightening two days in the event. If things work out, I hope I'll be able to spend more time in the UDS targeting the N-series of Ubuntu next fall.