blog

Sprint Report: Merging Mockup and Patternslib

| categories: mockup, javascript, patternslib, austria, sprint, foss, plone | View Comments

Alpine City Sprint, 21 to 26 January 2015

This is a report on what I did at the Plone Alpine City Sprint organized by Jens Klein and Christina Baumgartner in Innsbruck between 21 and 26 January 2015.

Firstly, I want to thank to them for organizing and hosting a great sprint and for being such friendly and generous hosts.

My goal for this sprint was to work on merging the Patternslib and Mockup Javascript frameworks and I'm happy to report that very good progress was made.

Long story short, the merge was successful and it's now possible to use patterns written for either project within a single framework.

Before I expand on how this was achieved during this sprint, I'll first provide some necessary background information to place things into context and to explain why this work was necessary.

What is Patternslib?

Patternslib brings webdesign and development together.

Patternslib's goal is to allow website designers to build rich interactive prototypes without having to write any Javascript. Instead the designer simply writes normal HTML and then adds dynamism and interactivity by adding special HTML classes and data-attributes to selected elements.

For example, here is a Pattern which injects a list of recent blog posts into a element in this page:

Click here to show recent blog posts

The declarative markup for this pattern looks as follows:

<section id="alpine-blog-injected">
    <a href="#portlet-recent-blogs"
       class="pat-inject"
       data-pat-inject="target: #alpine-blog-injected">

        Click here to show recent blog posts
    </a>
</section>

The dynamic behavior comes from the fact that I have the Patternslib Javascript library loaded in this page.

On pageload, Patternslib scans the DOM looking for elements which declare any of the already registered patterns. If it finds such an element, it invokes a Javacript module associated with that pattern.

In the above example, the <a> element declares that it wants the Inject pattern applied to it, by virtue of it having the pat-inject HTML class.

Patterns are configured with specific HTML5 data properties. In the example above, it is the data-pat-inject property which specifies that the target for injection is the #alpine-blog-injected element in the current page and the content being injected is specified by the href attribute of the <a> element. In this case, the href points to an anchor inside the current page, but it might just as well be a link to another page which will then get injected via Ajax.

More information on how to configure pat-inject can be found on the patternslib website.

Each pattern has a corresponding Javascript module which is often just a wrapper around an existing Javascript library (such as Parsley, Select2 or Pickadate) that properly defines the configuration options for the pattern and properly invokes or applies the 3rd party library.

So, in order to do all this, Patternslib can be broken down into the folowing components:

  • A registry which lists all the available patterns.
  • A scanner which scans the DOM to identify declared patterns.
  • A parser which parses matched DOM elements for configuration settings.
  • The individual Javascript modules which implement the different patterns.

What is Mockup?

Mockup, declarative interaction patterns for Plone..

Mockup was inspired by, and originally based upon, Patternslib and was meant to bring the power of declarative interaction patterns to Plone.

When Mockup started, Patternslib was undergoing significant refactoring and development and it was decided that Mockup should fork and go its own way.

What this means is that the 4 different components mentioned above, were all changed and due to these changes the Mockup project diverged from Patternslib and started developing in a similar but different direction.

So what's the problem?

While Mockup was being developed for the upcoming Plone 5 release, we at Syslab continued using and improving Patternslib in our projects.

Syslab built an intranet for the Star Alliance, which was based on a prototype design by Cornelis Kolbach, the conceptual creator of Patternslib. This design became the inspiration and blueprint for the Plone Intranet Consortium (PIC), which consists of 12 companies working together in a consortium to build an intranet solution on top of Plone.

So, the PIC are building a product using Patternslib, while Plone 5 itself is being built with Mockup, an incompatible fork of Patternslib.

This was clearly a bad situation because we now had:

  • Two incompatible Javascript frameworks being used with Plone.

    Not only were the two frameworks incompatible in the sense that patterns written for the one don't work on the other, but they could also not be used on the same page since the two implementations would compete with one another in invoking Javascript to act upon the same DOM elements.

  • Duplication of effort

    The same or similar patterns were being developed for both frameworks, and when one framework had a pattern which the other wanted, it could only be used after being modified such that it couldn't be used in the original framework anymore.

  • A splitting of the available workforce.

    Developers were either working on Mockup or Patternslib, but almost never on both, which meant that the expertise and experience of developers wasn't being shared between the two projects.

How could this be fixed?

To solve the 3 main problems mentioned above, we needed to merge the common elements of Mockup (specifically the registry, scanner and parser) back into Patternslib.

This will allow developers from both projects to work on the same codebase and enable us to use patterns from both projects together.

At the Alpine City Sprint in Innsbruck, I worked on achieving these goals.

Changes brought in by Mockup

After the fork, Mockup introduced various changes and features which set it apart from Patternslib.

In order to merge Mockup back into Patternslib, I studied these changes and with the help of others came up with strategies on what needed to be done.

Here are some differences and what was done about them:

Mockup allows patterns to also be configured via JSON, whereas Patternslib used a keyword:argument; format

A week before the sprint I added JSON parsing ability to the Patternslib parser, thereby resolving this difference.

Leaves first parsing versus root first parsing

Mockup parses the DOM from the outside in ("root first"), while Patternslib parses the DOM from the inside out ("leaves first").

According to Rok Garbas, the creator of Mockup, the outside-in parsing was done because it reduced complexity in the scanner and the individual patterns.

Wichert Akkerman who originally wrote the Patternslib scanner however provided IMO a very good reason why he chose "leaves first" DOM scanning:

If I remember correctly there are several patterns that rely on any changes in child nodes to have already been made.This is true for several reasons: 1) a pattern may want to attach event handlers to child nodes, which will break of those child nodes are later replaced, and 2) child nodes changing size might impact any measurements made earlier.

Indeed, while refactoring the Mockup code during the merge, I ran into just such a case where a pattern couldn't yet be initialized because patterns inside it weren't yet initialized. By turning around the order of DOM scanning, this problem was resolved and the code in that pattern could be simplified.

So, now after the merge, scanning is "leaves-first" for Mockup patterns as well.

Mockup patterns are extended from a Base object, very similar to how Backbone does it

Patternslib patterns on the other hand are simple Javascript objects without constructors.

The patternslib patterns are conceptually very simple and more explicit.

However, what I like about the Mockup approach is that you have a separate instance with its own private closure for each DOM element for which it is invoked.

After merging, we now effectively have two different methods for writing patterns for Patternslib. The original "vanilla" way, and the Mockup way.

The Mockup parser was completely rewritten

The Mockup parser looks nothing like the Patternslib one and also supports one less configuration syntax (the so-called "shorthand" notation).

This was one piece of code which could not be merged within the time available at the Alpine City Sprint.

So currently we still have two different argument parsers. The Patternslib parser needs to be configured much more expicitly, while the Mockup one is more implicit and makes more assumptions.

Merging these two parsers will probably have to be done at some future sprint.

There are some other more minor differences, such as that every Mockup pattern is automatically registered as a jQuery plugin, but merging these was comparatively easier and I'll won't go into further detail on them.

What I did during the Alpine Sprint

So, in summary:

I refactored the Patternslib registry, scanner and some core utils to let them handle Mockup patterns as well. Luckily the eventual patch for this was quite small and readable.

I changed the Mockup Base pattern so that patterns derived from it now rely on the registry and scanner from Patternslib.

I fixed lots of tests and also wrote some new tests.

This whole task would have been much more difficult and error prone if either Patternslib or Mockup had fewer tests. The Mockup team deserves praise for taking testing very seriously and this allowed me to refactor and merge with much more confidence.

What else is there to still be done?

Being able to use patterns from both projects and merging most of the forked code was a big win, but there are still various things that need to be done to make the merge complete and viable.

I've ranked them from what I think is most important to least important.

1. Update the documentation

Currently documentation is scattered and silo'd in various places (the Patternslib website, the Plone Intranet developer docs and the Mockup developer docs).

The Mockup docs are now out of date ater this merge and need to be brought up to date on these recent changes.

The Patternslib docs are less of a problem because they don't have to deal with Mockup (which can now be seen as an enhancement suite for it), but they can definitely still be improved upon, specifically with an eye on Mockup developers who will start relying on them.

The Plone Intranet consortium also has a useful walkthrough explaining how to create a Patternslib pattern from scratch..

2. Devise a way to also use the Patternslib parser for Mockup patterns

As mentioned, the Mockup patterns still use their own argument parser.

Letting them use the Patternslib's parser will either require extending the Mockup Base pattern to configure the Patternslib parser on a pattern's behalf or instead doing it explicitly in each of the individual patterns.

3. Decide on coding and configuration standards

Unfortunately the coding standards between the two projects differ significantly.

  • The Mockup developers use camelCase as declarative argument names while Patternslib uses dash-separated names.
  • The Mockup source code uses 2 spaces for indentation, Patternslib uses 4 spaces.

4. Remove unnecessary, duplicated patterns

Both projects have patterns for modals, injection, select2 and file upload. These should be merged to get rid of duplication.

5. Move generic patterns (e.g. pat-pickadate, pat-moment etc.) out of Mockup

Mockup has some generic patterns which might also be useful as Patternslib core patterns, in which case they should ideally be moved there.

Conclusion

The sprint was a big success and I'm very happy that all the work has already been merged into the master branches of the matternslib, mockup and mockup-core repositories.

Core code from Mockup is now succesfully merged back into Patternslib and we can use patterns from both projects in the same site.

I had a lot of fun working with talented and motivated software developers and had a lot of opportunities to improve my German.

Thanks again Jens and Christine for organising a great sprint and I look forward to doing such a sprint again!

Innsbruck, the alpine city
Read and Post Comments

Converse.js 0.7 released

| categories: converse.js, cryptography, xmpp, javascript, chat, otr, foss | View Comments

Now supports Off-the-record encryption

I'm happy to announce that the 0.7 release of Converse.js now supports Off-the-record (OTR) encrypted messaging.

The OTR protocol not only encrypts your messages, it provides ways to verify the identity of the person you are talking to, plausible deniability and perfect forward secrecy by generating new encryption keys for each conversation.

How this works is actually fascinating. The encryption keys are generated via a Diffie-Helman exchange. On youtube there is a video with a very simple and intuitive explanation on how the Diffie-Helman key exchange works. For more details, you can also read the OTR version 3 protocol spec.

I was able to add OTR support for converse.js by using Arlo Breault's excellent otr.js library, which is also used in Cryptocat and Diaspora.

Many thanks go out to him for this.

Note

Note:

Arlo cautions that the otr.js library has not yet been properly vetted by security researchers and shouldn't be used in life or death situations.

This word of caution applies doubly so to its integration into converse.js!

I'm not a professional cryptographer and this should be seen as an experimental feature.

Caveat emptor!

Screencast

Note: For detailed fullscreen, make sure that video quality is set to HD, by clicking the gear icon on the video player.

On Javascript Cryptography

Implementing Cryptography in Javascript is relatively controversial and in 2011 Matasano security wrote a scathing criticism called Javascript Cryptography Considered Harmful.

I read that article before I started working on OTR support for converse.js and I'd like to go through some of its criticisms step by step to explain how they might apply to converse.js and what I've done to mitigate them.

If you're going to read on, it would make sense to go read that article first.

Secure delivery of Javascript to browsers being a chicken-egg problem

Their criticism appears to apply to the case where developers use Javascript cryptography to implement secure communications with a server instead of relying on a standard protocol such as SSL/TLS.

The OTR crypto is not trying to compete with or replace SSL/TLS.

Sending data in the clear with normal HTTP is not secure and should be avoided. Therefore, if you are going to use OTR with converse.js, always make sure that the site users HTTPS (i.e. SSL/TLS).

Ideally you would also want a mechanism to sign the Javascript and then verify it's authenticity once it's been downloaded.

This can be done with code that runs in browser extensions (like Cryptocat does), but (AFAIK) unfortunately not with browser-agnostic javascript (like converse.js).

This looks to me like the achilles heel for browser javascript crypto.

Besides the webserver serving the Javascript, it's also important that the BOSH service you use employs SSL/TLS, as well as the XMPP server the BOSH service connects to.

The XMPP community is on a drive to secure all the major public XMPP servers. See the Manifesto created by Peter Saint-Andre and signed by many in the XMPP community.

On http://xmpp.net they let you test the security of servers on the Jabber/XMPP network.

Besides using javascript served via TLS, you can also download the converse.js tarball to your computer, unpack the files and access them directly. The index.html page will open up a page identical to the one on http://conversejs.org.

Browser Javascript being hostile to cryptography

The prevalence of content-controlled code.

If the website fetches javascript from thirdparty websites (for example to serve advertisements), then you not only have to trust the server to serve non-malicious code, but also all those 3rd party servers.

The solution is to not use such a site for secure communications. Clear your cache regularly, Use hosts you trust, or as I've mentioned above, bypass the webserver entirely by using converse.js from your own filesystem.

The malleability of the Javascript runtime.

Javascript is indeed very malleable. Methods and attributes on objects can be replaced during run-time (see Duck Typing and Monkey Patching),

This makes it easy to inject malicious code into Javascript.

However, in order to do this, the attacker's code needs access to those Javascript objects, and fortunately Javascript provides a way of hiding sensitive objects inside closures, thereby preventing an attacker from accessing and manipulating them.

Simply put, a closure defines a limited scope for objects, variables and functions which when used correctly, cannot be accessed from any other Javascript code.

Any variables, objects or functions defined locally inside another (parent) function (with the var) keyword, are not accessible outside of that parent function.

If the parent functions is anonoymous (is not referenced anywhere) or the child function's runtime outlives the parent's, you have a private, enclosed scope with data that cannot be accessed from outside that scope.

All the data in converse.js is encapsulated inside such closure.

The lack of systems programming primitives needed to implement crypto.

Matasano claims that Javascript lacks a secure random number generator RNG.

Without having authoritative knowledge on whether their claim is valid, I'll refer you to this blog post by Nadim Kobeissi (author of Cryptocat), where he claims:

This is in fact not the case. window.crypto.getRandomValues() is a secure random number generator available in JavaScript, and supported by major web browsers.

His blog post is well worth reading, and he also addresses some other criticisms for which I didn't have anything to add.

Conclusion

OTR encryption in converse.js provides increased security and privacy for your chats. Messages are encrypted and not logged or cached

Any Jabber/XMPP services that snoop on your communications by logging your messages (like for example Facebook does) will only have access to the encrypted ciphertext, making it useless for surveillance or exploitation.

Features coming future releases

The current implementation of OTR doesn't have any encryption policy support. With that, I mean the ability to set a policy such as Always encrypt messages.

Currently you need to always manually enable encryption.

Distributed services and self-hosting

There are more and more Free and Open Source (FOSS) self-hosted web-applications becoming available as alternatives to so-called centralized "cloud" solutions.

See for example roundcube, kolab, ownCloud, MailPile, friendica and buddycloud.

Converse.js can be integrated into any of these applications, and since you should ideally host them yourself, you'll have more control on what Javascript is actually being served.

Feedback, patches, donations

All of the work I did on OTR integration was in my free time and free of charge. A gift and my personal contribution towards the movement for providing free communication solutions that are not founded upon surveillance and commercial exploitation of personal data.

All the code is open source and contributions are always welcome, be it in audits, features, bugfixes, documentation, words of encouragement or tips.

If you're so inclined, I'd appreciate any tips at the following bitcoin address: 16FsPqE9DhFTryxrUenpsGX4LJ1TPu8GqS

Read and Post Comments

Converse.js: XMPP instant messaging with Javascript

| categories: backbone.js, instant messaging, xmpp, javascript, strophe.js, plone, converse.js | View Comments

It is now possible to use converse.js, the Javascript from collective.xmpp.chat, in a standalone way, to communicate with any public Jabber account.

Lately I've been spending time refactoring converse.js the Javascript library used by collective.xmpp.chat the instant messaging add-on for the Plone CMS. My goal is to make it usable on it's own, requiring nothing else except an XMPP server to communicate with.

This would enable any website owner to add instant messaging functionality to their website, and due to the federated nature of XMPP, users could chat to any other public XMPP account (once they have been accepted as a contact).

One thing that previously prevented you from using converse.js on its own, was the fact that it made XHR calls to the Plone backend to fetch user data. To fix this, I added vCard support, to converse.js but also to Plone by adding it to collective.xmpp.core.

Last week I reached a significant milestone on the path to this goal, and I'd like to take a moment and share with you.

It is now possible to use converse.js (in a static HTML page) to communicate with Jabber accounts on any public server.

In the demo below, I illustrate this by chatting to a Google user and to a Jabber.org user. In the converse.js page, I'm authenticated with a Jappix.com account and I also use their connection manager to connect to their XMPP server. If you're doing XMPP via HTTP (i.e in the browser), you'll need a connection manager as a bridge to your intended XMPP server. Thanks a lot to the Jappix guys for making their connection manager public!

Note: For detailed fullscreen, make sure that video quality is set to HD, by clicking the gear icon on the video player.

Note

UPDATE: Since this blog post, a production ready version of converse.js has been released. So the following is no longer applicable.

I'm sorry to say that converse.js is not yet 100% ready for primetime as a standalone JS app. There are a few more hurdles (and nice-to-haves) to overcome before we'll achieve this goal:

  • Searching and adding new users still does XHR calls to Plone and should instead query the XMPP server.
  • Service discovery support (e.g so that multi-user chat is only available for servers that support it).
  • Some kind of "Keep me logged in" support when users log in manually.
  • I'm not proud of the CSS and it could probably be improved upon quite a bit.
  • The Jasmine tests are out of date and not passing, also more tests are required.

If you are interested in the project, please contribute by forking the code on github.

Thanks a lot to Alec Ghica and David Ichim from Eau de Web who've made various improvements to both collective.xmpp.chat and converse.js over the past months.

Read and Post Comments

Slides and Video for my talk on XMPP+Plone at Plone Conf 2012

| categories: backbone.js, xmpp, javascript, ploneconf2012, strophe.js, plone | View Comments

The video of the talk I gave at in Arnhem at the 2012 Plone Conf has been uploaded to Youtube. Check it out if you are interested in integrating XMPP with Plone.

Thanks a lot to the person who uploaded this video! After seeing Cillian de Roiste spending much of his free time uploading the videos of the Munich Plone Konf, I realise how much effort this is!

Click here to view all the Plone Conf 2012 videos uploaded so far.

Follow-up info on XMPP-stuff after the conf

In the talk, I mention collaborative editing and showed demos of it (with TinyMCE). Unfortunately soon afterwards I came upon a significant flaw in the current approach. If you are interested in the collaborative editing possibilities for Plone, please read Collaborative editing of HTML with TinyMCE not going to happen soon.

Also since this talk, I've been quite busy improving collective.xmpp.chat to get it production ready. I've made an initial release to Pypi, which was actually supposed to be a 0.1 alpha release (but which I messed up somehow). Please consider collective.xmpp.chat to still be alpha software and not yet production ready! The most notable issue at the moment is poor support for having multiple tabs open. I'm aware of this and will work on a solution soon. Update: Fixed in the latest release: http://plone.org/products/collective.xmpp.chat/releases/0.2a2

I'm also in the process of extracting the Javascript files from collective.xmpp.chat and putting them in their own repository, called converse.js. The idea behind this is to completely decouple the JS from Plone and in doing so allow one to implement/deploy something like collective.xmpp.chat for other non-Plone backends.

That's it for now. Thanks for the support and encouragement from everyone in the Plone community. You guys rock!

Real-time, collaborative applications in Plone by jcbrand

Read and Post Comments

Instant Messaging for Plone with javascript and XMPP

| categories: backbone.js, xmpp, javascript, screencast, strophe.js, plone | View Comments

Late last year Yiorgis Gozadinos who at the time was still working for Jarn, released a video showing his work in integrating XMPP (eXtended Messaging and Presence Protocol) into Zope/Plone.

I had some experience in creating a chat client for Plone before and was inspired by the possibility of porting it to XMPP. In the end it wasn't so much of a port but rather a complete rewrite.

Today, I'm relieved and happy that it's finally in a state where I can show it to you. I've been spending most of my free time the last two months writing collective.xmpp.chat.

It's been both frustrating and fun (mostly fun) and a very valuable learning experience. Like that one week where I rewrote the entire codebase to make use of backbone.js. Aaah, fun times.

Some features already implemented:

Single user chat Roster support (adding/removing contacts) (XEP-0144) Setting presence status (online, offline, away, busy) Custom status messages Multi-user chatrooms (XEP-0045) Chat windows stay open upon page reload Offline storage of messages using HTML5 localstorage (still needs to be secured) Message actions (/me) Setting the topic of a chatroom (/topic)

Because almost all the code is written in Javascript (converse.js is about 1500 LOC), you can even stop the Zope server and still continue to chat. Unfortunately I forgot to show this in the screencast.

If anyone has any questions about the code, how to install, configure and use it, please don't hesitate to contact me. The add-on includes a buildout.cfg file that will set up Zope/Plone, Nginx and ejabberd for you. But please beware, the code is still in a pre-alpha state.

And then lastly, I plan to give a talk at the upcoming Plone Conference in Arnhem. This talk will focus on collective.xmpp.chat, the underlying technologies and how to write XMPP-enabled add-ons.

If you consider this an interesting topic, please go and vote for this talk to ensure that it gets accepted.

Screencast

Note: If you have trouble reading the test, click on the gear icon and set the video to High Definition.

Read and Post Comments