blog

Strophe.js and Converse now support passwordless login with client certificates

| categories: xmpp, converse, sasl, strophe.js, foss, openfire

Introduction

Did you know that x509 certificates, the certificates that webservers use to prove their identity during the establishment of an HTTPS connection, can also be used by a client (like your webbrowser) to prove its identity, and even to authenticate?

I'm talking here about so-called client certificate authentication.

Client certificate authentication is especially popular in environments with high security requirements. They can even be used to enforce 2-factor authentication, if in addition to a client certificate you also require a password. That usecase is however out of scope for this blog post.

With the release of Strophe.js 1.2.8, it's now possible to have passwordless login with TLS client certificates in Converse and any other Strophe.js-based webchat projects.

For Converse, you'll need at least version 2.0.0.

Here's what it looke like:

Logging in with an SSL client certificate

The technical details and background

XMPP and SASL

The XMPP logo

XMPP supports authentication with client certificates, because it uses SASL (Simple Authentication and Security Layer).

SASL provides an abstraction that decouples authentication mechanisms from application protocols.

This means that XMPP developers don't need to know about the implementation details of any authentication mechanisms, as long as they conform to SASL.

Up til version 1.2.7, Strophe.js supported the SASL auth mechanisms: ANONYMOUS, OAUTHBEARER, SCRAM-SHA1, DIGEST-MD5 and PLAIN.

For client certificate auth, we need another SASL mechanism, namely EXTERNAL. What EXTERNAL means, is that authentication happens externally, outside of the protocol layer. And this is exactly what happens in the case of client certificates, where authentication happens not in the XMPP layer, but in the SSL/TLS layer.

Strophe.js version 1.2.8 now supports SASL-EXTERNAL, which is why client certificate authentication now also works.

How do you communicate with an XMPP server from a web-browser?

There are two ways that you can communicate with an XMPP server from a web-browser (e.g. from a webchat client such as Converse).

  1. You can use XMLHttpRequests and BOSH, which you can think of as an XMPP-over-HTTP specification.
  2. You can use websockets.

Both of these protocols, HTTP and websocket, have secure SSL-reliant versions (HTTPS and WSS), and therefore in both cases client certificate authentication should be possible, as long as the server requests a certificate from the client.

I'm going to focus on BOSH and HTTPS, since this was my usecase.

The HTTPS protocol makes provision for the case where the server might request a certificate from the client.

Note

NOTE: Currently the only XMPP server that supports client certificate authentication with BOSH is Openfire, and funnily enough, only Openfire 3. In Openfire 4, they refactored the certificate handling code and broke client certificate authentication with BOSH. I've submitted a ticket for this to their tracker: https://issues.igniterealtime.org/browse/OF-1191

The authentication flow

So this is how the authentication flow works. I'll illustrate how the authentication flow works by using actual log output from Converse

Note

NOTE: My XMPP server's domain is called debian, because I was running it on a Debian server and because naming things is hard. In hindsight, this wasn't a good name since it might confuse the dear reader (that means you).

2016-09-15 12:07:05.481 converse-core.js:128 Status changed to: CONNECTING

Firstly, Converse sends out a BOSH stanza to the XMPP server debian, to establish a new BOSH session.

2016-09-15 12:07:05.482 converse-core.js:128
    <body rid="1421604076"
          xmlns="http://jabber.org/protocol/httpbind"
          to="debian" xml:lang="en" wait="60"
          hold="1" content="text/xml; charset=utf-8"
          ver="1.6" xmpp:version="1.0"
          xmlns:xmpp="urn:xmpp:xbosh"/>
2016-09-15 12:07:06.040 bosh.js:749 XHR finished loading: POST "https://debian:7445/http-bind/"

The above stanza was sent as an XMLHttpRequest POST, and the above XML was sent as the Request Payload.

Strophe.js takes care of all this, so nothing to worry about, but sometimes digging through the internals is fun right? Right?!

2016-09-15 12:07:06.042 converse-core.js:128
    <body xmlns="http://jabber.org/protocol/httpbind"
          xmlns:stream="http://etherx.jabber.org/streams" from="debian"
          authid="fe0ee6ab" sid="fe0ee6ab" secure="true" requests="2"
          inactivity="30" polling="5" wait="60"
          hold="1" ack="1421604076" maxpause="300" ver="1.6">
        <stream:features>
            <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
                <mechanism>EXTERNAL</mechanism>
            </mechanisms>
            <register xmlns="http://jabber.org/features/iq-register"/>
            <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/>
            <session xmlns="urn:ietf:params:xml:ns:xmpp-session">
                <optional/>
            </session>
        </stream:features>
    </body>

So now the XMPP server, debian, has responded, and it provides a list of SASL mechanisms that it supports. In this case it only supports EXTERNAL.

Luckily our webchat client supports SASL-EXTERNAL, so it responds in turn and asks to be authenticated.

2016-09-15 12:07:06.147 converse-core.js:128
    <body rid="1421604077" xmlns="http://jabber.org/protocol/httpbind"
          sid="fe0ee6ab">
        <auth xmlns="urn:ietf:params:xml:ns:xmpp-sasl"
              mechanism="EXTERNAL">dXNlcjAxQGRlYmlhbg==</auth>
    </body>

Now here comes the tricky part. The XMPP server's BOSH servlet, asks the webbrowser (which is establishing the HTTPS connection on our behalf) to give it the client certificate for this user.

The webbrowser will now prompt the user to choose the right client certificate. Once this is done, the XMPP server authenticates the user based upon this certificate.

2016-09-15 12:07:06.177 bosh.js:749 XHR finished loading: POST

2016-09-15 12:07:06.180 converse-core.js:128
    <body xmlns="http://jabber.org/protocol/httpbind" ack="1421604077">
        <success xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/>
    </body>

The XMPP server responds with success and we're logged in!

How to set up client certificate authentication with Converse and OpenFire 3.10.3

Note

NOTE: Thanks goes out to Dennis Shtemberg from Infusion, who initially tested client certificate authentication with BOSH on Openfire and on whose notes the following is based.

1. Install Openfire 3.10.3

The XMPP logo

On Debian(-based) Linux, you can simply do the following:

wget http://www.igniterealtime.org/downloadServlet?filename=openfire/openfire_3.10.3_all.deb
sudo dpkg -i openfire_3.10.3_all.deb

2. Configure Openfire's system properties

Open the admin console: http://localhost:9090/ (where localhost is the host the server is running on)

Navigate to Server > Server Manager > System Properties and add the following properties:

Property Value
xmpp.client.cert.policy needed
xmpp.client.certificate.accept-selfsigned true
xmpp.client.certificate.verify true
xmpp.client.certificate.verify.chain true
xmpp.client.certificate.verify.root true
sasl.mechs EXTERNAL

Make sure the xmpp.domain value is set to the correct host. If you're running Openfire on localhost, then you need to set it to localhost. If you're not using localhost, then replace all mention of localhost below with the xmpp.domain value.

3. Lay the groundwork for generating an SSL client certificate

First, make sure you have OpenSSL installed: aptitude install openssl Then create a directory for certificate files: mkdir ~/certs

Now create a config file called user01.cnf (~/certs/user01.cnf) with the following contents:

[req]
x509_extensions = v3_extensions
req_extensions = v3_extensions
distinguished_name = distinguished_name

[v3_extensions]
extendedKeyUsage = clientAuth
keyUsage = digitalSignature,keyEncipherment
basicConstraints = CA:FALSE
subjectAltName = @subject_alternative_name

[subject_alternative_name]
otherName.0 = 1.3.6.1.5.5.7.8.5;UTF8:user01@localhost

[distinguished_name]
commonName = user01@localhost

The otherName.0 value under subject_alternative_name assigns the user's JID to an ASN.1 Object Identifier of "id-on-xmppAddr". The XMPP server will check this value to figure out what the JID is of the user who is trying to authenticate.

For more info on the id-on-xmppAddr attribute, read XEP-178.

4. Generate an SSL client certificate

  • Generate a self-signed, leaf SSL certificate, which will be used for client authentication.

    • Generate a private RSA key

      openssl genrsa -out user01.key 4096

    • Generate a sigining request:

      openssl req -key user01.key -new -out user01.req -config user01.cnf -extensions v3_extensions

      • when prompted for a DN enter: user01@localhost
    • Generate a certificate by signing user01.req

      openssl x509 -req -days 365 -in user01.req -signkey user01.key -out user01.crt -extfile user01.cnf -extensions v3_extensions

    • Generate PKCS12 formatted certificate file, containing the private key and the certificate. This will be the client certificate which you will log in with.

      openssl pkcs12 -export -inkey user01.key -in user01.crt -out user01.pfx -name user01

      • when prompted for export password enter: user01

5. Install the PKCS12 certificate on your local machine

Double click the pfx file and follow the steps to import it into your machine's keystore.

6. Import the x509 certificate into Openfire

sudo keytool -importcert -keystore /etc/openfire/security/truststore -alias user01 -file ~/certs/user01.crt sudo keytool -importcert -keystore /etc/openfire/security/client.truststore -alias user01 -file ~/certs/user01.crt sudo systemctl restart openfire

Note

NOTE: The default keystore password is "changeit"

7. Create the user associated with the SSL client certificate

Go back to Openfire admin console, navigate to Users/Groups > Create New User and create a new user.

  • Username: user01
  • Password: user01 (This is not controlled by Openfire).
  • Click Create User

8. (When using Java 1.7) Patch Openfire

When trying to log in, I received the following error:

2016.09.08 00:28:20 org.jivesoftware.util.CertificateManager - Unkown exception while validating certificate chain: Index: 0, Size: 0

Turns out the likely cause for this is the fact that I was using the outdated Java version 1.7.

At the time, I didn't know that Java is the culprit, so I patched the following code

If you read the comments in the link above, you'll see there are two sections, with one being outcommented. I swopped out the two sections, and then recompiled Openfire.

After that, client certificate auth worked. The best way to avoid doing this is apparently to just use Java 1.8.

9. Test login with Converse

The Converse logo

Now you're done with setting up Openfire and you can test logging in with Converse.

Download the latest version of Converse from the releases page.

To hide the password field (since the password won't be checked for anyway), you need to open index.html in your text editor and add authentication: 'external to the converse.initialize call.

Then open index.html in your browser.

In the Converse login box, type the JID of the user, e.g. user01@localhost and click login.

Note

NOTE: If things go wrong, pass debug: true to converse.initialize, then open your browser's developer console and check the output. Check especially the XHR calls to http-bind. Checking the output in the Network tab can also be very helpful. There you'll see what Openfire responds to requests to its BOSH URL.

Conclusion

Client certificate authentication is a bit of a niche requirement, doing so with BOSH/HTTP even more so.

However, I expect webchat XMPP clients to become more and more prevalent in the coming years, even on the desktop, for example when packaged with Github's Electron (an Electron version of Converse is planned BTW, based on the fullscreen version inverse.js).

The fact that this works because of SASL-EXTERNAL authentication being added to Strophe.js means that this functionality is not only possible in Converse, but all webchat clients built on Strophe.js (granted that they use version 1.2.8 or higher).

Unfortunately XMPP server support is lacking, with only Openfire supporting this usecase currently, and not yet (at the time of writing) in the 4.0.x branch. To see whether this gets fixed, keep an eye on the relevant ticket


Open Source software and the expection of free labor

| categories: foss, open-source, economics

Over the last few years of starting and then maintaining an open source project that has received a decent amount of attention, converse.js, I've noticed some interesting things about the expectations some people have towards developers who work on FOSS (free and open source software).

Predictably irrational

Book cover: Predictably irrational

People of course love to receive something for nothing. Dan Ariely, in his book "Predictably Irrational" illustrates some of the biases people have when it comes to free stuff. When confronted with the words "free" (as in gratis), people do things that are irrational and are at odds with how a rational actor (the mythical homo economicus) is expected to behave, which is the bedrock upon which most economic theories are based.

The outcome of the various studies Ariely conducted was consistent: when faced with multiple choices, the free option was commonly chosen. With the opportunity to receive something for free, the actual value of the product or service is no longer considered. [1]

“Most transactions have an upside and a downside, but when something is FREE! we forget the downside. FREE! gives us such an emotional charge that we perceive what is being offered as immensely more valuable than it really is.”

Dan Ariely

The biases regarding "FREE!" apply not only to monetary costs, but also to time. We forgo some of our time when we wait in line for free popcorn or to enter a museum on a free-entrance day. We could have been doing something else at that time, so there's a resultant opportunity cost. [1]

Freedom isn't free, it costs folks like you and me

[2]

These biases of course can also come into play when people evaluate free (as in beer) software. In the same way that people didn't take into consideration the cost of the time they spend in trying to get something for "free", people often also don't consider the non-monetary costs of using FOSS.

A common retort that usually surfaces on Slashdot, Reddit or Hacker News whenever a discussion around using a Linux distribution on the Desktop takes place, is “Linux is only free if you don't value your time”.

That's of course completely true. I do value my time, took that into consideration and still concluded that I want to use GNU/Linux and free and open source software.

Using FOSS requires a certain amount of commitment, and it should be clear to the user why they are willing to go that route (freedom from vendor-locking, the ability to control and keep private your data, the ability to modify the code to your liking etc.).

I think people have been hyping the "FREE!" aspect of FOSS way too much.

Software for nothing and your support for free

[3]

Note

Disclaimer

I consider a certain amount of support and maintenance as a requirement for a successful open source project and not something you (as the developer) can ignore.

I try to channel bug reports and feature requests to the Github issue tracker and general support questions to a mailing list, where hopefully other people would also be willing to share the load by answering questions.

So while I complain about people wanting "something for nothing" below, I invariably mean people who write to me directly, instead of on the issue tracker and who are often trying to get me to work on something right away.

So, when considering that many people don't properly evaluate the costs involved in using FOSS, some requests and emails that I sometimes receive start to make sense.

“Please guide me”

One common recurrence, is to be contacted by someone who is integrating converse.js into a project for a paying client, and somehow got stuck. Perhaps they didn't read the docs or perhaps they don't have the requisite technical skills to do the job. These emails sometimes have a pleading, desparate tone to them. Perhaps to instill some sense of guilt or obligation or perhaps just because the person is really desperate and under time pressure.

What gets me every time however, is that as far as I can tell, these are people working for commercial businesses who get paid for the work they do. They then trawl the web looking for hapless FOSS developers to do their work for them for free, or as expressed in the commonly used phrase in these kinds of emails: “Please guide me”.

The novelty and warm fuzzy feeling of altruistically helping strangers solve their problems disappears like mist before the sun when you realise that they're getting paid for the work you're doing for them right now.

And make no mistake about it, maintenance and support for an open source project is work and sometimes even drudgery. The fun part is writing new code or trying out new things, not helping people who can't be bothered to study the documentation.

We need a feature and we hope you'll do it for free

Another common theme is emails where people somehow just assume that I'll implement some feature for them. At first this presumptuousness startled me.

I think it's totally fair to ask when the project is charitable and the people involved don't receive any payment themselves, but that's often not the case.

Instead, the underlying assumption appears to be that I love working on open source projects so much that I'll do it all for free and that I don't have ideas on what to work on next.

Sometimes people qualify their requests by stating that they're a small non-profit. Non-profits do however pay out salaries, don't they?

I'd be willing to reduce my hourly rate when working for a non-profit with a good cause, but I'm most likely not going to do work for free.

The software is free, but the time spent working on it costs money

A nuance that's perhaps lost on many people, is that I have often worked on converse.js for money. There was a rather long "bootstrapping" phase in the beginning where the project wasn't good enough for anyone to actually use or pay money for further development, but after the project stabilized I started getting small paying gigs of custom development on converse.js.

In all cases I made it clear that the Mozilla Public License forces me to open source any changes I made to the covered files, and therefore the work I did for these paying customers (bless their hearts) was open sourced as well.

The point is that while the software is free (as in beer and as in speech), the time spent working on it costs money.

Either someone else pays me to spend my time working on it, or I end up paying by doing something for free while I might be getting paid doing something else (opportunity costs) or by taking time away from other activities.

FOSS development costs money, either the developer is commissioned, or they pay for it themselves (perhaps unwittingly).

Doing work for free devalues it and takes the piss out of actual paying customers

The last point I'd like to make, is that by taking on these requests to do free work for commercial entities (and non-profits), I'm not only devaluing my work, but I'm also disincentivising paying customers (which includes non-profits).

After all, why would anyone pay me to do anything if I'm so eager to please that I'll do it all for free?

The only reason I could see to do that, is to get that mythical "exposure" that's often also sold to web and graphic designers.

The Oatmeal comic: "Exposure"

[4]

So what do you do if you need work done and can't pay for it?

Free and open source software is a beautiful, world-changing and paradigm shifting idea. However, software developers, like all people, need to be paid for their work, also when they work on FOSS.

If you can't pay for software development, then you can still try to incentivise FOSS developers in other ways, but be aware that it'll be more difficult.

One important lesson that I'm glad I learned early in life, is that when you're asking someone to do something for you, then you need to explain to them why it's in their best interests to do so.

People inherently look out for themselves. It's perfectly natural and doesn't necessarily mean they're selfish to the point of being anti-social, it just means that they need to take care of themselves and that they can't expect other people to do it for them or to even have their best interests at heart.

So when desiring something from someone, such as their help, the best approach is to explain to them what's in it for them

For example, if you want a friend to help you out with something, let's say to join a beach cleanup project, you don't tell them why it'll be good for you, you explain to them that it'll be an opportunity to chat, to meet new people, to go for a swim and to have the enjoyment of a clean unspoiled beach.

This is simple stuff, but many people apparently don't know this.

So if you want someone to help you with a software project, explain to them why it would be in their best interest. If you can't find a reason why, then perhaps it's actually not in their best interest and you need to create an incentive for them.

Money works pretty well as an incentive, but there are other ways as well. One sure-fire way to build up goodwill and gratitude (that might translate into more help and assistance) is to contribute. If you can't write code, fix typos in the docs, evangelize the project or contribute in other areas which you have some expertise, like translations, design, UX, helpful feedback etc.

FOSS development is a community effort and a team sport. There'll always be people who try to take more than they give, but on average, humans are matchers. When given something, they want to reciprocate and give back. Keep that in mind when you're trying to get something for nothing.

References

[1](1, 2) Wikipedia article on Predictable Irrational
[2]Sung to the tune of Freedom isn't free
[3]Sung to the tune of "Money for nothing" by Dire Straits.
[4]From The Oatmeal

Converting Youtube videos into mp3

| categories: howto

If you're using Linux or a Mac, then there are two neat commandline tools which you can use to download and convert Youtube videos into mp3 files. They are Youtube-dl and FFmpeg.

Installation

Installing youtube-dl

The download and install instructions for youtube-dl are here.

As a quick note, on Mac you can run:

brew install youtube-dl

And on GNU/Linux:

sudo curl https://yt-dl.org/downloads/2015.11.27.1/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl

For NixOS fans, it's available via nix-env.

Installing FFmpeg

Many Linux distros come with FFmpeg installed I believe.

If not, you can use your distros package manager. For example, on Ubuntu:

sudo apt-get install ffmpeg

Note

In some older versions of Ubuntu (and probably other distros), you should use avconv instead of ffmpeg. AVConv was simply a fork of FFmpeg and the two projects have subsequently been merged back into FFmpeg again.

On Mac you can use homebrew:

brew install ffmpeg

Downloading the video

To download a Youtube video, you simply run the following line (making sure to replace the URL with the one of the video you want to download):

youtube-dl https://www.youtube.com/watch?v=DLzxrzFCyOs

Converting video to mp3

To now convert the mp4 video to mp3, you can use FFmpeg (comes standard with most Linux distros I believe).

ffmpeg -i Rick\ Astley\ -\ Never\ Gonna\ Give\ You\ Up\ \[HQ\]-DLzxrzFCyOs.mp4 Rick\ Astley\ -\ Never\ Gonna\ Give\ You\ Up.mp3

A very common grammatical mistake that programmers make

| categories: grammar, programming, language, english

I recently read a fascinating profile about a guy who made about 50 000 edits to Wikipedia to correct a relatively obscure grammatical mistake. The mistake of using "comprised of" when one should be using "composed of" or "consists of". He is the ultimate wikignome. A name given to people who spend most of their time on the site making small corrections and fixes.

This reminded me of a grammatical mistake that programmers frequently make. I notice it in source code, on IRC, in mailing lists and in documentation.

Here are a few examples from actual code, which I found by using "grep":

def cleanupInstance(self):

def teardownSite(self):

def setup_test_class(self):
case "default":
  // setup event handlers

What is the grammatical mistake in all these examples?

Student programmers at the Technische Hochschule in Aachen, Germany in 1970 (Wikipedia)

Programmers making grammar mistakes (1970)

The compound nouns setup, teardown and cleanup are confused with the verbs set up, tear down and clean up.

In the above examples, they are used as verbs but spelled as nouns.

I ask you to set something up for me. I don't ask you to setup something for me.

Note

EDIT: Some readers misunderstood the point and thought I implied that there must always be a clause between the two words in the verb. This is not true. The point is that they are written separately.

Example: "Please set up the server for me:.

However, when I want to refer to the task you did (the noun), I refer to the setup.

When used as an adjective, setup also stays one word:

Go to the setup menu. [adjective]
The server's setup is not yet done. [noun]

This mistake happens mostly with setup, however here are similar words which also often get used incorrectly:

turnover:

Turn over the documents.
Our company increased our yearly turnover.

handover:

Hand yourself over to the police.
The hostage handover went smoothly.

cleanup:

Clean up your room.
The cleanup took hours.

teardown:

Tear down this wall!
This test needs a teardown method.

Note

EDIT: Here's some more:

* Callback <=> Call back
* Checkout <=> Check out
* Layout <=> Lay out
* Login <=> Log in
* Logout <=> Log out
* Lookup <=> Look up
* Plugin <=> Plug in
* Popup <=> Pop up
* Runaround <=> Run around
* Shutdown <=> Shut down
* Showtime <=> Show time
* Startup <=> Start up
* Takeoff <=> Take off
* Workout <=> Work out
* Writedown <=> Write down

Another interesting case is "maybe". It can be used as an adverb (one word), noun (one word) and verb (two words, a modal verb "may" and the main verb "be"):

At the moment it's definitely still just a maybe. (noun)
I'll maybe come over tonight. (adverb)
He may be waiting for us. (verb)

What about something like download?

We don't say load it down. We say download it.

That's different. In the first category of compound nouns (setup, teardown, handover), the part of the word that denotes the act is first, followed by an adverb such as up, down, over or under:

setup = set [verb] + up [adverb]

In the second category, the acting word is second (load, grade, take, throw) and the first word denotes position:

download = down [preposition] + [load] verb or noun

Other examples of the second category are:

  • upgrade
  • downgrade
  • undertake
  • overthrow

Note

EDIT: Here's some more:

* understand

These do not get split up.

I hope I made things clear and that programmers will now undertake to clean up their code by removing these grammatical mistakes. ;)

Note

EDIT: I've thought about this some more, especially in light of some of the comments provided by readers. I can see how using the word "setup" as a type of shorthand for "set up", to avoid unnecessary underscores or capitals, can be useful and valid. If you undersand the distinction and prefer writing it as one word, then go for it. I think however what's important is for people to at least understand that it's an exception and not the general rule outside of computer science.


Sprint Report: Merging Mockup and Patternslib

| categories: mockup, javascript, patternslib, austria, sprint, foss, plone

Alpine City Sprint, 21 to 26 January 2015

This is a report on what I did at the Plone Alpine City Sprint organized by Jens Klein and Christina Baumgartner in Innsbruck between 21 and 26 January 2015.

Firstly, I want to thank to them for organizing and hosting a great sprint and for being such friendly and generous hosts.

My goal for this sprint was to work on merging the Patternslib and Mockup Javascript frameworks and I'm happy to report that very good progress was made.

Long story short, the merge was successful and it's now possible to use patterns written for either project within a single framework.

Before I expand on how this was achieved during this sprint, I'll first provide some necessary background information to place things into context and to explain why this work was necessary.

What is Patternslib?

Patternslib brings webdesign and development together.

Patternslib's goal is to allow website designers to build rich interactive prototypes without having to write any Javascript. Instead the designer simply writes normal HTML and then adds dynamism and interactivity by adding special HTML classes and data-attributes to selected elements.

For example, here is a Pattern which injects a list of recent blog posts into a element in this page:

Click here to show recent blog posts

The declarative markup for this pattern looks as follows:

<section id="alpine-blog-injected">
    <a href="#portlet-recent-blogs"
       class="pat-inject"
       data-pat-inject="target: #alpine-blog-injected">

        Click here to show recent blog posts
    </a>
</section>

The dynamic behavior comes from the fact that I have the Patternslib Javascript library loaded in this page.

On pageload, Patternslib scans the DOM looking for elements which declare any of the already registered patterns. If it finds such an element, it invokes a Javacript module associated with that pattern.

In the above example, the <a> element declares that it wants the Inject pattern applied to it, by virtue of it having the pat-inject HTML class.

Patterns are configured with specific HTML5 data properties. In the example above, it is the data-pat-inject property which specifies that the target for injection is the #alpine-blog-injected element in the current page and the content being injected is specified by the href attribute of the <a> element. In this case, the href points to an anchor inside the current page, but it might just as well be a link to another page which will then get injected via Ajax.

More information on how to configure pat-inject can be found on the patternslib website.

Each pattern has a corresponding Javascript module which is often just a wrapper around an existing Javascript library (such as Parsley, Select2 or Pickadate) that properly defines the configuration options for the pattern and properly invokes or applies the 3rd party library.

So, in order to do all this, Patternslib can be broken down into the folowing components:

  • A registry which lists all the available patterns.
  • A scanner which scans the DOM to identify declared patterns.
  • A parser which parses matched DOM elements for configuration settings.
  • The individual Javascript modules which implement the different patterns.

What is Mockup?

Mockup, declarative interaction patterns for Plone..

Mockup was inspired by, and originally based upon, Patternslib and was meant to bring the power of declarative interaction patterns to Plone.

When Mockup started, Patternslib was undergoing significant refactoring and development and it was decided that Mockup should fork and go its own way.

What this means is that the 4 different components mentioned above, were all changed and due to these changes the Mockup project diverged from Patternslib and started developing in a similar but different direction.

So what's the problem?

While Mockup was being developed for the upcoming Plone 5 release, we at Syslab continued using and improving Patternslib in our projects.

Syslab built an intranet for the Star Alliance, which was based on a prototype design by Cornelis Kolbach, the conceptual creator of Patternslib. This design became the inspiration and blueprint for the Plone Intranet Consortium (PIC), which consists of 12 companies working together in a consortium to build an intranet solution on top of Plone.

So, the PIC are building a product using Patternslib, while Plone 5 itself is being built with Mockup, an incompatible fork of Patternslib.

This was clearly a bad situation because we now had:

  • Two incompatible Javascript frameworks being used with Plone.

    Not only were the two frameworks incompatible in the sense that patterns written for the one don't work on the other, but they could also not be used on the same page since the two implementations would compete with one another in invoking Javascript to act upon the same DOM elements.

  • Duplication of effort

    The same or similar patterns were being developed for both frameworks, and when one framework had a pattern which the other wanted, it could only be used after being modified such that it couldn't be used in the original framework anymore.

  • A splitting of the available workforce.

    Developers were either working on Mockup or Patternslib, but almost never on both, which meant that the expertise and experience of developers wasn't being shared between the two projects.

How could this be fixed?

To solve the 3 main problems mentioned above, we needed to merge the common elements of Mockup (specifically the registry, scanner and parser) back into Patternslib.

This will allow developers from both projects to work on the same codebase and enable us to use patterns from both projects together.

At the Alpine City Sprint in Innsbruck, I worked on achieving these goals.

Changes brought in by Mockup

After the fork, Mockup introduced various changes and features which set it apart from Patternslib.

In order to merge Mockup back into Patternslib, I studied these changes and with the help of others came up with strategies on what needed to be done.

Here are some differences and what was done about them:

Mockup allows patterns to also be configured via JSON, whereas Patternslib used a keyword:argument; format

A week before the sprint I added JSON parsing ability to the Patternslib parser, thereby resolving this difference.

Leaves first parsing versus root first parsing

Mockup parses the DOM from the outside in ("root first"), while Patternslib parses the DOM from the inside out ("leaves first").

According to Rok Garbas, the creator of Mockup, the outside-in parsing was done because it reduced complexity in the scanner and the individual patterns.

Wichert Akkerman who originally wrote the Patternslib scanner however provided IMO a very good reason why he chose "leaves first" DOM scanning:

If I remember correctly there are several patterns that rely on any changes in child nodes to have already been made.This is true for several reasons: 1) a pattern may want to attach event handlers to child nodes, which will break of those child nodes are later replaced, and 2) child nodes changing size might impact any measurements made earlier.

Indeed, while refactoring the Mockup code during the merge, I ran into just such a case where a pattern couldn't yet be initialized because patterns inside it weren't yet initialized. By turning around the order of DOM scanning, this problem was resolved and the code in that pattern could be simplified.

So, now after the merge, scanning is "leaves-first" for Mockup patterns as well.

Mockup patterns are extended from a Base object, very similar to how Backbone does it

Patternslib patterns on the other hand are simple Javascript objects without constructors.

The patternslib patterns are conceptually very simple and more explicit.

However, what I like about the Mockup approach is that you have a separate instance with its own private closure for each DOM element for which it is invoked.

After merging, we now effectively have two different methods for writing patterns for Patternslib. The original "vanilla" way, and the Mockup way.

The Mockup parser was completely rewritten

The Mockup parser looks nothing like the Patternslib one and also supports one less configuration syntax (the so-called "shorthand" notation).

This was one piece of code which could not be merged within the time available at the Alpine City Sprint.

So currently we still have two different argument parsers. The Patternslib parser needs to be configured much more expicitly, while the Mockup one is more implicit and makes more assumptions.

Merging these two parsers will probably have to be done at some future sprint.

There are some other more minor differences, such as that every Mockup pattern is automatically registered as a jQuery plugin, but merging these was comparatively easier and I'll won't go into further detail on them.

What I did during the Alpine Sprint

So, in summary:

I refactored the Patternslib registry, scanner and some core utils to let them handle Mockup patterns as well. Luckily the eventual patch for this was quite small and readable.

I changed the Mockup Base pattern so that patterns derived from it now rely on the registry and scanner from Patternslib.

I fixed lots of tests and also wrote some new tests.

This whole task would have been much more difficult and error prone if either Patternslib or Mockup had fewer tests. The Mockup team deserves praise for taking testing very seriously and this allowed me to refactor and merge with much more confidence.

What else is there to still be done?

Being able to use patterns from both projects and merging most of the forked code was a big win, but there are still various things that need to be done to make the merge complete and viable.

I've ranked them from what I think is most important to least important.

1. Update the documentation

Currently documentation is scattered and silo'd in various places (the Patternslib website, the Plone Intranet developer docs and the Mockup developer docs).

The Mockup docs are now out of date ater this merge and need to be brought up to date on these recent changes.

The Patternslib docs are less of a problem because they don't have to deal with Mockup (which can now be seen as an enhancement suite for it), but they can definitely still be improved upon, specifically with an eye on Mockup developers who will start relying on them.

The Plone Intranet consortium also has a useful walkthrough explaining how to create a Patternslib pattern from scratch..

2. Devise a way to also use the Patternslib parser for Mockup patterns

As mentioned, the Mockup patterns still use their own argument parser.

Letting them use the Patternslib's parser will either require extending the Mockup Base pattern to configure the Patternslib parser on a pattern's behalf or instead doing it explicitly in each of the individual patterns.

3. Decide on coding and configuration standards

Unfortunately the coding standards between the two projects differ significantly.

  • The Mockup developers use camelCase as declarative argument names while Patternslib uses dash-separated names.
  • The Mockup source code uses 2 spaces for indentation, Patternslib uses 4 spaces.

4. Remove unnecessary, duplicated patterns

Both projects have patterns for modals, injection, select2 and file upload. These should be merged to get rid of duplication.

5. Move generic patterns (e.g. pat-pickadate, pat-moment etc.) out of Mockup

Mockup has some generic patterns which might also be useful as Patternslib core patterns, in which case they should ideally be moved there.

Conclusion

The sprint was a big success and I'm very happy that all the work has already been merged into the master branches of the matternslib, mockup and mockup-core repositories.

Core code from Mockup is now succesfully merged back into Patternslib and we can use patterns from both projects in the same site.

I had a lot of fun working with talented and motivated software developers and had a lot of opportunities to improve my German.

Thanks again Jens and Christine for organising a great sprint and I look forward to doing such a sprint again!

Innsbruck, the alpine city

« Previous Page -- Next Page »