blog

Converse 4 released

| categories: xmpp, converse, omemo, community

After more than 7 months of active development, Converse 4 has finally been released.

Converse is an open source XMPP-based chat client written in JavaScript and which runs in your browser.

This release contains lots of highlights, including rewriting the UI to use Bootstrap 4, support for OMEMO Encryption of private messages, message corrections and file-sharing via HTTP file upload.

XMPP is an IETF standardized messaging and presence protocol with multiple independent server and client implementations.

Unlike other popular open source teamchat applications like Mattermost and Rocket.chat, Converse doesn't depend on any particular server (e.g. backend) application. Any XMPP server which supports the relevant extensions (aka XEPs) will do.

Converse is 100% front-end JavaScript and CSS. The only backend you need is an XMPP server, which you can either set up and host yourself or you can sign up on an existing one.

XMPP's killer feature

Picture of the USS Enterprise, defending the federation, I think... I'm not really a trekkie

XMPP allows for something called federation. Even if you've never heard the word being used in this context, you already know what it means if you've used email.

When you sign up for an email account from one provider (for example Gmail, Fastmail or Kolab), you can still send emails to people who have accounts at other providers. That's because Email is also federated.

A federated protocol allows you to communicate with user-accounts on foreign hosts. The network is decentralized between many providers (who all adhere to a common protocol), instead of being controlled by a single centralized behemoth (like Facebook, Twitter or LinkedIn).

Federation is getting quite a lot of renewed attention lately, in large part due to the hype around the federated social networking site Mastodon which is based on the recently published ActivityPub standard.

You're not forced to federate when running an XMPP server for your organisation, friends or family, and sometimes it makes sense not to.

However it's in my opinion the killer feature of XMPP and one of the main reasons why I'm working on Converse.

Federation implies user-freedom and user-choice, and it's one thing that Slack or Discord simply cannot do. Not because it's not technically feasible, but because their business models, based on user lock-in and control, don't allow it.

It's also something which the open source Slack-alikes like Mattermost and Rocket.chat don't have.

I believe people should have the freedom to use the chat app of their choice, while still being able to effectively communicate with their contacts who prefer to use a different chat app.

XMPP and federation can make this a reality.

In a world of federated chat apps, some people will use a commandline client like Poezio, a desktop client like Dino or a mobile client like Conversations (or a combination of all three).

There will also be a growing need for web-based chat clients, and that's were Converse comes in.

Federation not only provides choice between chat clients, it can also foster inter-departmental or inter-organisational communication.

In Germany, where I currently live, there is a lot of talk of digitization of government. Government bureaucracy often consists of various different departments who need coordinate and send resources around.

The same of course applies to large enterprises.

A decentralized and federation communications protocol can play a large role here, and I'm not just talking about chat apps.

The Salut à Toi project has shown how you can create a decentralized and federated issue tracker, while Movim has blogging and microblogging capabilities, all based on XMPP.

Converse's history

Converse was originally developed at Syslab.com as part of a web-based intranet for the Star Alliance airline group.

Back then it was developed as a Gtalk-like popover chat, like you still see on LinkedIn.

After open sourcing it, my initial goal was to make it as configurable and capable as possible, so that it could be used in any website in almost any conceivable manner.

A powerful plugin architecture and API was added, which allows you to add or remove features and change just about any behaviour.

Over time, as the project matured and people started using it, it became clear that there are many more usecases for a webchat than simply popover chats.

Full-page teamchat apps like Slack also became spectacularly successful.

I subsequently updated Converse to have different "view modes", which enable you to use it in different ways, depending on your needs.

The focus also shifted slightly from presenting it merely as something to be integrated into an existing website, to showing that it can be an independent chat client in its own right.

There is the "overlay" mode, which matches the original UX design:

A screenshot of Converse with overlayed chatboxes

A screenshot of Converse with overlayed chatboxes

Then there are the "embedded" and "mobile" modes:

A screenshot of Converse in embedded mode

A screenshot of Converse in embedded mode

A screenshot of Converse in mobile mode

A screenshot of Converse in mobile mode

And lastly there's the "fullscreen" mode, more similar in UX to teamchat applications like Slack and Discord.

A screenshot of Converse in fullpage mode

A screenshot of Converse in fullpage mode

All these modes use the same JavaScript and CSS, and you can switch between the view modes. Currently it's not possible to do so without reloading the page, but we'll work on making that possible as well.

Working on Converse fulltime

In the beginning of this year, I decided to drop other work commitments and to dedicate this year fully to working on Converse and to see where that leads me.

This was in many ways a daunting leap into the unknown, because I knew I would now have to find more ways to try and earn a living off this work.

I've been doing consulting and custom development around Converse for a few years, but never exclusively. I also wasn't sure what business model I should adopt.

Consulting? Proprietary plugins? Hosting? Sponsorships?

I decided to first just focus on building the product, to make Converse better, and not to worry about making money just yet.

In part this is due to my bias as engineer, and being out of my comfort zone when trying to monetize something.

But I also believe that a compelling product, that users love, can go a long way to helping you come up with a sustainable and feasible business model. I hoped that along the way people might appear who would help me realize this vision.

It's the "if you build it, they will come" approach.

During this time, various people did in fact reach out to me and some of them funded further development.

I also got enough consulting and development gigs to keep the lights on and to keep me going.

Marc Laporte from Wikisuite funded many hours of work moving the UI to Bootstrap 4, and then some more.

Converse will become part of Wikisuite, together with the OpenFire XMPP server.

He gave me valuable advice on how to make a living off open source software and encouraged me to make the leap.

I also got a call from Happy Dev, who needed a federated webchat solution for their Startin' Blox project (more on that in future posts). We're doing some interesting work for them, which addresses some pain points with XMPP and will be of benefit to the larger community.

Nicolas Vèrité, of Nayego funded responsiveness improvements while individuals like Christian Weiske and Rafael Munoz generously contributed funds as well.

Some people also backed me on Patreon and Liberapay.

Every gig or donation I got was for me a vote of confidence, and gave me the opportunity to stay the course and continue down this path.

Besides monetary contributions, many people contribute features and bugfixes, and hang out in the Converse chatroom, answering questions, providing their expertise and generally making it a more lively, interesting place.

In many ways this release would not have been possible without all of their contributions, and I therefore dedicate it to everyone who has contributed money, time or energy (all interchangeable actually) to this project and therefore has contributed to its success. Thank you!

The archetypal nature of free software communities

Over the last year I've witnessed the Converse project becoming a community and not just a collection of software. It's a subset of the larger XMPP community and one that overlaps with various other XMPP and free software related communities.

This has been for me one of the most magical and satisfying things about being involved in open source.

Community is archetypally feminine [1], in the sense that it's something that cannot be enforced, demanded or created out of domination, control or rule-making (which are archetypally masculine attributes). Instead, it's something that arises seemingly spontaneously, after an unknown period of gestation and only after the groundwork has been laid and perhaps after enough loving care and attention has been provided. Its organic, emergent nature means that it doesn't strictly adhere to top-down schedules and deadlines.

My wife and I, both interested and fascinated by Jungian archetypes, are amazed at the archetypally feminine aspects of her pregnancy, which is soon coming to an end (and new beginning).

As much as we as a society try to control, monitor and manage a pregnancy, if you decide to have a natural birth, then you're waiting for it to happen on its own accord, and aren't able to impose your will and deadlines on it.

In some ways the same is true for a software commons. A commons is a shared resource which no-one has exclusive control or ownership of. This lack of ownership and exclusive control makes people resilient against the software being used as a tool for domination and exploitation. It also means that the project (i.e. the software and its surrounding community) is more organic.

This coming together between software as inanimate artifact, and community as a living breathing organic hive-mind, brings to mind a Taoist union of Yin and Yang, the archetypal feminine and masculine.

Incidentally, the Converse logo is a stylized Yin Yang symbol. Many people don't know this because I did such a poor job of designing it. I'm still looking for a better logo design which encapsulates these principles of harmony between community and software.

What does the future hold for Converse?

This has turned out to be a relatively strange blog post, where I wanted to originally write about the features of the 4.0 release and instead took an ever more esoteric detour.

In any case, let's try to salvage what's left of the original intent by closing off with what's in the pipeline for future releases.

Message markup

Converse still doesn't support markdown-like syntax highlighting. This is a top-priority for future releases.

Voice messages

With voice messages I mean sending voice recordings, not audio streaming (although that might also come sometime).

My wife, friends and family use voice messages (via Conversations) a lot and I believe they'd be a useful addition to Converse as well.

Email notifications

One area where XMPP needs to improve is sending (email) notifications when you're offline and mentioned in a groupchat. XMPP's groupchat presence rules have made this difficult. Together with Happy Dev and Matthew Wild from Prosody I'm working on a potential solution.

Some of the code for this solution is already in Converse, and we'll submit a specification for standardization (a so-called XEP) once the time is right.

Making Converse a progressive web-app

There are various features of so-called progressive web apps which will make Converse a much better client, such as:

  • Push notifications even when Converse is not open.
  • Using IndexedDB for caching, thereby avoiding storage limitations of localStorage and sessionStorage.
  • Offline support for editing and sending messages.

Persistent login

Other webchat clients like Mattermost and Rocket.chat can keep you constantly logged in via cookies or perhaps JWT.

XMPP-authentication isn't cookie-based and it's not safe to store user credentials in the browser cache. This means that users often have to login anew when they open Converse in a browser tab.

The Credential Management API might help here, but it's still Chrome-only and I haven't studied it enough to know whether it's a solution.

Another approach is to allow Converse to also use authentication cookies by adding support for them to XMPP servers.

Matthew Wild has already added support for cookie authentication to Prosody but more work needs to be done to get this working on conversejs.org.

OAuth-based login

Converse already supports OAuth-based login, but it's not yet deployed to conversejs.org and more work can be done to fetch and show your user profile from the OAuth provider.

Let's build this together

There are many more features and improvements planned, too many to list here.

The health and sustainability of this project depends in large part on its community of users and contributors.

The community is invaluable in finding bugs, making feature suggestions and doing quality control that would otherwise not be possible given the circumstances.

Another important requirement for the long-term health and viability of the project is funding.

I and others are spending much of our own time working on Converse, but to make this project sustainable we need more funding. Either through donations (e.g. Patreon), funded feature requests or by using my services as an XMPP and Converse consultant and developer.

Another option is allowing an employee to spend some of their paid time on helping the project with bugfixes and other contributions.

This applies in general to open source projects. Companies derive immense value from them, and the vast majority don't have wealthy corporate backers like Facebook and Google.

I also intend to soon start offering hosted solutions for organisations who want to use Converse as their preferred teamchat solution.

Please reach out to me here if you're interested.

References

[1]

I've been thinking a lot about archetypes (and Jungian-analysis) the last while, in part due of various books I've read that deal (partly) with the topic.

Recommended reading, if you're interested:


2018 Gulaschprogrammiernacht and organizing sprints for XMPP

| categories: xmpp, converse, foss, omemo, plone

Recently I attended the Gulaschprogrammiernacht for the first time.

It's a hacker/maker event in the Zentrum für Kunst und Medien (Centre for Arts and Media) in Karlsruhe, Germany.

AFAIK it's organized by the local chapter of the infamous Chaos Computer Club.

I heard about it from Daniel Gultsch on Twitter. It sounded like fun, so I decided to attend and spend the time adding OMEMO support to Converse.

Guus der Kinderen and I intended to organize an XMPP sprint for that weekend in Düsseldorf, but we were cutting it a bit fine with the organization, so I hoped that we could just shift the whole sprint to GPN.

Unfortunately Guus couldn't attend, but Daniel and Maxime Buquet (pep) did and I spent most of the event hanging out with them and working on XMPP-related stuff. The developers behind the Dino XMPP client also attended and hung out with us for a while and there was someone working on writing an XMPP connector for Empathy in C++.

XMPP hackers at Gulaschprogrammiernacht

Maxime worked on adding OMEMO support, to Poezio, and Daniel provided us with know-how and moral support. Daniel worked mainly on the Conversations Push Proxy.

We had some discussions around the value of holding regular sprints and I told them about my experience with sprints in the Plone community.

The Plone community regulary organizes sprints and they've been invaluable in getting difficult work done that no single company could or would sponsor internally. To me it's a beautiful example of what's been termed Commons-based peer production.

The non-profit Plone foundation provides funding and an official seal of approval to these sprints, and usually a sprint has a particular focus (such as adding Python3 support). Sprints can range from 3 people to 30 or more.

One difference between the Plone and XMPP communities, is that Plone is a single open source product on which multiple companies and developers build their businesses, whereas XMPP is a standardized protocol upon which multiple companies and developers create multiple products, some open source and some closed source.

In both cases however there is a single commons which community members have an incentive to maintain and improve as they build their businesses around it.

Another difference is between the Plone Foundation and the XMPP Standards Foundation. The XSF, for better or worse, interprets its role and function fairly strictly as being a standards organisation primarily focused on standardising extensions to XMPP, and less on community building or supporting software development.

Despite these differences, I still consider sprints a great way to foster community and to improve the extent and quality of XMPP-related software and documentation.

There is an interesting dynamic between cooperation and competition in both the Plone and XMPP communities. Participants compete with one another but they also have the shared goal of maintaining and growing a healthy software ecosystem.

Maxime was particularly excited by our discussion and very quickly put word into action by planning and announcing an XMPP sprint in Cambridge, UK in August.

There's still time to vote on the date of the sprint and to suggest topics.

Hopefully this will be first of many more sprints and communit events.

Unmanned laptops at the Gulaschprogrammiernacht

Slack's bait and switch

| categories: xmpp, converse, foss, slack

Slack has finally decided to close down their IRC and XMPP gateways.

True to form, you can only read their announcement if you already have a Slack account and are logged in to a workspace.

Here's the gist of their announcement:

As Slack has evolved over the years, we've built features and capabilities —
like Shared Channels, Threads, and emoji reactions (to name a few) — that the
IRC and XMPP gateways aren't able to handle. Our priority is to provide a
secure and high-quality experience across all platforms, and so the time has
come to close the gateways.

They're of course being economical with the truth here.

Perhaps their XMPP gateway can't handle "Shared Channels" and "Threads", but that's because they purposefully stopped working on it.

A "Shared Channel" simply means a chatroom which people from outside your workspace can participate in. If a workspace is mapped to a members-only chatroom, then making something a shared channel simply means updating the members list or making the chatroom open (so anybody can join it).

Threads can be implemented by adding a <thread> element in the message stanza, as documented in XEP-201.

And emoji... there's nothing in XMPP that prevents people from sending emoji.

Note

UPDATE: Several readers have pointed out that "emoji reactions" and emoji are different things. Emoji reactions can be tacked to a particular message.

Still, there's nothing fundamental about XMPP that prevents emoji reactions, and work is currently underway to add support for them.

The protocol is designed to be eXtensible (hence the X in XMPP) and new features are continuously being added.

Classic bait and switch

We all know the real reason Slack has closed off their gateways. Their business model dictates that they should.

Slack's business model is to record everything said in a workspace and then to sell you access to their record of your conversations.

They're a typical walled garden, information silo or Siren Server

So they have to close everything off, to make sure that people can't extract their conversations out of the silo.

We saw it with Google, who built Gtalk on XMPP and even federated with other XMPP servers, only to later stop federation and XMPP support in favour of trying to herd the digital cattle into the Google+ enclosure.

Facebook, who also built their chat app on XMPP at first allowed 3rd party XMPP clients to connect and then later dropped interoperability.

Twitter, although not using or supporting XMPP, had a vibrant 3rd party client ecosystem which they killed off once they felt big enough.

Slack, like so many others before them, pretend to care about interoperability, opening up just so slightly, so that they can lure in people with the promise of "openness", before eventually closing the gate once they've achieved sufficient size and lock-in.

On Federation

When we talk about "federation" in networks, we mean the ability to communicate between different service providers.

For example, email is federated. You can set up your own email server, and then send emails to people with their own email servers, or to people with Gmail or Yahoo! accounts.

You can email any other email address in the world, regardless of where that email address is hosted.

If email never existed, and a company like Slack today would come out with this brand new concept of "Electronic Mail", let's call it digimail, do you think they would standardise the digimail protocol and allow you to send messages to other digimail purveyors?

We all know the answer to that. They won't, and neither would Google, Microsoft or Facebook.

Heck, Facebook is actively trying to replace email since years.

The reason email is federated, is because it was developed before surveillance capitalism was a thing and because it was established and entrenched long before these companies came around.

There's a reason why your email address is still the de facto way to sign up for any service on the web (sometimes with one or two degrees of separation), and it's because of federation.

XMPP is designed to allow federation. Think about that. Instead of having to sign up to various different chat providers, all which try to lock you in and monetize your conversations, you could instead have one chat account, and use that to chat with anybody else, regardless of which chat provider they are using.

Alas, that's the dream, but because XMPP came much later to the scene, it didn't develop the critical mass as email has, and here we are. With dozens of chat apps, all non-interoperable and closed off.

What would it take for XMPP to take off?

One of the sad things that has come out of Slack's meteoric rise to success, has been how many free and open source projects have jumped over to using it (after previously using IRC or XMPP).

In so doing, they have closed off their discussions from search engines and they prevent people from accessing their past archives.

Slack has many cool features, and they work very well, I'm not going to deny it.

However, the XMPP Software Foundation has done a lot of work in recent years to enable protocol extensions that provide features that people have come to expect from chat applications, for example:

Unfortunately XMPP clients have been lagging far behind in various respects.

One of the main problems is funding. The modern digital economy is largely set up around surveillance capitalism and user lock-in.

So attempts to create software that doesn't follow these precepts, often end up unfunded or underfunded.

However, our "weakness", is also our strength.

XMPP clients, and the XMPP network can provide something that Slack never can. Federation, free and open software, interoperability, extensibility and user choice.

Converse

For the last few years I've been working in my spare time on making a JavaScript XMPP chat client, called Converse.

Originally the idea was to make a Gtalk like chat client that you integrate in your website, and it can still be used like that.

A screenshot of Converse as overlayed chatboxes

However, in the last year I've updated it further so that it can also be used as a fullscreen application, like Slack is used.

You can try the fullscreen version at https://inverse.chat/

A screenshot of Converse as a fullscreen application

If you have no-one to chat to, then come join the discuss@conference.conversejs.org chat room.

This link will take you directly there (via inverse.chat): https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org

Converse still lacks lots of features that Slack has, but that's not because XMPP itself can't support those features.

What Converse however does have, is that it's free and open source software, based on a standard protocol and it can be extended, updated and improved upon, by anyone.

We're actively working on adding new features and more and more people are joining in.

Moreover, anybody can host it and you can integrate it into any website.

Ultimately, I believe in the power and utility of interoperability and software freedom, even though the current trend is to close off and lock down.

These information silos are as powerful as we make them. If enough projects choose standardised protocols and FOSS software, we will be able to create viable alternatives that foster freedom instead of lock-in.

Note

Hacker News discussion

This blog post triggered a lively discussion on Hacker news, you can read that here: https://news.ycombinator.com/item?id=16567009


A badge for linking to your XMPP chat

| categories: xmpp, converse

You've probably noticed the coloured badges at the top of project READMEs on Github and Gitlab.

These badges indicate various metrics about the project's current state and its supporting infrastructure.

For example, the README page for Converse contains the following badge:

Bountysource bounties

It indicates how many open bounties there are for Converse on Bountysource.

There are dozens such badges available nowadays, just take a look at shields.io.

Some projects have badges that link to a chatroom, hosted by Gitter, Slack etc.

Converse, being an XMPP-based project, has its own XMPP chatroom at discuss@conference.conversejs.org.

So I created a badge for its README, which points to that chatroom and also indicates how many users are currently in it.

It looks like this:

Link to the Converse chatroom on inverse.chat

The Markdown syntax for showing the badge looks like this:

[![Link to XMPP chat](https://inverse.chat/badge.svg?room=discuss@conference.conversejs.org)](https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org)

The target URL for this badge is https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org, which means that it takes the user to https://inverse.chat and then once the user has logged in, the room discuss@conference.conversejs.org will be opened.

To go to a different room, just specify a different JID (Jabber ID) in the URL parameters.

The image URL is https://inverse.chat/badge.svg?room=discuss@conference.conversejs.org

Here you specify the room JID with the room URL parameter.

The image is an SVG which is generated by a tiny Flask app, which includes a chatbot written with SleekXMPP that checks how many users are in a particular chatroom.

The code is open source, and hosted on Github at jcbrand/xmpp-chat-badge.

If you're using this badge for your project, I'd be happy to hear about it.

You can let me know in the Converse XMPP chatroom at discuss@conference.conversejs.org (or through the web with inverse.chat) or by using this website's contact form.


Using a virtual DOM with Backbone views

| categories: virtual-dom, converse, backbone.js

Backbone.js, first released in 2010, provided one-way event binding with unidirectional data-flow and therefore a means to do reactive programming before it was cool.

Backbone views provide a render method, which is supposed to insert HTML into the DOM. How this was actually done was left up to the user as a homework assignment.

Generally, jQuery's html method was used. Like so:

const MyView = Backbone.View.extend({

    initialize () {
        this.model.on('change', this.render, this);
    },

    render () {
        this.$el.html(this.template(this.model.toJSON()));
        return this;
    }
});

Each view has an associated DOM element, this.$el if you're using jQuery or this.el otherwise. In the above example, the element's contents would be replaced.

What this approach however lacks, is a way to rerender views containing forms without losing the input focus or user-provided form values.

This is because back in 2010, the virtual DOM as popularized by React wasn't yet around.

Additionally, since rendering the HTML and inserting it into the DOM were both done inside the render method, and the implementation details were left up to the user, enough rope was provided for you to hang yourself by putting things inside the render method which shouldn't be there.

Contrast this to a React component's render method, which is a "pure function" that doesn't modify the DOM, and instead simply returns JSX:

class MyComponent extends React.Component {

        render() {
                return <p>Hello World!</p>;
        }
}

The markup is then inserted into the DOM separately with ReactDOM.render

ReactDOM.render(
        <MyComponent />,
        document.getElementById('root')
);

The fact that a component's render method must return JSX, means that they are pure functions without side-effects.

Backbone Views which use a virtual DOM

In Converse we still use Backbone Models and Views, and for the most part, Backbone still works beautifully. It's a small, lightweight library that's fit for purpose and doesn't get in the way.

However, we've been missing out on that sweet sweet virtual DOM.

No more!

To make our Backbone Views more reactive, as is done in React, I wrote Backbone.VDOMView.

This library adds a new view, called Backbone.VDOMView, which uses a Virtual DOM (provided by Snabbdom) to only update those DOM elements (within the view) as necessary.

Instead of writing a render method, for Backbone.VDOMView you need to implement a toHTML method, which is a pure function that simply returns the view's HTML.

Here's what it looks like:

const MyView = Backbone.VDOMView.extend({

    initialize () {
        this.model.on('change', this.render, this);
    },

    toHTML () {
        return this.template(this.model.toJSON());
    }
});

Backbone.VDOMView takes the output of toHTML and converts it into virtual DOM nodes and automatically diffs and patches the DOM.

When the view's associated Backbone.Model (referenced via this.model) changes, we can simply call render again, knowing that only the changed DOM elements will be updated.

We write less to the DOM and we prevent the loss of form input values and loss of element focus.

Include the root element in your HTML

There's one important difference with normal Backbone.Views and that is that toHTML (as opposed to render) must including the view's root element in the returned HTML.

With normal Backbone.View, you don't include the view's root element when you render a template.

React has a similar requirement to this. JSX returned in the render method of a component must have a root node which contains everything else.

jQuery not required

With Backbone, you can use Backbone.NativeView to remove the default dependency on jQuery.

Backbone.VDOMView checks whether you are using Backbone.NativeView and will then subclass it instead of Backbone.View.

Usage in Converse

In release 3.3.0 of Converse I've removed jQuery as dependency and started integrating Backbone.VDOMView.

Various Backbone.View components from Converse have already been migrated over to Backbone.VDOMView.

So far, I'm very happy with the results, and haven't run into any issues.

If you're using Backbone.VDOMView or have any feedback, feel free to leave a comment or contact me here.


Next Page »