I’m happy to announce that, after almost a year of development, the latest major version of Converse has been released.
Converse is an XMPP chat client, written in JavaScript, that runs in your web browser. It can be used as a single application, or integrated into an existing website and works with any standards-conforming XMPP server.
If you’d like to try out Converse, you can head over to https://conversejs.org (to use it with multiple overlayed chat boxes) or https://conversejs.org/fullscreen.html to use it as a fullpage app with one chat visible at a time. If you need to register an account, you can find lists of public servers at jabber.at, 404.city or xmpp.net.
This release contains multiple significant technological and architectural updates, enough to provide fodder for multiple blogposts. Let’s look at a few highlights.
Highlights
Message Styling
XMPP’s message styling specification describes a way to add markdown-like highlighting to chat messages.
Converse 8 finally adds support for this form of text-styling.
Here’s the output of one of the styling tests, showing how the different styling directives are rendered:
OMEMO File Encryption
Converse now also supports OMEMO encryption of files, via the XEP-0454 OMEMO Media sharing spec.
This means that sent files (like images, or PDFs), can be end-to-end encrypted and then, when received on the other end, decrypted and shown in the chat.
Room Activity Indicators
XEP-0437 Room Activity Indicators (RAI) is a more obscure XMPP extension that deals with some of the drawbacks of the presence-based model of XMPP’s multi-user chat (MUC) specification.
The goal of RAI, is to let the chat client leave a MUC when it’s not longer visible, while still receiving a notification when new messages have been sent in that room.
In collaboration with Matthew Wild and Seve Ferrer de la Peñita, I also wrote (and added support for) a complementary XEP called XEP-0452 MUC mention notifications which informs you when you’ve been mentioned in a MUC that you’re not currently joined to.
Converse will automatically subscribe for activity and mention indicators as
soon as the hidden flag on a chat room is enabled.
The only XMPP server that I’m aware of that currently supports these XEPs is Prosody, via mod_muc_rai and mod_muc_mention_notifications.
Rendering of URL previews
Converse can now render URL previews (aka “unfurls”), if it receives a message with structured metadata in Open Graph Protocol (OGP) format for a particular URL.
Most websites use OGP (one notable exception being Twitter) to provide unfurl metadata.
Prosody’s mod_ogp can be used to send this metadata to Converse, and to render Tweet previews mod_tweet_data can be used.
Below is a screenshot showing the URL preview of a Youtube URL:
Pausing of GIF images
Converse now lets you pause animated GIF images. When an animated GIF is
received, it’s passed to a converse-gif web component, which renders it with a
<canvas> element.
I’m considering releasing this code as a separate open source project, since it might be useful to others outside of Converse as well.
Project layout changes
The layout of the project directories and source files has been changed. The plugins used to be in single (quite large) files, but now they’re broken up into multiple smaller files and moved into their own folders. Tests and styles are now also broken up and put into the same plugin folders.
More can be done here, especially concerning exracting plugin-specific CSS and moving it to the relevant plugin folders, but we’ve made a very good start.
IndexedDB is the new storage mechanism
IndexedDB is now used as the default storage mechanism. This solves the problem of users running into the storage limit of localStorage.
IndexedDB has no storage limit, and it’s also asynchronous, so it doesn’t block rendering of the UI. I started working on adding IndexedDB quite a while back, but ran into performance issues that were difficult to solve. Eventually I managed to find a solution, by using a utility method called mergebounce, which I originally wrote for a different use-case. You can read about that solution in my previous blogpost
There is no tool to migrate data from localStorage to IndexedDB. For almost all data this is no big deal since the data can be fetched again from the XMPP server. However, decrypted OMEMO messages cannot be fetched again from the server, since the double ratchet makes it impossible to decrypt the messages again.
Web components
The Converse UI is now rendered declaratively via web components, using Lit.
Previously Converse was using Backbone Views to render the UI, which reflects the age of this project. Converse was started in 2012, when Backbone was still widely used and many of the big JS projects and language features that people rely on today didn’t exist yet.
Around 2016, Backbone started to really stagnate I realized we’ll have to eventually get rid of it.
I decided it would be preferable to gradually “evolve” the codebase, instead of doing a large rewrite with a newer framework, which could hold up feature development, leave existing users and integrators in the lurch and potentially cause second system syndrome.
Over the years, I chipped away at the problem (e.g. removing jQuery, replacing undescore with lodash, adding a virtual DOM to Backbone Views), spent a lot of time on modernizing the code (e.g. promises instead of callbacks) and on introducing new tools. We were on the JavaScript treadmill, incurring development costs that projects with more mature languages don’t have.
Getting rid of Backbone however proved elusive and difficult. I kept an eye on the developing web component standards, hoping that they might provide a way forward.
One of my goals for Converse is to make it as customizable and hackable as
possible, to allow developers to tailor it to their needs.
By using web components, we could add a registery in Converse (via api.elements.define),
to allow developers to replace the core components with their own at runtime.
Another goal is to make sure Converse can be integrated into any website. Web components are framework agnostic, and because they’re actual DOM elements, they can be used with existing JavaScript frameworks.
Lit also allowed me to start rewriting and updating the code piecemeal, fitting the desire to evolve the code instead of rewriting it.
lit-html, Lit’s templating library, uses tagged template literals to render to the DOM, which allows it to update only those parts of the DOM that have changed. It doesn’t require a virtual DOM and is blazing fast.
From early on Converse used separate template files for HTML, something which I considered desirable given the above-stated goal of hackability, since separate template files make it easier for developers and integrators to override and change the HTML that Converse renders.
lit-html provided a way for us to keep using separate templates files, while providing a faster and superior templating language. So the first thing I did was to update all the templates to use lit-html.
Once that was done, I started rewriting Backbone views into web components using lit-element.
Frustrated with the lack of updates to Backbone, I also created a fork called Skeletor, so that I could start modernizing it.
In Skeletor I created an ElementView class, which was almost exactly like a
Backbone View, but with the added benefit of also being a web component.
Having the ElementView made it easier to turn even more of our existing views into web components, without rewriting lots of code. Over time, I could then refactor these new web component views into Lit components.
This is how I was able to gradually update the UI layer of Converse to use web components without causing significant disruption.
Speed improvements
Converse now renders much faster than before, mainly due to the usage of Lit. The easiest way to see this is by running the tests.
Running the 483 tests for the 8.0.1 release, takes 71 seconds on my laptop. Running 434 tests (49 fewer tests!) on the 7.x.x branch takes 160 seconds.
Thanks
A big thanks goes out to everyone who contributed to this release, from development to bug reports, ticket triage, pull requests and translations.
A toast to messaging freedom!
