Your Site's RSS Feed

Introducing Converse 11: A Major Leap Forward for XMPP Webchat

Wed, 21 May 2025 00:00:00 GMT

I’m thrilled to announce the official release of Converse version 11.0.0! 🚀

You can try it out online at chat.conversejs.org or download the desktop app (the Tauri based releases are coming soon).

First and foremost, thanks to all the contributors whose efforts helped make this release possible.

Many people provided code contributions, bug reports, translations, feature requests or just hung out in our chatroom to provide feedback.

I ran the numbers comparing this release to the previous major release.

The previous major release was 933 days ago, that’s about 2.5 years, and over a thousand commits have been made since then. Some of these commits made it into subsequent minor 10.x.x releases, but there’s no denying that the 11.0.0 release is a big one. A quick look at the release notes also shows how much work has gone into it.

Running git rev-list --count v10.0.0..HEAD
1062

I’m particularly excited about this release, because it addresses some long-standing painpoints and issues.

A friend of mine likes to talk about “embarrassment-driven development”.

That’s a bit how I feel about this release. It fixes some problems that have been particularly glaring and embarrassing since a number of years.

Let’s take a closer look at some of the most exciting features and improvements in this latest release:

TypeScript

I’ve always taken a fairly conservative approach with Converse, trying to avoid jumping on any bandwagons and generally trying to evolve the codebase to keep up with the times, instead of opting for huge rewrites.

This project started in 2013, JavaScript developers who have been around since that time, will appreciate the massive amount of churn and changes that have happened in our ecosystem from then until now! There’s even a term coined for it: “JavaScript fatigue”.

I was wary of just rewriting everything in TypeScript. I had limited time to work on Converse, and didn’t want to spend all of it on a rewrite, instead of adding new features and making bugfixes.

Reading about the decision of the team behind Svelte, to use JSDoc comments to add types to their codebase, gave me confidence to try the same with the Converse codebase. I felt that this would be a less disruptive change than trying to rewrite everything in TypeScript, and would fit nicely with the approach I’ve taken until now.

Version 11.0.0 is the first release that includes TypeScript type definitions for Converse.

The tsc compiler is still not running in strict mode, and there are still untyped sections of the code, but it’s a good start, and we’re already reaping benefits.

Virtualized chat history

This is one update I’m particularly proud of. Until now, Converse was rendering all messages it knew about in a particular chat.

If a chat had 2000 messages, all those messages would be rendered, causing all kinds of performance issues.

List virtualization is a technique whereby only a subset of a list is rendered, while the impression is maintained that the entire list is shown.

Converse finally has list virtualization for the chat history.

Mind the gap

Related to history or list virtualization, another update to Converse is to better deal with gaps in the chat history.

Until now, when opening a chat, the entire history from the previous session until now was fetched and shown. When opening Converse in a browser where it didn’t run for, say, 8 months, everything would crawl to a halt as Converse tried to fetch the entire history.

I fixed this, but only fetching the most recent messages, and then placing a special placeholder element in the gap where there are still missing messages.

I then used the IntersectionObserver API, to identify when this gap becomes visible, which would happen example when the user scrolls up to read older messages. As soon as this gap became visible, Converse would fetch a limited number of the most recent messages in that gap, and then place a new placeholder above them, in case there were still missing messages.

In this way, we avoid fetching all messages immediately, but instead we load them lazily as the user scrolls up to read older messages.

I’m happy with the usability and performance improvements this change has brought. One drawback, is that we can’t inform the user of being mentioned, if he or she was mentioned in a message inside that gap of unfetched messages.

This is however to a large extent a limitation of XMPP’s message archiving extension (MAM).

An attempt has been made to address this limitation with the Inbox XEP, but it hasn’t seen any real uptake, particularly among server developers.

Hopefully this drawback will be addressed in due time.

PEP Bookmarks

Adding support for PEP (Personal Eventing Protocol) bookmarks, was some unglamorous modernization work that was also long overdue.

Converse was using an older bookmarks protocol extension, which was causing compatibility issues with other XMPP clients, unless the user’s XMPP server provided a compatibility layer between the two protocols.

By updating to the PEP standard, these issues have now been resolved.

Additionally, Converse now no longer lets the user manually bookmark or unbookmark chats. By doing so, it was unnecessarily exposing protocol details to users, making the app more confusing and difficult to use.

Instead, bookmarks are automatically set and unset as users join and leave groupchats, similarly to how some other XMPP clients already do it.

Updated retractions and moderation

Converse has supported retracting (i.e. deleting) and moderating (i.e. deletion by a moderator) of messages since a number of years already. In the meantime, these protocol extensions have been updated based on community feedback. This release has incorporated those changes.

MUC Private Messages

Finally Converse supports private messages in groupchats.

This is another one of those embarrassment-driven features. Ideally Converse would have had this feature long ago already, but hey, better late then never.

What’s also nice is that some of the work done for this feature has opened up the door for Slack-like threads.

Tagged template literals to render XML stanzas

This is more a new feature I made to Strophe, the underlying XMPP library that Converse uses, but it’s still something pretty cool.

Previously, Strophe’s API for generating XML stanzas, looked like this:

const msg = $msg({
    to: buddy_jid,
    type: 'chat',
    id: u.getUniqueId()
}).c('body').t('Hello world').up()
    .c('active', {'xmlns': 'http://jabber.org/protocol/chatstates'});

Now, with the magic of JavaScript’s tagged template literals, you can write them like this:

const msg = stx`
    <message xmlns="jabber:client" type="chat" id="${u.getUniqueId()}">
        <body>Hello world</body>
        <active xmlns="http://jabber.org/protocol/chatstates"/>
    </message>`;

Much nicer and more readable to my eye. And it’s all standard JavaScript, no compiler necessary.

Bootstrap 5

Converse now uses Bootstrap 5, and the markup has been updated to follow the best practices and conventions for Bootstrap 5.

Right to left support

Converse is translated into over 40 languages. Four of these languages are displayed right to left, namely Arabic, Farsi, Hebrew and Uyghur.

When one of these languages is used (e.g. based on the user’s browser language), Converse will render its UI inverted, from right to left, instead of the usual left to right.

Future directions

There are many features and improvements that could still be developed for a future release.

Some highlights include:

Microblogging (Github issue, XEP-0277 and XEP-0472).
Slack-like message threads.
Push notifications (Github issue)
A mobile app (Github issue)
Audio/video calls integration (Github issue)

If you’re considering sponsoring any of these features, please reach out to me via the contact form.

I’m also currently looking for shorter term freelance contracts, whether Converse-related or not. If you’re interested in hiring me, please reach out.

We also accept general donations and sponsorships via Patreon, Liberapay and Github. Sponsors who donate $100 or more per month will have their logo displayed on the Converse website and app.

Fin

That’s all, thanks for reading and I hope you enjoy the new release.

Adopting Bitcoin, Cape Town, 2025

Thu, 30 Jan 2025 00:00:00 GMT

I attended the Adopting Bitcoin conference in Cape Town the last weekend and thought I’d do a quick writeup about some interesting things I learned.

Some background, South Africa has quite high crypto adoption (Bitcoin in particular). Our 2nd biggest retailer accepts crypto payments, there are a bunch of circular bitcoin economy projects around the country and more are popping up. Most of this activity is bottom-up, the government is not pro-crypto.

Pick ‘n Pay adoption of crypto payments

The first presentation was about SA’s 2nd largest retailer, Pick ‘n Pay, who has adopted crypto payments at its stores about 2 years ago. I don’t have a PnP at the small town where I live, so I’ve never used them.

While at the conf, I went to the local PnP and bought coffee and a magazine with Bitcoin. You can use the self-custodial Blink wallet (previously Bitcoin Beach) to pay. Apparently any wallet could work, but there is some API call that it needs to make which only Blink and some South African exchanges currently support.

3% of Pick ‘n Pay’s sales are in Crypto (not just Bitcoin, but also Solana and others), which is up from 0.5% 2 years ago. This is one million Rand per month, which is about $50.000. So not a large absolute amount, but trending up.

Bitcoin: the South African story

This was a first presentations was from Stafford Masie and Carel de Jager about their lobbying efforts to do the following:

Convince South Africa’s monopoly energy provider Eskom to embrace Bitcoin mining to facilitate grid management and expansion. Carel showed some calculations that if Eskom did CapEx of R1.7 Billion in 2021, and kept 50% of the mining proceeds, they would have made a surplus of 400 Billion Rand, which is equal to their total current debt (which is backstopped by the tax payer). They hoped to announce a mining agreement at the conf, but the talks are still in progress. Apparently the holdup is on the government’s side, which is the shareholder.
Convince the Reserve Bank and Intergovernmental Fintech Working Group (IFWG) to change legislation which currently blocks the creation of a Bitcoin holding company modeled on Microstrategy.

Circular Bitcoin economies and microgrids

Other smaller talks were about establishing circular Bitcoin economies in South Africa and other African countries (Kenya, Malawi, Burundi) of which there are a bunch by now. One South African township shopkeeper mentioned that since he accepts and saves in Bitcoin, he has stopped drinking alcohol and instead of partying at the tavern over the weekend, he spends time with his wife and child. Saving in Bitcoin changes people for the better.

There was a talk about BTrust, which is a non-profit which received 500 Bitcoin from Jack Dorsey and Jay-Z, and about Gridless, which has built 5000 micro-grids in rural Africa to provide electricity to rural communities, subsidised by Bitcoin mining and which would otherwise not have been economically feasible.

Politicians

Helen Zille

Political Bigwig Helen Zille held a talk about her son who orange-pilled her some years ago and how she’s now saving in Bitcoin. Somewhat cynically I think she came mostly because her party didn’t want to be upstaged by another politician, Mzwanele Manyi, who came to announce his party’s proposal for a strategic Bitcoin reserve for South Africa.

Mzwanele Manyi

One very strange thing is that Manyi’s party, MK, is not just calling for a strategic Bitcoin reserve, but also the nationalization of the central bank, the private banks, mines, factories and forced land expropriation without compensation, all to be owned by the state for the purposes of redistribution. Doesn’t sound like the kind of party that would propose a strategic Bitcoin reserve, does it?

In Many’s talk, he highlighted illegal exchange rate manipulation from the private banks in 2023 to enrich themselves (they were fined for this).

He mentioned key advantages of Bitcoin that are well known to Bitcoiners.

The removal of unnecessary financial intermediaries, which according to him is where a lot of corruption happens.
Fixed supply
Operates on principles of transparency and fairness.
Would reduce South Africa’s dependency on international institutions like the IMF.

Some other things he mentioned:

El Salvador. They’ve demonstrated the benefits. They successfully repaid an $800M bond, helped by Bitcoin.
Single biggest line item in SA’s budget is debt servicing. Higher than anything else.
Wants to use Bitcoin to reduce SA’s debt burden. (Zille’s position was that this would cause a currency crisis and is not a good idea).
Proposal: that SA must allocate between 4-5% of reserves into Bitcoin, R50 billion. Expects it to grow to R380 billion in 5 years.
He is explicit that he wants SA to front-run the USA.
They are soon going to table a discussion on this in Parliament.

The game theory that Bitcoiners have mentioned since years is starting to play out.

A question was asked about how to reconcile MK’s non-free market interventionist policies with Bitcoin’s libertarian ethos. Manyi’s response is that he doesn’t want to be boxed in ideologically and that he is interested in pragmatism and not ideology. Says libertarians want the same as everyone, but the starting position is not the same. He says “let’s level the playing field so that we can then have a free market.” Says affirmative action must not be perpetual, must eventually stop, but first needed to level the playing field.

Beverley Schafer

Up next, was Beverley Schafer, who was a politician for 15 years.

35 million people in SA earning less than R1000 per month. Complains that she can’t open accounts at exchanges because she’s been an active politician in SA and is now a “politically exposed person”.

Says she was forced to find non-KYC alternatives to stack sats. Says existing fiat systems don’t prioritise freedom, they prioritise control. In South Africa, the central bank ties us to a depreciating currency which between 2010 and 2023 lost 60% against the US dollar.

She’s quite clued in. Quotes Jeff Booth. Encourages self custody and taking a long-term approach. Says she made mistakes on her Bitcoin journey, but has learned how to save and to commit to the long game.

Cross-border payments via Lightning

There was an announcement by Orange, which is using the Lightning network to facilitate cross-border remittances across Africa. By using Lightning, you need zero float, i.e. you don’t need to keep large pools of rapidly depreciating African fiat currencies, because sats are so fungible. They partner with OTC desks on both sides. In doing so, they drastically reduce fees for customers.

They did two live demos.

R104 worth of Nigerian Naira was sent from a Nigerian bank account to a South African one, with the cross border payment made over Lightning. It took about 2 minutes for the money to show up in the South African bank account.

Another where Kenyan Pesa worth R111 was made from a Kenyan mobile account (via MPesa, which is big in Kenya) to a South African bank account.

Chatting with the organizers

I had some conversations with the organizers and heard a few interesting tidbits. Not everything appropriate for sharing publicly here.

They ran their own BTCPay server and everyone who attended paid in Bitcoin, with most paying via Lightning.

People already started paying for tickets in the middle of 2024, so they made a nice profit. They also saved around R40.000 in credit card processing fees.

The attendees

The attendees were an eclectic mix. There was a well known Billionaire, the politicians, a think tank CEO, some civil society leaders, VCs, local entrepreneurs and plebs.

One guy raises chickens and sells the eggs for Bitcoin. I saw a lady I know from outside of Bitcoin, who started an off-grid hippie community in a nearby forest.

The conference was small. Compared to the large overseas ones, it’s more like a large meetup. The nice thing however was that this made everyone accessible, and I had lots of interesting conversations.

Converse 8 has been released

Mon, 13 Sep 2021 00:00:00 GMT

I’m happy to announce that, after almost a year of development, the latest major version of Converse has been released.

Converse is an XMPP chat client, written in JavaScript, that runs in your web browser. It can be used as a single application, or integrated into an existing website and works with any standards-conforming XMPP server.

If you’d like to try out Converse, you can head over to https://conversejs.org (to use it with multiple overlayed chat boxes) or https://conversejs.org/fullscreen.html to use it as a fullpage app with one chat visible at a time. If you need to register an account, you can find lists of public servers at jabber.at, 404.city or xmpp.net.

This release contains multiple significant technological and architectural updates, enough to provide fodder for multiple blogposts. Let’s look at a few highlights.

Highlights

Message Styling

XMPP’s message styling specification describes a way to add markdown-like highlighting to chat messages.

Converse 8 finally adds support for this form of text-styling.

Here’s the output of one of the styling tests, showing how the different styling directives are rendered:

OMEMO File Encryption

Converse now also supports OMEMO encryption of files, via the XEP-0454 OMEMO Media sharing spec.

This means that sent files (like images, or PDFs), can be end-to-end encrypted and then, when received on the other end, decrypted and shown in the chat.

Room Activity Indicators

XEP-0437 Room Activity Indicators (RAI) is a more obscure XMPP extension that deals with some of the drawbacks of the presence-based model of XMPP’s multi-user chat (MUC) specification.

The goal of RAI, is to let the chat client leave a MUC when it’s not longer visible, while still receiving a notification when new messages have been sent in that room.

In collaboration with Matthew Wild and Seve Ferrer de la Peñita, I also wrote (and added support for) a complementary XEP called XEP-0452 MUC mention notifications which informs you when you’ve been mentioned in a MUC that you’re not currently joined to.

Converse will automatically subscribe for activity and mention indicators as soon as the hidden flag on a chat room is enabled.

The only XMPP server that I’m aware of that currently supports these XEPs is Prosody, via mod_muc_rai and mod_muc_mention_notifications.

Rendering of URL previews

Converse can now render URL previews (aka “unfurls”), if it receives a message with structured metadata in Open Graph Protocol (OGP) format for a particular URL.

Most websites use OGP (one notable exception being Twitter) to provide unfurl metadata.

Prosody’s mod_ogp can be used to send this metadata to Converse, and to render Tweet previews mod_tweet_data can be used.

Below is a screenshot showing the URL preview of a Youtube URL:

Pausing of GIF images

Converse now lets you pause animated GIF images. When an animated GIF is received, it’s passed to a converse-gif web component, which renders it with a <canvas> element.

I’m considering releasing this code as a separate open source project, since it might be useful to others outside of Converse as well.

Project layout changes

The layout of the project directories and source files has been changed. The plugins used to be in single (quite large) files, but now they’re broken up into multiple smaller files and moved into their own folders. Tests and styles are now also broken up and put into the same plugin folders.

More can be done here, especially concerning exracting plugin-specific CSS and moving it to the relevant plugin folders, but we’ve made a very good start.

IndexedDB is the new storage mechanism

IndexedDB is now used as the default storage mechanism. This solves the problem of users running into the storage limit of localStorage.

IndexedDB has no storage limit, and it’s also asynchronous, so it doesn’t block rendering of the UI. I started working on adding IndexedDB quite a while back, but ran into performance issues that were difficult to solve. Eventually I managed to find a solution, by using a utility method called mergebounce, which I originally wrote for a different use-case. You can read about that solution in my previous blogpost

There is no tool to migrate data from localStorage to IndexedDB. For almost all data this is no big deal since the data can be fetched again from the XMPP server. However, decrypted OMEMO messages cannot be fetched again from the server, since the double ratchet makes it impossible to decrypt the messages again.

Web components

The Converse UI is now rendered declaratively via web components, using Lit.

Previously Converse was using Backbone Views to render the UI, which reflects the age of this project. Converse was started in 2012, when Backbone was still widely used and many of the big JS projects and language features that people rely on today didn’t exist yet.

Around 2016, Backbone started to really stagnate I realized we’ll have to eventually get rid of it.

I decided it would be preferable to gradually “evolve” the codebase, instead of doing a large rewrite with a newer framework, which could hold up feature development, leave existing users and integrators in the lurch and potentially cause second system syndrome.

Over the years, I chipped away at the problem (e.g. removing jQuery, replacing undescore with lodash, adding a virtual DOM to Backbone Views), spent a lot of time on modernizing the code (e.g. promises instead of callbacks) and on introducing new tools. We were on the JavaScript treadmill, incurring development costs that projects with more mature languages don’t have.

Getting rid of Backbone however proved elusive and difficult. I kept an eye on the developing web component standards, hoping that they might provide a way forward.

One of my goals for Converse is to make it as customizable and hackable as possible, to allow developers to tailor it to their needs. By using web components, we could add a registery in Converse (via api.elements.define), to allow developers to replace the core components with their own at runtime.

Another goal is to make sure Converse can be integrated into any website. Web components are framework agnostic, and because they’re actual DOM elements, they can be used with existing JavaScript frameworks.

Lit also allowed me to start rewriting and updating the code piecemeal, fitting the desire to evolve the code instead of rewriting it.

lit-html, Lit’s templating library, uses tagged template literals to render to the DOM, which allows it to update only those parts of the DOM that have changed. It doesn’t require a virtual DOM and is blazing fast.

From early on Converse used separate template files for HTML, something which I considered desirable given the above-stated goal of hackability, since separate template files make it easier for developers and integrators to override and change the HTML that Converse renders.

lit-html provided a way for us to keep using separate templates files, while providing a faster and superior templating language. So the first thing I did was to update all the templates to use lit-html.

Once that was done, I started rewriting Backbone views into web components using lit-element.

Frustrated with the lack of updates to Backbone, I also created a fork called Skeletor, so that I could start modernizing it.

In Skeletor I created an ElementView class, which was almost exactly like a Backbone View, but with the added benefit of also being a web component.

Having the ElementView made it easier to turn even more of our existing views into web components, without rewriting lots of code. Over time, I could then refactor these new web component views into Lit components.

This is how I was able to gradually update the UI layer of Converse to use web components without causing significant disruption.

Speed improvements

Converse now renders much faster than before, mainly due to the usage of Lit. The easiest way to see this is by running the tests.

Running the 483 tests for the 8.0.1 release, takes 71 seconds on my laptop. Running 434 tests (49 fewer tests!) on the 7.x.x branch takes 160 seconds.

Thanks

A big thanks goes out to everyone who contributed to this release, from development to bug reports, ticket triage, pull requests and translations.

A toast to messaging freedom!

Mergebounce: Increasing performance by batching IndexedDB writes

Wed, 05 May 2021 00:00:00 GMT

Converse is a browser-based XMPP chat client that I’ve been working on. It stores lots of user-data in the browser, so much data, that users were regularly running into the localStorage limit of about 10MB.

Luckily there is an alternative storage solution in modern browsers, namely IndexedDB, which doesn’t have a fixed size limit.

IndexedDB has a very different API than localStorage, and it’s also asynchronous, so updating Converse to use it would be a big task.

Luckily I found the excellent localForage library, which provides a unified API for storing data in either localStorage or IndexedDB (including other potential backends such as sessionStorage).

localForage helped a lot, but it still took a long time to add Converse support for IndexedDB (via localForage), mainly because of its asynchronous API. The Converse code (and its test suite) was structured around quick synchronous writes to localStorage, so there were implicit assumptions in the code that were now broken.

Eventually I managed to get Converse to use localForage, so that the user could choose which persistent store they’d like to use, sessionStorage, localStorage or IndexedDB.

Unfortunately it soon became clear that Converse was painfully slow when using IndexedDB for moderate to heavy chatting.

Turns out the issue is that IndexedDB’s writes are slow, and even though they were asynchronous, they were still slowing the app down. The reason for this is because with XMPP, you can’t process subsequent messages until processing of previous ones have been finalized, otherwise you can’t handle messages that refer to earlier ones, such as corrections or retractions.

So Converse was first waiting until a message object was created (and persisted), before handing the next incoming message.

When you opened a chat room, you’d get a 100 messages, but they’d appear slowly as Converse waits for IndexedDB writes to complete before processing each subsequent message.

It was clear that the number of writes to IndexedDB had to be reduced, but I wasn’t sure how. So I left the problem and moved on to other things, believing that I’ll eventually think of (or stumble upon) a solution.

Eventually I did stumbled upon a partial solution in the form of the localForage-setItems plugin, which lets you write write multiple key-value pairs in a single transaction, something which standard localForage doesn’t support.

This was useful because while debugging the issue I noticed that whenever a message was created, two localForage setItem calls were made. Once to add the message to the collection of messages belonging to the chat room, and once to save the attributes of the actual message.

With localForage-setItems I could collapse these two calls into a single one.

The second part of the puzzle came months later, when I had an insight while thinking about reducing HTTP calls in an app. Multiple function calls could be collapsed into one, if the parameters for each call were objects that could be merged into a single object.

Debouncing lets us reduce a number of successive function calls into a single call, but only the parameters of the final call of the debounced function are ever passed to the inner executing function. Earlier passed in parameters are lost.

If we could create a special kind of debounced function that merges the parameters of each call, then we could in essence batch multiple function calls into a single one, without losing any final data compared to executing all the calls separately.

I set out to create such a function, and called it mergebounce.

It uses lodash’s debounce and merge functions to create a mergebounce function.

I then realized that this function, together with localForage-setItems could solve the IndexedDB-related slowness in Converse, because the data that was being saved was objects that could be merged into one and then persisted with a single write.

So I set about adding mergebounce to the storage-layer of Converse.

What I did, was to use mergebounce to create a wrapper function of localForage-setItems that intercepts all calls and after 50ms of inactivity calls the inner function with a single merged object from all the individual calls.

Like debounce, mergebounce has a flush method which can be used to immediately execute the inner bounced function. I added event handlers to flush all writes when the user logs out of the app, and also when the tab unloads.

There were a few other kinks to iron out, but I’m happy to report that it turned out to be a big success.

The effect on performance was drastic and immediately noticeable.

I also did some rudimentary profiling. Without mergebounce, I counted over a hundred and twenty IndexedDB writes when logging into Converse with my personal XMPP account. With mergebounce, this dropped to about 17. An improvement of almost a factor of ten.

Truth be told, I’m kind of suprised that I haven’t seen something like mergebounce before. Maybe it’s out there and I just missed it. In any case, I’m quite chuffed at having cracked this nut.

This improvement has been merged to the master branch of Converse and will be included as part of the upcoming 8.0.0 release.

Since then I’ve also used mergebounce to reduce HTTP calls to fetch resources in a different project and I expect to reach for it again in the future.

Pehaps it can be of value to others as well.

Feel free to reach out to me via my site’s contact form if you’d like to know more.

Bitcoin as commons

Thu, 04 Feb 2021 00:00:00 GMT

I recently saw an interesting interview with former Canadian prime minister Stephan Harper¹, where the topic of Bitcoin briefly came up, and I noticed that he referred to Bitcoin as a “private digital currency”.

I think it’s wrong to call Bitcoin a private currency. In fact, I view Bitcoin as a type of commons, a digital commons.

The word “commons” comes from the Latin communis, the same root that gives us “commoner”, “community”, “communism” and “commune”, and describes something “belonging to all, owned or used jointly, general, of a public nature or character”².

According to Wikipedia (which is another spectacularly successful digital commons):

The commons is the cultural and natural resources accessible to all members of a society […]. These resources are held in common, not owned privately.

At first glance, the above sentence might seem to dispute the claim that Bitcoin is a commons. After all, Bitcoins are privately owned, and no-one can spend or take ownership of your Bitcoins without getting hold of your private key (which amounts to stealing them from you).

In fact, I posted a message on the Fediverse claiming that Bitcoin is a commons and the argument that Bitcoins themselves are privately owned was brought up as rebuttal by some people.

You can read the whole thread here: https://mastodon.xyz/@jcbrand/105646757598892890

The paradox in calling a privately held asset a commons is resolved when one zooms out and looks at the Bitcoin network as a whole, from which Bitcoins arise and derive their utility.

When I call Bitcoin a commons, I’m referring to the Bitcoin network.

In fact, it’s debatable in which sense Bitcoins themselves actually exist. They have no corporeal existence and no separate existence independent of the Bitcoin network. They exist only as entries in a distributed ledger (the blockchain) that is secured, updated and maintained via mining nodes on the network.

It is the network that gives the Bitcoins value, how much value is largely subjective and based on what people are willing to exchange for them³.

The next rebuttal that one might think of, is that the network isn’t a commons, because it’s comprised of private entities that all act in their own self-interest and use their own private hardware. As someone replied on Mastodon, “the network is just a metaphor for privately owned hardware”.

I believe this to be an oversimplification of what the network really is.

The network transcends the hardware, but even if it was just the hardware, that hardware can be owned by anyone. It can be owned by governments, public universities, book clubs, communes, worker cooperatives and any other form of human organisation that one could imagine.

You could also theoretically run Bitcoin without computer hardware on a pen and paper system. People could do the proof of work (PoW) calculations (the factoring of prime numbers) by hand and then mail copies to other participants, who would verify them and update the blockchain (written on paper) with the new transactions.

Transactions would take days or weeks to clear and reorgs would probably happen more frequently. Also the PoW difficulty would have to be such that human beings could do the factoring by hand, but aside from that, I’m not aware of anything about Bitcoin that intrinsically requires digital computers and hardware (although they are a practical requirement).

The network comes into being as soon as the participants (aka nodes) are able to connect to one another, agree on a protocol and adhere to the rules of the protocol.

Bitcoin is a peer-to-peer (p2p) network, which means that any node can connect to any other node and there is no central hub through which traffic must be routed (as would be the case in a centralized network like Facebook or Visa).

The distributed consensus mechanism based on proof-of-work (aka the “mining” of Bitcoins) is the way in which the rules of the protocol are enforced in a decentralized, p2p manner without a central authority.

The Bitcoin protocol is described and implemented by the Bitcoin-core software, which Satoshi Nakomoto wrote and published as free and open source software.

It’s generally accepted that free and open source software forms a digital commons, since anyone in the world is allowed to study, modify, run and distribute the software without encumberance⁴.

The global database used by the Bitcoin network, the blockchain, is also a commons in the sense that anyone can download it, study and modify it, and distribute it to others.

If you want others to accept your modifications to the blockchain as valid updates to the ledger, you have to conform to the protocol rules and also provide proof of work by way of generating a hash that conforms to a specific difficulty function. This requirement is in order to maintain the integrity of the ledger, the security of the network and to create distributed consensus in a trustless environment.

In fact, in order to fully participate in the Bitcoin network, you have to download the entire blockchain.

One could argue that the requirement to fulfill the proof-of-work in order to update the ledger in some way negates the proposition that Bitcoin is a commons. To me this is akin to the argument that Wikipedia is not a commons because there are moderators who delete unwanted edits to a page. I don’t think this argument holds⁵.

The Bitcoin protocol itself is designed in such a way as to make the network permissionless. What this means is that no-one can interfere in the operations of the network in order to prevent you from participating (in the form of mining, sending or receiving Bitcoins or verifying transactions).

With the caveat that no single entity acquires 51% of the hashing power of the entire network. But even then, in such a scenario this would be seen as an attack on the network and an attempt to break the agreed upon rules.

If a successful attack or hard fork were to happen which changes the properties of Bitcoin so that it becomes permissioned in some way, it would no longer be a commons, but it would also no longer be Bitcoin.

So, given the fact that the necessary software to bootstrap, run and maintain the network is a digital commons, the fact that the network is permissionless and allows anyone in the world to participate, the fact that the database is public and accessible to all, and the fact that you’re not dependent on any particular private entity’s hardware to participate, it’s clear to me that the Bitcoin network (the thing that gives birth to Bitcoin the currency and to some extent gives it value) is a digital commons.

But let’s take this analysis a little further by comparing Bitcoin to something that’s meant to be similar, but is actually very different.

Various central banks are working on what’s called “Central bank digital currencies”, or CBDCs. These CBDCs are directly inspired by Bitcoin and cryptocurrencies in general and are in some sense a response to the existence and rise of Bitcoin⁶.

In any case, CBDCs will most likely not form a commons. They won’t be permissionless because only trusted parties (banks) will be able to “mine” new currency units, and the software necessary to bootstrap, run and maintain the CBDC network will most likely not be free and open source.

In fact, given that central banks and their member banks are usually private entities owned by private shareholders, I’d contend that CBDCs are actually the real private currencies!

Similarly, Libra, the proposed cryptocurrency initiative by Facebook and a consortium of private componies, would also not be a commons because issuence won’t be permissionless and the node software probably not free and open source either.

Bitcoin, and some other cryptocurrencies, are special and different from CBDCs and private currencies like the proposed Libra, and that comes from it’s nature as commons. An open network that’s non-exclusionary, with the source code, protocol and the database publicly visible for all to see and verify.

Footnotes

I came away very impressed by Stephen Harper, especially since he’s willing to criticize central bank policies and is willing to admit that they’ve blown huge asset bubbles (thereby massively increasing income inequality).↩
etymonline ↩
This would also largely be my response to Stephen Harpers question as to what makes them a store of value.↩
These are known as the four freedoms that define free software.↩
I like my puns intended.↩
I also believe that they arise out of the desire of central banks to do away with physical cash altogether and to have totalitarian control over the issuance and usage of money.↩

Report on the Dusseldorf XMPP sprint

Thu, 29 Nov 2018 17:30:00 GMT

Recently we held another XMPP sprint, this time in Dusseldorf at the Chaosdorf clubhouse.

We had 10 participants and an ambitious agenda with a long list of proposed topics to sprint on. We didn’t get to work on everything, and we spent a considerable amount of time discussing and prioritizing the agenda. Some of the things not worked on can however easily be moved to future sprints. After the sprint, iNPUTmice suggested that we consider having more focused sprints, for example focusing only on implementing XEP-369 MIX in clients and servers.

I think having a more focused, single-topic sprint is a great idea, and MIX is a good topic to tackle. AFAIK there’s currently no FOSS MIX implementation for either clients or servers. IMO this can partly be attributed to a stalemate situation where everyone is waiting for someone else to make the first move. By organizing a sprint around it we can break the deadlock by starting to work on it together. Discussions

Easy-muc-joining (jcbrand, Ge0rG, pep., Link Mauve)

This was related to the Easy XMPP Invitation work already done by Ge0rG. When using a supporting client, you can generate a URL to a landing page which makes it easy for someone to create a new XMPP account and/or to join a groupchat (aka multi-user chat, MUC).

For “easy MUC joining”, the goal is to figure out whether a MUC is hosted somewhere in a webchat, and to present this to the user so that they’re not forced to choose, download and install a desktop client.

Various approaches were discussed and eventually we decided that since the invitation URL is generated by a MUC service, that MUC service can also encode in the URL whether it has an associated webchat or not.

Initially we thought about using XEP-0156 service discovery on the MUC, but after discussing it futher with Ge0rG, we agreed that XEP-0401 Easy User Onboarding can be extended to MUCs as well, so that the MUC can be queried similarly to how a user invitation is generated.

MUC Push Notifications (jcbrand, iNPUTmice, Ge0rG, MattJ, Link Mauve)

XMPP has an extension XEP-0357 which allows you to register an external push service (such as Google Play for Android) to which messages can be pushed when you’re offline.

This approach can also be used for registering a push service that sends out email notifications. This is especially useful in a teamchat setup (like Slack, Mattermost, etc.).

XEP-0357 calls the push service the “app server”, but I find this terminology confusing and too much tied to the mobile use-case, so I’ll stick with “push service” instead.

XEP-0357 currently doesn’t mention MUCs at all and we wanted to discuss how registering a push service for MUCs would work.

Before this sprint, I wrote a proof-of-concept Prosody module, mod_muc_cloud_notify which can be used with Converse.

That module has the requirement that the MUC knows the JID (Jabber ID) of all the users, so that it can keep track of the push services registered for them, and it needs to know the JID even when the user is offline. Additionally the module requires that XEP-0372 references are used to make it clear when someone is mentioned.

This restriction means that anonymous rooms are excluded, and requires that all users register their nicknames on the MUC.

iNPUTmice proposed simply keeping some kind of token other than the JID, to keep track of registered push services, removing the need for nickname registrations.

Additionally he suggested sending all messages to the push service, instead of sending only messages in which the offline user is explicitly mentioned. This opens the possibility of configuring the push service to also notify you based on other criteria, for example when a particular keyword or phrase is mentioned.

We agreed that XEP-0372 is broad enough for this usecase and that we don’t need a new XEP.

Sync open conversations across clients (pep., iNPUTmice, Ge0rG, Link Mauve, MattJ, jcbrand)

A common request I’ve received from customers, and something which isn’t yet solved in a standardized way, is the ability keep track of open conversations across multiple chat clients (e.g. Desktop, mobile and web) or sessions and to quickly get information on the amount of unread messages or mentions in those conversations.

We spent a substantial amount of time discussing this feature, trying to cover as many use-cases as possible while coming up with something elegant.

Eventually it was decided to split the requirement into two parts and to only discuss one, leaving the other for a later date.

The two new requirements are:

To keep track of open conversations across chat clients (and chat sessions)
To get an overview of unread messages and mentions for those open conversations

The proposed solution for #1 is to use a private PEP node containing a list of JIDs representing open chats.

Like the Bookmarks2 specification, items should be stored separately, so that they can be added and removed individually (i.e. without having to specify the whole list).

This will require us to fix PEP so that we can get deletion notifications.

MUC JIDs aren’t stored in this list, since we already have the auto-join feature for MUC bookmarks. Private MUC chats will however be stored as well.

XEP-0283 <moved/> (pep., Ge0rG)

XEP-0283 provides a mechanism for informing your contacts that you’re changed your account and now have a new JID.

pep. and Ge0rG had a discussion on how to automate the operation entirely.

Currently the security considerations contain a clause which to them defeats the point of the XEP:

“In order to prevent other users from maliciously altering contacts the client SHOULD NOT automatically subscribe to a JID when it receives an unsubscribe and SHOULD NOT automatically unsubscribe to a JID when it receives a subscribe.”

Also the XEP uses presence stanza only, which makes the whole operation rather ephemeral.

The plan is to change the XEP to use PEP mostly.

Programming, documentation and various other tasks

iNPUTmice added two new tests to the compliance tester, one for MUC Avatars and a check for CORS Headers on HTTP Upload, which is especially important for webchat clients like Converse.

He also tried to fix a bug in Conversations that happens intermittently on Android 9.

MattJ worked on the 0.11 release of Prosody. He also worked on adding support for CORS in mod_http_upload.

pep. registered an XMPP assembly for 35C3. He also worked on the slixmpp OMEMO plugin and got decryption working. Additionally he and Link Mauve worked on a French translation of the Prosody release announcement.

Link Mauve worked on implementing Jingle in Converse, fixing bugs and updating strophe.jingle.js to not use jQuery. As part of this, he and I looked at better exception handling in Converse.

He also helped pep. and syndace with the slixmpp plugin interface and spent some time trying to deploy JSXC.

Roel worked on the homebrewserver.club article on how to configure a modern XMPP server. MattJ helped out on Sunday as well and AFAIK their collaboration also caused MattJ to add some clarifications and simplifications to the Prosody config file.

Roel and rtq3 also worked on a Spanish translation of the homebrewserver.club article.

Syndace worked on resurrecting a non-merged PR from Andy Straub for the OMEMO XEP which adds various changes to the XEP, including using an XML-based wire format. By choosing our own wire format, we also make permissive OSS licensed OMEMO implementations possible. Such a change will however not be backwards compatible with current OMEMO implementations.

Andy joined us on Sunday and participated in discussions around the PR and OMEMO in general with iNPUTmice, pep and Syndace.

Haeckse showed us a demo of Chatty, the chat (and SMS) client that she is working on for the upcoming Purism phone.

I worked on adding support for XEP-0156 to Converse, based on a PR from Link Mauve. The original PR was closed, due to being very old and difficult to merge, and work is now continuing in a branch with a new PR #1340.

I also made smaller fixes and changes, for example only rendering image URLs inline if they use HTTPS (to avoid mixed-content warnings in the browser).

Conclusion

Chaosdorf

Much thanks to Chaosdorf, for generously making their clubhouse available for the sprint.

Kuro even made some chilli for lunch and baked a cake!

In the evenings we went out to dinner. Dusseldorf has a large Japanese population, so we were quite keen to try out some Japanese food.

I had proper Ramen for the first time on Sunday night and on Saturday we had some great sushi.

Future sprints

A sprint can be measured by its output in the form of code, documentation, translations and so forth, but this only tells part of the story.

Much of what makes sprints valuable is the social aspect of making friends and working together on something you all care about.

One way to judge the success of the sprint, is to consider how excited the participants are about organising and attending future sprints.

Roel intends to organize a sprint in Brussels, around the time of the XMPP summit and FOSDEM. This sprint will focus on UI/UX issues and on finding a common user-facing terminology for XMPP clients. Roel comes from a design background and will be inviting some fellow like-minded designers to join.

Link Mauve indicated that he’d like to organize a sprint in Paris in 2019 and pep. thinks we should go to Sweden.

During our sprint we also found out that the Berlin XMPP Meetup group is considering organizing a sprint there soon.

Then there’s also the upcoming 35C3, where there will be an XMPP assembly and the opportunity to hack together.

Thanks to everyone who joined the Dusseldorf sprint, some of whom spent a considerable sum of money in order to be there. I had a great time and am looking forward to more such events coming in 2019.

Come chat with us

We hang out in the XMPP-Sprint groupchat and this is also where upcoming XMPP sprints will be discussed and planned.

You can visit the webchat at https://chat.cluxia.eu/anon/#xmpp-sprint or click on xmpp:xmpp-sprint@chat.cluxia.eu?join to open it in an installed client.

Converse 4 released

Fri, 21 Sep 2018 00:00:00 GMT

After more than 7 months of active development, Converse 4 has finally been released.

Converse is an open source XMPP-based chat client written in JavaScript and which runs in your browser.

This release contains lots of highlights, including rewriting the UI to use Bootstrap 4, support for OMEMO Encryption of private messages, message corrections and file-sharing via HTTP file upload.

XMPP is an IETF standardized messaging and presence protocol with multiple independent server and client implementations.

Unlike other popular open source teamchat applications like Mattermost and Rocket.chat, Converse doesn’t depend on any particular server (e.g. backend) application. Any XMPP server which supports the relevant extensions (aka XEPs) will do.

Converse is 100% front-end JavaScript and CSS. The only backend you need is an XMPP server, which you can either set up and host yourself or you can sign up on an existing one.

XMPP’s killer feature

XMPP allows for something called federation. Even if you’ve never heard the word being used in this context, you already know what it means if you’ve used email.

When you sign up for an email account from one provider (for example Gmail, Fastmail or Kolab), you can still send emails to people who have accounts at other providers. That’s because Email is also federated.

A federated protocol allows you to communicate with user-accounts on foreign hosts. The network is decentralized between many providers (who all adhere to a common protocol), instead of being controlled by a single centralized behemoth (like Facebook, Twitter or LinkedIn).

Federation is getting quite a lot of renewed attention lately, in large part due to the hype around the federated social networking site Mastodon which is based on the recently published ActivityPub standard.

You’re not forced to federate when running an XMPP server for your organisation, friends or family, and sometimes it makes sense not to.

However it’s in my opinion the killer feature of XMPP and one of the main reasons why I’m working on Converse.

Federation implies user-freedom and user-choice, and it’s one thing that Slack or Discord simply cannot do. Not because it’s not technically feasible, but because their business models, based on user lock-in and control, don’t allow it.

It’s also something which the open source Slack-alikes like Mattermost and Rocket.chat don’t have.

I believe people should have the freedom to use the chat app of their choice, while still being able to effectively communicate with their contacts who prefer to use a different chat app.

XMPP and federation can make this a reality.

In a world of federated chat apps, some people will use a commandline client like Poezio, a desktop client like Dino or a mobile client like Conversations (or a combination of all three).

There will also be a growing need for web-based chat clients, and that’s were Converse comes in.

Federation not only provides choice between chat clients, it can also foster inter-departmental or inter-organisational communication.

In Germany, where I currently live, there is a lot of talk of digitization of government. Government bureaucracy often consists of various different departments who need coordinate and send resources around.

The same of course applies to large enterprises.

A decentralized and federation communications protocol can play a large role here, and I’m not just talking about chat apps.

The Salut à Toi project has shown how you can create a decentralized and federated issue tracker, while Movim has blogging and microblogging capabilities, all based on XMPP.

Converse’s history

Converse was originally developed at Syslab.com as part of a web-based intranet for the Star Alliance airline group.

Back then it was developed as a Gtalk-like popover chat, like you still see on LinkedIn.

After open sourcing it, my initial goal was to make it as configurable and capable as possible, so that it could be used in any website in almost any conceivable manner.

A powerful plugin architecture and API was added, which allows you to add or remove features and change just about any behaviour.

Over time, as the project matured and people started using it, it became clear that there are many more usecases for a webchat than simply popover chats.

Full-page teamchat apps like Slack also became spectacularly successful.

I subsequently updated Converse to have different “view modes”, which enable you to use it in different ways, depending on your needs.

The focus also shifted slightly from presenting it merely as something to be integrated into an existing website, to showing that it can be an independent chat client in its own right.

There is the “overlay” mode, which matches the original UX design:

Then there are the “embedded” and “mobile” modes:

And lastly there’s the “fullscreen” mode, more similar in UX to teamchat applications like Slack and Discord.

All these modes use the same JavaScript and CSS, and you can switch between the view modes. Currently it’s not possible to do so without reloading the page, but we’ll work on making that possible as well.

Working on Converse fulltime

In the beginning of this year, I decided to drop other work commitments and to dedicate this year fully to working on Converse and to see where that leads me.

This was in many ways a daunting leap into the unknown, because I knew I would now have to find more ways to try and earn a living off this work.

I’ve been doing consulting and custom development around Converse for a few years, but never exclusively. I also wasn’t sure what business model I should adopt.

Consulting? Proprietary plugins? Hosting? Sponsorships?

I decided to first just focus on building the product, to make Converse better, and not to worry about making money just yet.

In part this is due to my bias as engineer, and being out of my comfort zone when trying to monetize something.

But I also believe that a compelling product, that users love, can go a long way to helping you come up with a sustainable and feasible business model. I hoped that along the way people might appear who would help me realize this vision.

It’s the "if you build it, they will come" approach.

During this time, various people did in fact reach out to me and some of them funded further development.

I also got enough consulting and development gigs to keep the lights on and to keep me going.

Marc Laporte from Wikisuite funded many hours of work moving the UI to Bootstrap 4, and then some more.

Converse will become part of Wikisuite, together with the OpenFire XMPP server.

He gave me valuable advice on how to make a living off open source software and encouraged me to make the leap.

I also got a call from Happy Dev, who needed a federated webchat solution for their Startin’ Blox project (more on that in future posts). We’re doing some interesting work for them, which addresses some pain points with XMPP and will be of benefit to the larger community.

Nicolas Vèrité, of Nayego funded responsiveness improvements while individuals like Christian Weiske and Rafael Munoz generously contributed funds as well.

Some people also backed me on Patreon and Liberapay.

Every gig or donation I got was for me a vote of confidence, and gave me the opportunity to stay the course and continue down this path.

Besides monetary contributions, many people contribute features and bugfixes, and hang out in the Converse chatroom, answering questions, providing their expertise and generally making it a more lively, interesting place.

In many ways this release would not have been possible without all of their contributions, and I therefore dedicate it to everyone who has contributed money, time or energy (all interchangeable actually) to this project and therefore has contributed to its success. Thank you!

The archetypal nature of free software communities

Over the last year I’ve witnessed the Converse project becoming a community and not just a collection of software. It’s a subset of the larger XMPP community and one that overlaps with various other XMPP and free software related communities.

This has been for me one of the most magical and satisfying things about being involved in open source.

Community is archetypally feminine¹, in the sense that it’s something that cannot be enforced, demanded or created out of domination, control or rule-making (which are archetypally masculine attributes). Instead, it’s something that arises seemingly spontaneously, after an unknown period of gestation and only after the groundwork has been laid and perhaps after enough loving care and attention has been provided. Its organic, emergent nature means that it doesn’t strictly adhere to top-down schedules and deadlines.

My wife and I, both interested and fascinated by Jungian archetypes, are amazed at the archetypally feminine aspects of her pregnancy, which is soon coming to an end (and new beginning).

As much as we as a society try to control, monitor and manage a pregnancy, if you decide to have a natural birth, then you’re waiting for it to happen on its own accord, and aren’t able to impose your will and deadlines on it.

In some ways the same is true for a software commons. A commons is a shared resource which no-one has exclusive control or ownership of. This lack of ownership and exclusive control makes people resilient against the software being used as a tool for domination and exploitation. It also means that the project (i.e. the software and its surrounding community) is more organic.

This coming together between software as inanimate artifact, and community as a living breathing organic hive-mind, brings to mind a Taoist union of Yin and Yang, the archetypal feminine and masculine.

Incidentally, the Converse logo is a stylized Yin Yang symbol. Many people don’t know this because I did such a poor job of designing it. I’m still looking for a better logo design which encapsulates these principles of harmony between community and software.

What does the future hold for Converse?

This has turned out to be a relatively strange blog post, where I wanted to originally write about the features of the 4.0 release and instead took an ever more esoteric detour.

In any case, let’s try to salvage what’s left of the original intent by closing off with what’s in the pipeline for future releases.

Message markup

Converse still doesn’t support markdown-like syntax highlighting. This is a top-priority for future releases.

Voice messages

With voice messages I mean sending voice recordings, not audio streaming (although that might also come sometime).

My wife, friends and family use voice messages (via Conversations) a lot and I believe they’d be a useful addition to Converse as well.

Email notifications

One area where XMPP needs to improve is sending (email) notifications when you’re offline and mentioned in a groupchat. XMPP’s groupchat presence rules have made this difficult. Together with Happy Dev and Matthew Wild from Prosody I’m working on a potential solution.

Some of the code for this solution is already in Converse, and we’ll submit a specification for standardization (a so-called XEP) once the time is right.

Making Converse a progressive web-app

There are various features of so-called progressive web apps which will make Converse a much better client, such as:

Push notifications even when Converse is not open.
Using IndexedDB for caching, thereby avoiding storage limitations of localStorage and sessionStorage.
Offline support for editing and sending messages.

Persistent login

Other webchat clients like Mattermost and Rocket.chat can keep you constantly logged in via cookies or perhaps JWT.

XMPP-authentication isn’t cookie-based and it’s not safe to store user credentials in the browser cache. This means that users often have to login anew when they open Converse in a browser tab.

The Credential Management API might help here, but it’s still Chrome-only and I haven’t studied it enough to know whether it’s a solution.

Another approach is to allow Converse to also use authentication cookies by adding support for them to XMPP servers.

Matthew Wild has already added support for cookie authentication to Prosody but more work needs to be done to get this working on conversejs.org.

OAuth-based login

Converse already supports OAuth-based login, but it’s not yet deployed to conversejs.org and more work can be done to fetch and show your user profile from the OAuth provider.

Let’s build this together

There are many more features and improvements planned, too many to list here.

The health and sustainability of this project depends in large part on its community of users and contributors.

The community is invaluable in finding bugs, making feature suggestions and doing quality control that would otherwise not be possible given the circumstances.

Another important requirement for the long-term health and viability of the project is funding.

I and others are spending much of our own time working on Converse, but to make this project sustainable we need more funding. Either through donations (e.g. Patreon), funded feature requests or by using my services as an XMPP and Converse consultant and developer.

Another option is allowing an employee to spend some of their paid time on helping the project with bugfixes and other contributions.

This applies in general to open source projects. Companies derive immense value from them, and the vast majority don’t have wealthy corporate backers like Facebook and Google.

I also intend to soon start offering hosted solutions for organisations who want to use Converse as their preferred teamchat solution.

Please reach out to me here if you’re interested.

References

I’ve been thinking a lot about archetypes (and Jungian-analysis) the last while, in part due of various books I’ve read that deal (partly) with the topic.

Recommended reading, if you’re interested:
↩

Disabling Disqus comments on my blog

Fri, 01 Jun 2018 00:00:00 GMT

I’ve decided to disable Disqus comments on this website. I was quite shocked recently when I saw tacky ads displayed in my blog. I’ve been using and ad blocker for so long that I never saw them, and only when I loaded my site in a new browser were the ads visible.

Advertising has its time and place. Being informed of a useful product or service can be valuable.

Like so many things in modern day society, the reality is far, far removed from the ideal. Much of modern day advertising is manipulation, trying to make you feel inadequate and lacking, so that you’ll be more susceptible and willing to consume a new product.

Adam Curtis’ documentary “Century of the Self” brilliantly lays out the development of marketing as a tool of mass manipulation.

The dominant form of advertising on the web, so-called Adtech, is outright fraudulent and user-hostile. Megabytes of JavaScript are included in webpages with the sole purpose of tracking you as you visit various websites so that they can build a more complete psychological profile of you.

In theory such a profile can help to inform you of products and services more akin to your needs and taste. In practice, it gets used to manipulate you and to reinforce your filter bubble.

I knew that Disqus is in this game as well, and should have acted sooner. I enabled Disqus comments probably about 5 or 6 years ago, when I was less concerned or aware of the problems of privacy and manipulation.

As always with information silos, the comments that were made on this website are now all locked away and lost to readers. Perhaps with the GDPR I can ask for them to be downloaded and then I can create a static representation of older comments.

Eventually I’d like to create a comments plugin with Converse.js and based on XMPP and use that on this site.

2018 Gulaschprogrammiernacht and organizing sprints for XMPP

Thu, 31 May 2018 00:00:00 GMT

Recently I attended the Gulaschprogrammiernacht for the first time.

It’s a hacker/maker event in the Zentrum für Kunst und Medien (Centre for Arts and Media) in Karlsruhe, Germany.

AFAIK it’s organized by the local chapter of the infamous Chaos Computer Club.

I heard about it from Daniel Gultsch on Twitter. It sounded like fun, so I decided to attend and spend the time adding OMEMO support to Converse.

Guus der Kinderen and I intended to organize an XMPP sprint for that weekend in Düsseldorf, but we were cutting it a bit fine with the organization, so I hoped that we could just shift the whole sprint to GPN.

Unfortunately Guus couldn’t attend, but Daniel and Maxime Buquet (pep) did and I spent most of the event hanging out with them and working on XMPP-related stuff. The developers behind the Dino XMPP client also attended and hung out with us for a while and there was someone working on writing an XMPP connector for Empathy in C++.

Maxime worked on adding OMEMO support, to Poezio, and Daniel provided us with know-how and moral support. Daniel worked mainly on the Conversations Push Proxy.

We had some discussions around the value of holding regular sprints and I told them about my experience with sprints in the Plone community.

The Plone community regulary organizes sprints and they’ve been invaluable in getting difficult work done that no single company could or would sponsor internally. To me it’s a beautiful example of what’s been termed Commons-based peer production.

The non-profit Plone foundation provides funding and an official seal of approval to these sprints, and usually a sprint has a particular focus (such as adding Python3 support). Sprints can range from 3 people to 30 or more.

One difference between the Plone and XMPP communities, is that Plone is a single open source product on which multiple companies and developers build their businesses, whereas XMPP is a standardized protocol upon which multiple companies and developers create multiple products, some open source and some closed source.

In both cases however there is a single commons which community members have an incentive to maintain and improve as they build their businesses around it.

Another difference is between the Plone Foundation and the XMPP Standards Foundation. The XSF, for better or worse, interprets its role and function fairly strictly as being a standards organisation primarily focused on standardising extensions to XMPP, and less on community building or supporting software development.

Despite these differences, I still consider sprints a great way to foster community and to improve the extent and quality of XMPP-related software and documentation.

There is an interesting dynamic between cooperation and competition in both the Plone and XMPP communities. Participants compete with one another but they also have the shared goal of maintaining and growing a healthy software ecosystem.

Maxime was particularly excited by our discussion and very quickly put word into action by planning and announcing an XMPP sprint in Cambridge, UK in August.

There’s still time to vote on the date of the sprint and to suggest topics.

Hopefully this will be first of many more sprints and communit events.

Slack's bait and switch

Mon, 12 Mar 2018 00:00:00 GMT

Slack has finally decided to close down their IRC and XMPP gateways. True to form, you can only read their announcement if you already have a Slack account and are logged in to a workspace.

Here ‘s the gist of their announcement:

As Slack has evolved over the years, we’ve built features and capabilities --- like Shared Channels, Threads, and emoji reactions (to name a few) --- that the IRC and XMPP gateways aren’t able to handle. Our priority is to provide a secure and high-quality experience across all platforms, and so the time has come to close the gateways.

They’re of course being economical with the truth here. Perhaps their XMPP gateway can’t handle “Shared Channels” and “Threads”, but that’s because they purposefully stopped working on it.

A “Shared Channel” simply means a chatroom which people from outside your workspace can participate in. If a workspace is mapped to a members-only chatroom, then making something a shared channel simply means updating the members list or making the chatroom open (so anybody can join it). Threads can be implemented by adding a <thread> element in the message stanza, as documented in XEP-0201.

Emoji reactions can be tacked to a particular message. There’s nothing fundamental about XMPP that prevents emoji reactions, and [work is currently underway to add support for them](https://github.com/jabber t/jabbercat/issues/80). The protocol is designed to be eXtensible (hence the X in XMPP) and new features are continuously being added.

bait and switch

We all know the real reason Slack has closed off their gateways. Their business model dictates that they should.

Slack’s business model is to record everything said in a workspace and then to sell you access to their record of your conversations.

They’re a typical walled garden, information silo or Siren Server

So they have to close everything off, to make sure that people can’t extract their conversations out of the silo.

We saw it with Google, who built Gtalk on XMPP and even federated with other XMPP servers, only to later stop federation and XMPP support in favour of trying to herd the digital cattle into the Google+ enclosure.

Facebook, who also built their chat app on XMPP at first allowed 3rd party XMPP clients to connect and then later dropped interoperability.

Twitter, although not using or supporting XMPP, had a vibrant 3rd party client ecosystem which they killed off once they felt big enough.

Slack, like so many others before them, pretend to care about interoperability, opening up just so slightly, so that they can lure in people with the promise of “openness”, before eventually closing the gate once they’ve achieved sufficient size and lock-in.

On Federation

When we talk about “federation” in networks, we mean the ability to communicate between different service providers.

For example, email is federated. You can set up your own email server, and then send emails to people with their own email servers, or to people with Gmail or Yahoo! accounts.

You can email any other email address in the world, regardless of where that email address is hosted.

If email never existed, and a company like Slack today would come out with this brand new concept of “Electronic Mail”, let’s call it digimail, do you think they would standardise the digimail protocol and allow you to send messages to other digimail purveyors?

We all know the answer to that. They won’t, and neither would Google, Microsoft or Facebook.

Heck, Facebook is actively trying to replace email since years.

The reason email is federated, is because it was developed before surveillance capitalism was a thing and because it was established and entrenched long before these companies came around.

There’s a reason why your email address is still the de facto way to sign up for any service on the web (sometimes with one or two degrees of separation), and it’s because of federation.

XMPP is designed to allow federation. Think about that. Instead of having to sign up to various different chat providers, all which try to lock you in and monetize your conversations, you could instead have one chat account, and use that to chat with anybody else, regardless of which chat provider they are using.

Alas, that’s the dream, but because XMPP came much later to the scene, it didn’t develop the critical mass as email has, and here we are. With dozens of chat apps, all non-interoperable and closed off.

What would it take for XMPP to take off?

One of the sad things that has come out of Slack’s meteoric rise to success, has been how many free and open source projects have jumped over to using it (after previously using IRC or XMPP).

In so doing, they have closed off their discussions from search engines and they prevent people from accessing their past archives.

Slack has many cool features, and they work very well, I’m not going to deny it.

However, the XMPP Software Foundation has done a lot of work in recent years to enable protocol extensions that provide features that people have come to expect from chat applications, for example:

Message threading XEP-0201
Last message corrections XEP-0308
Message delivery receipts XEP-0184
Markdown-like syntax for messages XEP-0393

Unfortunately XMPP clients have been lagging far behind in various respects.

One of the main problems is funding. The modern digital economy is largely set up around surveillance capitalism and user lock-in.

So attempts to create software that doesn’t follow these precepts, often end up unfunded or underfunded.

However, our “weakness”, is also our strength.

XMPP clients, and the XMPP network can provide something that Slack never can. Federation, free and open software, interoperability, extensibility and user choice.

Converse

For the last few years I’ve been working in my spare time on making a JavaScript XMPP chat client, called Converse.

Originally the idea was to make a Gtalk like chat client that you integrate in your website, and it can still be used like that.

However, in the last year I’ve updated it further so that it can also be used as a fullscreen application, like Slack is used.

You can try the fullscreen version at https://inverse.chat/

If you have no-one to chat to, then come join the discuss@conference.conversejs.org chat room.

This link will take you directly there (via inverse.chat): https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org

Converse still lacks lots of features that Slack has, but that’s not because XMPP itself can’t support those features.

What Converse however does have, is that it’s free and open source software, based on a standard protocol and it can be extended, updated and improved upon, by anyone.

We’re actively working on adding new features and more and more people are joining in.

Moreover, anybody can host it and you can integrate it into any website.

Ultimately, I believe in the power and utility of interoperability and software freedom, even though the current trend is to close off and lock down.

These information silos are as powerful as we make them. If enough projects choose standardised protocols and FOSS software, we will be able to create viable alternatives that foster freedom instead of lock-in.

This blog post triggered a lively discussion on Hacker news, you can read that here:

A badge for linking to your XMPP chat

Mon, 19 Feb 2018 00:00:00 GMT

You’ve probably noticed the coloured badges at the top of project READMEs on Github and Gitlab.

These badges indicate various metrics about the project’s current state and its supporting infrastructure.

For example, the README page for Converse contains the following badge:

It indicates how many open bounties there are for Converse on Bountysource.

There are dozens such badges available nowadays, just take a look at shields.io.

Some projects have badges that link to a chatroom, hosted by Gitter, Slack etc.

Converse, being an XMPP-based project, has its own XMPP chatroom at discuss@conference.conversejs.org.

So I created a badge for its README, which points to that chatroom and also indicates how many users are currently in it.

It looks like this:

The Markdown syntax for showing the badge looks like this:

[Link to XMPP chat](https://inverse.chat/badge.svg?room=discuss@conference.conversejs.org)

The target URL for this badge is https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org, which means that it takes the user to https://inverse.chat and then once the user has logged in, the room discuss@conference.conversejs.org will be opened.

To go to a different room, just specify a different JID (Jabber ID) in the URL parameters.

The image URL is https://inverse.chat/badge.svg?room=discuss@conference.conversejs.org

Here you specify the room JID with the room URL parameter.

The image is an SVG which is generated by a tiny Flask app, which includes a chatbot written with SleekXMPP that checks how many users are in a particular chatroom.

The code is open source, and hosted on Github at jcbrand/xmpp-chat-badge.

If you’re using this badge for your project, I’d be happy to hear about it.

You can let me know in the Converse XMPP chatroom at discuss@conference.conversejs.org (or through the web with inverse.chat) or by using this website’s contact form.

The 2018 XSF Summit

Sat, 03 Feb 2018 00:00:00 GMT

This year I again attended the XMPP Software Foundation’s summit in Brussels.

The summit spans two days and is held at Cisco’s offices.

It’s a rare opportunity to get community members together to discuss and make decisions around protocol extensions (the so-called XEPs).

On the second day interesting discussions were had regarding the state of XMPP today, for example how and by whom its used, how to foster and support client development and the merits of federation (i.e. communication between different messaging providers).

We heard that XMPP is heavily used and respected within the Telecoms industry, and that there’s a lot of “hidden” use of XMPP that even most of the Summit participants were unaware of.

The distinction was made between XMPP, which represents the protocol, and Jabber®, which represents the ideal of federated instant messaging as used by end users.

Besides what some people may think, XMPP is alive and well, and used extensively in many applications and in various sectors. Jabber®, on the other hand, not so much. Mindshare has moved largely to siloed (i.e. non-federated and non-standardized) messaging apps like Slack et al.

The Tigase guys Andrzej Wójcik and Daniel Wisnewski gave a demo of the new Internet-of-things module of the Tigase XMPP server which also drew attention at the realtime communications stand at FOSDEM.

Photo by Guus der Kinderen

Minutes for the XSF Summit

I volunteered to be the minute taker this year, for which I used my favorite notes-taking app: Vimwiki.

With Vimwiki you can generate HTML from your wiki files, so I was able to make the notes available online roughly as I was typing them. which I did by pushing the generated HTML to a git repo and then pulling again via cron on a server and serving with Nginx.

I also wanted to make the minutes available on the XMPP wiki, but Vimwiki’s syntax is a little bit different than the Mediawiki syntax.

Pandoc came to the rescue, and with a one-liner I could generate Mediawiki files from all my notes.

So here are the 2018 XSF Summit minutes on the XMPP wiki.

Final thoughts

It was fun to meet up again with kindred spirits and to see some familiar and friendly faces from the XMPP world.

This year was also the first time I attended the fancy XSF dinner. I had the sea bass with endive and beurre blanc.

See you folks next year and thanks for all the fish!

Converse 3.3 has been released

Thu, 18 Jan 2018 00:00:00 GMT

Last night I released version 3.3.0 of Converse.js, and as often happens with big releases, I made a quick bugfix release (3.3.1) today. The bugfix release turns off some CSS3 animations for new messages which caused degraded performance on Firefox. On Chrome the animations render smoothly, so if you’d like you can still turn them on with the show_message_load_animation config option.

What’s in the release?

Maintaining a long-term open source front-end JavaScript library almost feels like a Sisyphean task sometimes. As soon as you’ve rolled the big stone up the hill, the whole JS ecosystem, best practices and tooling changes and you find yourself at the bottom of the hill again.

This release is therefore heavy on changes under the hood, with the aim of modernizing and improving the quality of the code.

Besides that, I also spent time squashing lots of small usability bugs and on improving performance.

Converse.js now uses a Virtual DOM

Various views, such as the registration form, are now rendered by means of a Virtual DOM. I wrote a new Backbone view called Backbone.VDOMView for this, and blogged about it here.

No more jQuery

Looking at the git log, I started rewriting code to not use jQuery in January 2017.

So this change has been a year in the works. I often asked myself whether I should spend time on this and not rather do something else, like adding new features, especially since removing jQuery has taken a lot of time.

However, there were some good reasons, or perhaps motivations, for me to remove jQuery.

Before ES6 promises were available, I used $.Deferred. However, jQuery’s deferreds weren’t compatible with Promises, so when ES6 Promises came around, I had to rewrite lots of code to use Promises.

I used $.show and $.hide quite a bit, and then it turned out that the way jQuery was doing it (by adding/removing display: none to the DOM element) is not good practice and also very slow.

So I started writing my own utility functions to replace jQuery’s.

The last straw for me was when jQuery 3 came out, and half of Converse.js’s ~240 tests failed once I plugged it in.

After spending some time trying to figure out what backward incompatible changes they made and how I should update the code, I decided to instead rip jQuery out entirely.

It’s still used in the tests, but it’s no longer included in any build.

Since removing it, I’ve noticed a remarkable reduction in time to run the tests.

By looking at how quickly the tests run now, the code seems to run much faster without jQuery.

Less weight

Besides removing jQuery, I also updated Converse.js to load translations at runtime, and only the exact translation JSON file that’s needed.

Due to these changes, the unminified built has shrunk from 3.38MB to 2.8MB, and the minified build from 1.66MB to 1.2MB.

And this is while adding the virtual DOM code.

Route to a specific chat room via the URL

It’s now possible to directly link to a specific chat room, or to the registration page (instead of the login page) via the URL.

For example, the URL https://inverse.chat/#converse/room?jid=discuss@conference.conversejs.org will take you immediately to the Converse.js “Discuss” chat room, once you’ve logged in.

What else?

Lots of other bugfixes and improvements have been added in this release. For more details, check the changelog.

Notable absent from this release are some desired features, such as file sharing, message corrections, message receipts and the like.

I would love to add some of these often requested features, however I had to get the house in order so to speak, by modernizing the code and squashing lots and lots of little usability and performance bugs.

That said, Converse.js takes up a LOT of my free time and not a single line of code in this release was paid for.

If you or your company make use of converse.js, please consider sponsoring it on Patreon or Liberapay.

Thanks

Thanks goes out to everyone who’s made pull requests and bug reports over the last months.

And thanks also to the folks who hang out in the Converse.js Discusss chat room and who have there provided valuable feedback.

Using a virtual DOM with Backbone views

Tue, 16 Jan 2018 00:00:00 GMT

Backbone.js, first released in 2010, provided one-way event binding with unidirectional data-flow and therefore a means to do reactive programming before it was cool. Backbone views provide a render method, which is supposed to insert HTML into the DOM. How this was actually done was left up to the user as a homework assignment. Generally, jQuery’s html method was used. Like so:

const MyView = Backbone.View.extend({
    initialize () {
        this.model.on('change', this.render, this);
    },
    render () {
        this.$el.html(this.template(this.model.toJSON()));
        return this;
    }
});

Each view has an associated DOM element, this.$el if you’re using jQuery or this.el otherwise. In the above example, the element’s contents would be replaced.

What this approach however lacks, is a way to rerender views containing forms without losing the input focus or user-provided form values.

This is because back in 2010, the virtual DOM as popularized by React wasn’t yet around.

Additionally, since rendering the HTML and inserting it into the DOM were both done inside the render method, and the implementation details were left up to the user, enough rope was provided for you to hang yourself by putting things inside the render method which shouldn’t be there.

Contrast this to a React component’s render method, which is a “pure function” that doesn’t modify the DOM, and instead simply returns JSX:

class MyComponent extends React.Component {
    render() {
        return <p>Hello World!</p>;
    }
}

The markup is then inserted into the DOM separately with ReactDOM.render

ReactDOM.render( <MyComponent />, document.getElementById(‘root’) );

The fact that a component’s render method must return JSX, means that they are pure functions without side-effects.

Backbone Views which use a virtual DOM

In Converse we still use Backbone Models and Views, and for the most part, Backbone still works beautifully. It’s a small, lightweight library that’s fit for purpose and doesn’t get in the way.

However, we’ve been missing out on that sweet sweet virtual DOM.

No more!

To make our Backbone Views more reactive, as is done in React, I wrote Backbone.VDOMView.

This library adds a new view, called Backbone.VDOMView, which uses a Virtual DOM (provided by Snabbdom) to only update those DOM elements (within the view) as necessary.

Instead of writing a render method, for Backbone.VDOMView you need to implement a toHTML method, which is a pure function that simply returns the view’s HTML.

Here’s what it looks like:

const MyView = Backbone.VDOMView.extend({
    initialize () {
        this.model.on('change', this.render, this);
    },
    toHTML () {
        return this.template(this.model.toJSON());
    }
});

Backbone.VDOMView takes the output of toHTML and converts it into virtual DOM nodes and automatically diffs and patches the DOM.

When the view’s associated Backbone.Model (referenced via this.model) changes, we can simply call render again, knowing that only the changed DOM elements will be updated.

We write less to the DOM and we prevent the loss of form input values and loss of element focus.

Include the root element in your HTML

There’s one important difference with normal Backbone.Views and that is that toHTML (as opposed to render) must including the view’s root element in the returned HTML.

With normal Backbone.View, you don’t include the view’s root element when you render a template.

React has a similar requirement to this. JSX returned in the render method of a component must have a root node which contains everything else.

jQuery not required

With Backbone, you can use Backbone.NativeView to remove the default dependency on jQuery.

Backbone.VDOMView checks whether you are using Backbone.NativeView and will then subclass it instead of Backbone.View.

Usage in Converse

In release 3.3.0 of Converse I’ve removed jQuery as dependency and started integrating Backbone.VDOMView.

Various Backbone.View components from Converse have already been migrated over to Backbone.VDOMView.

So far, I’m very happy with the results, and haven’t run into any issues.

If you’re using Backbone.VDOMView or have any feedback, feel free to leave a comment or contact me here.

Custom SASL auth mechanisms in Strophe.js

Sat, 05 Nov 2016 00:00:00 GMT

If you’ve decided to read this fairly technical blogpost, then you probably have at least a rough idea what SASL is about and why one would want to create custom SASL auth mechanisms or whitelist the supported mechanisms.

I’ll therefore provide just a very brief recap of the topics involved:

The Simple Authentication and Security Layer or SASL RFC 4422 is a framework for adding authentication support to connection-based protocols.

It provides an abstraction layer for authentication mechanisms, so that protocols, such as XMPP don’t have to deal with the intricacies and complexities of supporting multiple authentication mechanisms.

It therefore makes auth mechanisms pluggable (if they are SASL compatible).

Strophe.js has supported SASL since a long time, but it didn’t provide an easy way add custom SASL mechanisms, or to whitelist the mechanisms to be used.

Until now… or rather, since the 1.2.9 release.

Creating a custom SASL auth mechanism

To create a custom SASL authentication mechanism is fairly simple.

You can glean what’s required by simply looking at how the default mechanisms are created.

See for example how the SASLPlain mechanism is defined.

And look as the SASLMechanism prototype to see the interface that the mechanism supports.

Perty much it boils down to creating constructor, settings its prototype to an invoked Strophe.SASLMechanism instance, providing its name, a boolean to indicate whether it should proactively respond without an initial server challenge, and an integer value specifying its priority amongst the supported mechanisms.

The default mechanisms and their respective priorities are:

EXTERNAL - 60
OAUTHBEARER - 50
SCRAM-SHA1 - 40
DIGEST-MD5 - 30
PLAIN - 20
ANONYMOUS - 10

Then it’s a matter of implementing onChallenge and any of the other methods provided by the SASLMechanism prototype.

onChallenge is called once the server challenges the client to authenticate itself or proactively if the mechanism requires that the client initiates authentication (configured with the isClientFirst parameter of Strophe.SASLMechanism).

So, lets create a fictional auth mechanism called SASL-FOO which works similarly to SASL-PLAIN, except that the password is encrypted with double-encoding ROT13 (hint: this is a joke).

We would then create the authentication mechanism like so:


Strophe.SASLFoo = function () {};
Strophe.SASLFoo.prototype = new Strophe.SASLMechanism("FOO", true, 60);

Strophe.SASLFoo.prototype.onChallenge = function (connection) {
    let auth_str = connection.authzid;
    auth_str = auth_str + "u0000";
    auth_str = auth_str + connection.authcid;
    auth_str = auth_str + "u0000"; auth_str = auth_str + DoubleROT13(connection.pass);
    return utils.utf16to8(auth_str);
};

Whitelisting the supported SASL auth mechanisms

Now with SASL-FOO in hand, we can whitelist the supported authentication mechanisms by specifying a list of mechanisms in the options map passed in when we instantiate a new Strohe.Connection.

var service = 'chat.example.org';
var options = { 'mechanisms': [SASLFoo, Strophe.SASLPlain ] };
var conn = new Strophe.Connection(service, options);

Bonus: Whitelisting SASL auth mechanisms in Converse.js

Due to the above changes it’ll also be possible to whitelist SASL mechanisms in Converse.js (version 2.0.1 and upwards).

This is done via the connection_options configuration setting:


converse.initialize({
  connection_options: {
      'mechanisms': [converse.env.Strophe.SASLMD5, converse.env.Strophe.SASLPlain]
  }
});

Happy hacking.

Strophe.js and Converse now support passwordless login with client certificates

Thu, 15 Sep 2016 00:00:00 GMT

Did you know that x509 certificates, the certificates that webservers use to prove their identity during the establishment of an HTTPS connection, can also be used by a client (like your webbrowser) to prove its identity, and even to authenticate?

I’m talking here about so-called client certificate authentication.

Client certificate authentication is especially popular in environments with high security requirements. They can even be used to enforce 2-factor authentication, if in addition to a client certificate you also require a password. That usecase is however out of scope for this blog post.

With the release of Strophe.js 1.2.8, it’s now possible to have passwordless login with TLS client certificates in Converse and any other Strophe.js-based webchat projects.

For Converse, you’ll need at least version 2.0.0.

Here’s what it looke like:

The technical details and background

XMPP and SASL

XMPP supports authentication with client certificates, because it uses SASL (Simple Authentication and Security Layer).

SASL provides an abstraction that decouples authentication mechanisms from application protocols.

This means that XMPP developers don’t need to know about the implementation details of any authentication mechanisms, as long as they conform to SASL.

Up til version 1.2.7, Strophe.js supported the SASL auth mechanisms: ANONYMOUS, OAUTHBEARER, SCRAM-SHA1, DIGEST-MD5 and PLAIN.

For client certificate auth, we need another SASL mechanism, namely EXTERNAL. What EXTERNAL means, is that authentication happens externally, outside of the protocol layer. And this is exactly what happens in the case of client certificates, where authentication happens not in the XMPP layer, but in the SSL/TLS layer.

Strophe.js version 1.2.8 now supports SASL-EXTERNAL, which is why client certificate authentication now also works.

How do you communicate with an XMPP server from a web-browser?

There are two ways that you can communicate with an XMPP server from a web-browser (e.g. from a webchat client such as Converse).

You can use XMLHttpRequests and BOSH, which you can think of as an XMPP-over-HTTP specification.
You can use websockets.

Both of these protocols, HTTP and websocket, have secure SSL-reliant versions (HTTPS and WSS), and therefore in both cases client certificate authentication should be possible, as long as the server requests a certificate from the client.

I’m going to focus on BOSH and HTTPS, since this was my usecase.

The HTTPS protocol makes provision for the case where the server might request a certificate from the client.

NOTE: At the time of writing the only XMPP server that supports client certificate authentication with BOSH is Openfire, and funnily enough, only Openfire 3. In Openfire 4, they refactored the certificate handling code and broke client certificate authentication with BOSH. I’ve submitted a ticket for this to their tracker: https://issues.igniterealtime.org/browse/OF-1191

The authentication flow

So this is how the authentication flow works. I’ll illustrate how the authentication flow works by using actual log output from Converse

NOTE: My XMPP server’s domain is called debian, because I was running it on a Debian server and because naming things is hard. In hindsight, this wasn’t a good name since it might confuse the dear reader (that means you).

2016-09-15 12:07:05.481 converse-core.js:128 Status changed to: CONNECTING

Firstly, Converse sends out a BOSH stanza to the XMPP server debian, to establish a new BOSH session.

2016-09-15 12:07:05.482 converse-core.js:128

<body rid="1421604076"
    xmlns="http://jabber.org/protocol/httpbind"
    to="debian"
    xml:lang="en" wait="60" hold="1"
    content="text/xml; charset=utf-8" ver="1.6"
    xmpp:version="1.0"
    xmlns:xmpp="urn:xmpp:xbosh"/>

2016-09-15 12:07:06.040 bosh.js:749 XHR finished loading: POST https://debian:7445/http-bind

The above stanza was sent as an XMLHttpRequest POST, and the above XML was sent as the Request Payload.

Strophe.js takes care of all this, so nothing to worry about, but sometimes digging through the internals is fun right? Right?!

2016-09-15 12:07:06.042 converse-core.js:128

<body xmlns="http://jabber.org/protocol/httpbind"
      xmlns:stream="http://etherx.jabber.org/streams"
      from="debian" authid="fe0ee6ab" sid="fe0ee6ab"
      secure="true" requests="2" inactivity="30" polling="5"
      wait="60" hold="1" ack="1421604076" maxpause="300"
      ver="1.6">

  <stream:features>
    <mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
      <mechanism>EXTERNAL</mechanism>
    </mechanisms>
    <register xmlns="http://jabber.org/features/iq-register"/>
    <bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/>
    <session xmlns="urn:ietf:params:xml:ns:xmpp-session>"> <optional/> </session>
  </stream:features>
</body>

So now the XMPP server, debian, has responded, and it provides a list of SASL mechanisms that it supports. In this case it only supports EXTERNAL.

Luckily our webchat client supports SASL-EXTERNAL, so it responds in turn and asks to be authenticated.


2016-09-15 12:07:06.147 converse-core.js:128

<body rid="1421604077" xmlns="http://jabber.org/protocol/httpbind" sid="fe0ee6ab">
  <auth xmlns="urn:ietf:params:xml:ns:xmpp-sasl"
        mechanism="EXTERNAL">dXNlcjAxQGRlYmlhbg==</auth>
</body>

Now here comes the tricky part. The XMPP server’s BOSH servlet, asks the webbrowser (which is establishing the HTTPS connection on our behalf) to give it the client certificate for this user.

The webbrowser will now prompt the user to choose the right client certificate. Once this is done, the XMPP server authenticates the user based upon this certificate.


2016-09-15 12:07:06.177 bosh.js:749 XHR finished loading: POST
2016-09-15 12:07:06.180 converse-core.js:128

<body xmlns="http://jabber.org/protocol/httpbind" ack="1421604077">
  <success xmlns="urn:ietf:params:xml:ns:xmpp-sasl"/>
</body>

The XMPP server responds with success and we’re logged in!

How to set up client certificate authentication with Converse and OpenFire 3.10.3

NOTE: Thanks goes out to Dennis Shtemberg from Infusion, who initially tested client certificate authentication with BOSH on Openfire and on whose notes the following is based.

1. Install Openfire 3.10.3

On Debian(-based) Linux, you can simply do the following:

wget http://www.igniterealtime.org/downloadServlet?filename=openfire/openfire_3.10.3_all.deb
sudo dpkg -i openfire_3.10.3_all.deb

2. Configure Openfire’s system properties

Open the admin console: http://localhost:9090/ (where localhost is the host the server is running on)

Navigate to Server > Server Manager > System Properties and add the following properties:

Property                                    Value
------------------------------------------- ----------
xmpp.client.cert.policy                     needed
xmpp.client.certificate.accept-selfsigned   true
xmpp.client.certificate.verify              true
xmpp.client.certificate.verify.chain        true
xmpp.client.certificate.verify.root         true
sasl.mechs                                  EXTERNAL

Make sure the xmpp.domain value is set to the correct host. If you’re running Openfire on localhost, then you need to set it to localhost. If you’re not using localhost, then replace all mention of localhost below with the xmpp.domain value.

3. Lay the groundwork for generating an SSL client certificate

First, make sure you have OpenSSL installed: aptitude install openssl Then create a directory for certificate files: mkdir ~/certs

Now create a config file called user01.cnf (~/certs/user01.cnf) with the following contents:


[req]
x509_extensions = v3_extensions
req_extensions = v3_extensions
distinguished_name = distinguished_name

[v3_extensions]
extendedKeyUsage = clientAuth
keyUsage = digitalSignature,keyEncipherment
basicConstraints = CA:FALSE
subjectAltName = @subject_alternative_name

[subject_alternative_name]
otherName.0 = 1.3.6.1.5.5.7.8.5;UTF8:user01@localhost

[distinguished_name]
commonName = user01@localhost

The otherName.0 value under subject_alternative_name assigns the user’s JID to an ASN.1 Object Identifier of “id-on-xmppAddr”. The XMPP server will check this value to figure out what the JID is of the user who is trying to authenticate.

For more info on the id-on-xmppAddr attribute, read XEP-178.

4. Generate an SSL client certificate

Generate a self-signed, leaf SSL certificate, which will be used for client authentication.

Generate a private RSA key

openssl genrsa -out user01.key 4096

Generate a sigining request:

openssl req -key user01.key -new -out user01.req -config user01.cnf -extensions v3_extensions

when prompted for a DN enter: user01@localhost

Generate a certificate by signing user01.req

openssl x509 -req -days 365 -in user01.req -signkey user01.key -out user01.crt -extfile user01.cnf -extensions v3_extensions

Generate PKCS12 formatted certificate file, containing the private key and the certificate. This will be the client certificate which you will log in with.

openssl pkcs12 -export -inkey user01.key -in user01.crt -out user01.pfx -name user01

when prompted for export password enter: user01

5. Install the PKCS12 certificate on your local machine

Double click the pfx file and follow the steps to import it into your machine’s keystore.

6. Import the x509 certificate into Openfire

sudo keytool -importcert -keystore /etc/openfire/security/truststore -alias user01 -file ~/certs/user01.crt
sudo keytool -importcert -keystore /etc/openfire/security/client.truststore -alias user01 -file ~/certs/user01.crt
sudo systemctl restart openfire

NOTE: The default keystore password is "changeit"

7. Create the user associated with the SSL client certificate

Go back to Openfire admin console, navigate to Users/Groups > Create New User and create a new user.

Username: user01
Password: user01 (This is not controlled by Openfire).
Click Create User

8. (When using Java 1.7) Patch Openfire

When trying to log in, I received the following error:

2016.09.08 00:28:20 org.jivesoftware.util.CertificateManager - Unkown exception while validating certificate chain: Index: 0, Size: 0

Turns out the likely cause for this is the fact that I was using the outdated Java version 1.7.

At the time, I didn’t know that Java is the culprit, so I patched the following code.

If you read the comments in the link above, you’ll see there are two sections, with one being outcommented. I swopped out the two sections, and then recompiled Openfire.

After that, client certificate auth worked. The best way to avoid doing this is apparently to just use Java 1.8.

9. Test login with Converse

Now you’re done with setting up Openfire and you can test logging in with Converse.

Download the latest version of Converse from the releases page.

To hide the password field (since the password won’t be checked for anyway), you need to open index.html in your text editor and add authentication: 'external' to the converse.initialize call.

Then open index.html in your browser.

In the Converse login box, type the JID of the user, e.g. user01@localhost and click login.

NOTE: If things go wrong, pass debug: true to converse.initialize, then open your browser’s developer console and check the output. Check especially the XHR calls to http-bind. Checking the output in the Network tab can also be very helpful. There you’ll see what Openfire responds to requests to its BOSH URL.

Conclusion

Client certificate authentication is a bit of a niche requirement, doing so with BOSH/HTTP even more so.

However, I expect webchat XMPP clients to become more and more prevalent in the coming years, even on the desktop, for example when packaged with Github’s Electron.

The fact that this works because of SASL-EXTERNAL authentication being added to Strophe.js means that this functionality is not only possible in Converse, but all webchat clients built on Strophe.js (granted that they use version 1.2.8 or higher).

Unfortunately XMPP server support is lacking, with only Openfire supporting this usecase currently, and not yet (at the time of writing) in the 4.0.x branch. To see whether this gets fixed, keep an eye on the relevant ticket

Open Source software and the expection of free labor

Sun, 29 Nov 2015 00:00:00 GMT

Over the last few years of starting and then maintaining an open source project that has received a decent amount of attention, converse.js, I’ve noticed some interesting things about the expectations some people have towards developers who work on FOSS (free and open source software).

Predictably irrational

People of course love to receive something for nothing. Dan Ariely, in his book “Predictably Irrational” illustrates some of the biases people have when it comes to free stuff. When confronted with the words “free” (as in gratis), people do things that are irrational and are at odds with how a rational actor (the mythical homo economicus) is expected to behave, which is the bedrock upon which most economic theories are based.

The outcome of the various studies Ariely conducted was consistent: when faced with multiple choices, the free option was commonly chosen. With the opportunity to receive something for free, the actual value of the product or service is no longer considered.¹

“Most transactions have an upside and a downside, but when something is FREE! we forget the downside. FREE! gives us such an emotional charge that we perceive what is being offered as immensely more valuable than it really is.”

Dan Ariely

The biases regarding “FREE!” apply not only to monetary costs, but also to time. We forgo some of our time when we wait in line for free popcorn or to enter a museum on a free-entrance day. We could have been doing something else at that time, so there’s a resultant opportunity cost.²

Freedom isn’t free, it costs folks like you and me

These biases of course can also come into play when people evaluate free (as in beer) software. In the same way that people didn’t take into consideration the cost of the time they spend in trying to get something for “free”, people often also don’t consider the non-monetary costs of using FOSS.

A common retort that usually surfaces on Slashdot, Reddit or Hacker News whenever a discussion around using a Linux distribution on the Desktop takes place, is ”Linux is only free if you don’t value your time“.

That’s of course completely true. I do value my time, took that into consideration and still concluded that I want to use GNU/Linux and free and open source software.

Using FOSS requires a certain amount of commitment, and it should be clear to the user why they are willing to go that route (freedom from vendor-locking, the ability to control and keep private your data, the ability to modify the code to your liking etc.).

I think people have been hyping the “FREE!” aspect of FOSS way too much.

Software for nothing and your support for free

⁴

Disclaimer

I consider a certain amount of support and maintenance as a requirement for a successful open source project and not something you (as the developer) can ignore.

I try to channel bug reports and feature requests to the Github issue tracker and general support questions to a mailing list, where hopefully other people would also be willing to share the load by answering questions.

So while I complain about people wanting “something for nothing” below, I invariably mean people who write to me directly, instead of on the issue tracker and who are often trying to get me to work on something right away.

So, when considering that many people don’t properly evaluate the costs involved in using FOSS, some requests and emails that I sometimes receive start to make sense.

“Please guide me”

One common recurrence, is to be contacted by someone who is integrating converse.js into a project for a paying client, and somehow got stuck. Perhaps they didn’t read the docs or perhaps they don’t have the requisite technical skills to do the job. These emails sometimes have a pleading, desparate tone to them. Perhaps to instill some sense of guilt or obligation or perhaps just because the person is really desperate and under time pressure.

What gets me every time however, is that as far as I can tell, these are people working for commercial businesses who get paid for the work they do. They then trawl the web looking for hapless FOSS developers to do their work for them for free, or as expressed in the commonly used phrase in these kinds of emails: ”Please guide me“.

The novelty and warm fuzzy feeling of altruistically helping strangers solve their problems disappears like mist before the sun when you realise that they’re getting paid for the work you’re doing for them right now.

And make no mistake about it, maintenance and support for an open source project is work and sometimes even drudgery. The fun part is writing new code or trying out new things, not helping people who can’t be bothered to study the documentation.

We need a feature and we hope you’ll do it for free

Another common theme is emails where people somehow just assume that I’ll implement some feature for them. At first this presumptuousness startled me.

I think it’s totally fair to ask when the project is charitable and the people involved don’t receive any payment themselves, but that’s often not the case.

Instead, the underlying assumption appears to be that I love working on open source projects so much that I’ll do it all for free and that I don’t have ideas on what to work on next.

Sometimes people qualify their requests by stating that they’re a small non-profit. Non-profits do however pay out salaries, don’t they?

I’d be willing to reduce my hourly rate when working for a non-profit with a good cause, but I’m most likely not going to do work for free.

The software is free, but the time spent working on it costs money

A nuance that’s perhaps lost on many people, is that I have often worked on converse.js for money. There was a rather long “bootstrapping” phase in the beginning where the project wasn’t good enough for anyone to actually use or pay money for further development, but after the project stabilized I started getting small paying gigs of custom development on converse.js.

In all cases I made it clear that the Mozilla Public License forces me to open source any changes I made to the covered files, and therefore the work I did for these paying customers (bless their hearts) was open sourced as well.

The point is that while the software is free (as in beer and as in speech), the time spent working on it costs money.

Either someone else pays me to spend my time working on it, or I end up paying by doing something for free while I might be getting paid doing something else (opportunity costs) or by taking time away from other activities.

FOSS development costs money, either the developer is commissioned, or they pay for it themselves (perhaps unwittingly).

Doing work for free devalues it and takes the piss out of actual paying customers

The last point I’d like to make, is that by taking on these requests to do free work for commercial entities (and non-profits), I’m not only devaluing my work, but I’m also disincentivising paying customers (which includes non-profits).

After all, why would anyone pay me to do anything if I’m so eager to please that I’ll do it all for free?

The only reason I could see to do that, is to get that mythical “exposure” that’s often also sold to web and graphic designers.

Image by “The Oatmeal”

⁵

So what do you do if you need work done and can’t pay for it?

Free and open source software is a beautiful, world-changing and paradigm shifting idea. However, software developers, like all people, need to be paid for their work, also when they work on FOSS.

If you can’t pay for software development, then you can still try to incentivise FOSS developers in other ways, but be aware that it’ll be more difficult.

One important lesson that I’m glad I learned early in life, is that when you’re asking someone to do something for you, then you need to explain to them why it’s in their best interests to do so.

People inherently look out for themselves. It’s perfectly natural and doesn’t necessarily mean they’re selfish to the point of being anti-social, it just means that they need to take care of themselves and that they can’t expect other people to do it for them or to even have their best interests at heart.

So when desiring something from someone, such as their help, the best approach is to explain to them what’s in it for them

For example, if you want a friend to help you out with something, let’s say to join a beach cleanup project, you don’t tell them why it’ll be good for you, you explain to them that it’ll be an opportunity to chat, to meet new people, to go for a swim and to have the enjoyment of a clean unspoiled beach.

This is simple stuff, but many people apparently don’t know this.

So if you want someone to help you with a software project, explain to them why it would be in their best interest. If you can’t find a reason why, then perhaps it’s actually not in their best interest and you need to create an incentive for them.

Money works pretty well as an incentive, but there are other ways as well. One sure-fire way to build up goodwill and gratitude (that might translate into more help and assistance) is to contribute. If you can’t write code, fix typos in the docs, evangelize the project or contribute in other areas which you have some expertise, like translations, design, UX, helpful feedback etc.

FOSS development is a community effort and a team sport. There’ll always be people who try to take more than they give, but on average, humans are matchers. When given something, they want to reciprocate and give back. Keep that in mind when you’re trying to get something for nothing.

References

Wikipedia article on Predictable Irrational ↩
Wikipedia article on Predictable Irrational ↩
Sung to the tune of Freedom isn’t free ↩
Sung to the tune of “Money for nothing” by Dire Straits.↩
From The Oatmeal ↩

Converting Youtube videos into mp3

Sun, 29 Nov 2015 00:00:00 GMT

If you’re using Linux or a Mac, then there are two neat commandline tools which you can use to download and convert Youtube videos into mp3 files. They are Youtube-dl and FFmpeg. Installation

Installing youtube-dl

The download and install instructions for youtube-dl are here.

As a quick note, on Mac you can run:

brew install youtube-dl

And on GNU/Linux:

sudo curl <https://yt-dl.org/downloads/2015.11.27.1/youtube-dl> -o
/usr/local/bin/youtube-dl sudo chmod a+rx /usr/local/bin/youtube-dl

For NixOS fans, it’s available via nix-env.

Installing FFmpeg

Many Linux distros come with FFmpeg installed I believe.

If not, you can use your distros package manager. For example, on Ubuntu:

sudo apt-get install ffmpeg

Note: In some older versions of Ubuntu (and probably other distros), you should use avconv instead of ffmpeg. AVConv was simply a fork of FFmpeg and the two projects have subsequently been merged back into FFmpeg again.

On Mac you can use homebrew:

brew install ffmpeg

Downloading the video

To download a Youtube video, you simply run the following line (making sure to replace the URL with the one of the video you want to download):

youtube-dl <https://www.youtube.com/watch?v=DLzxrzFCyOs>

Converting video to mp3

To now convert the mp4 video to mp3, you can use FFmpeg (comes standard with most Linux distros I believe).

ffmpeg -i RickAstley-NeverGonnaGiveYouUp\[HQ\]-DLzxrzFCyOs.mp4
RickAstley-NeverGonnaGiveYouUp.mp3

A common grammatical mistake that programmers make

Fri, 06 Feb 2015 00:00:00 GMT

I recently read a fascinating profile about a guy who made about 50 000 edits to Wikipedia to correct a relatively obscure grammatical mistake. The mistake of using “comprised of” when one should be using “composed of” or “consists of”. He is the ultimate wikignome. A name given to people who spend most of their time on the site making small corrections and fixes.

This reminded me of a grammatical mistake that programmers frequently make. I notice it in source code, on IRC, in mailing lists and in documentation.

Here are a few examples from actual code, which I found by using “grep”:

def cleanupInstance(self):

def teardownSite(self):

def setup_test_class(self):

case "default":
    // setup event handlers

What is the grammatical mistake in all these examples?

The compound nouns setup, teardown and cleanup are confused with the verbs set up, tear down and clean up.

In the above examples, they are used as verbs but spelled as nouns.

I ask you to set something up for me. I don’t ask you to setup something for me.

EDIT: Some readers misunderstood the point and thought I implied that there must always be a clause between the two words in the verb. This is not true. The point is that they are written separately.

For Example: “Please set up the server for me:.

However, when I want to refer to the task you did (the noun), I refer to the setup.

When used as an adjective, setup also stays one word:

Go to the setup menu. [adjective]
The server's setup is not yet done. [noun]

This mistake happens mostly with setup, however here are similar words which also often get used incorrectly:

turnover:

Turn over the documents.
Our company increased our yearly turnover.

handover:

Hand yourself over to the police.
The hostage handover went smoothly.

cleanup:

Clean up your room.
The cleanup took hours.

teardown:

Tear down this wall!
This test needs a teardown method.

UPDATE: Here are some more:

Callback <=> Call back
Checkout <=> Check out
Layout <=> Lay out
Login <=> Log in
Logout <=> Log out
Lookup <=> Look up
Plugin <=> Plug in
Popup <=> Pop up
Runaround <=> Run around
Shutdown <=> Shut down
Showtime <=> Show time
Startup <=> Start up
Takeoff <=> Take off
Workout <=> Work out
Writedown <=> Write down

Another interesting case is “maybe”. It can be used as an adverb (one word), noun (one word) and verb (two words, a modal verb “may” and the main verb “be”):

At the moment it's definitely still just a maybe. (noun)
I'll maybe come over tonight. (adverb)
He may be waiting for us. (verb)

What about something like download?

We don’t say load it down. We say download it.

That’s different. In the first category of compound nouns (setup, teardown, handover), the part of the word that denotes the act is first, followed by an adverb such as up, down, over or under:

setup = set [verb] + up [adverb]

In the second category, the acting word is second (load, grade, take, throw) and the first word denotes position:

download = down [preposition] + [load] verb or noun

Other examples of the second category are:

upgrade
downgrade
undertake
overthrow
understand
overcome

These do not get split up.

I hope I made things clear and that programmers will now “undertake” to “clean up” their code by removing these grammatical mistakes. ;)

EDIT: I've thought about this some more, especially in light of some of the comments provided by readers. I can see how using the word "setup" as a type of shorthand for "set up", to avoid unnecessary underscores or capitals, can be useful and valid. If you undersand the distinction and prefer writing it as one word, then go for it. I think however what's important is for people to at least understand that it's an exception and not the general rule outside of computer science.

Sprint Report: Merging Mockup and Patternslib

Sun, 25 Jan 2015 00:00:00 GMT

This is a report on what I did at the Plone.

Alpine City Sprint organized by Jens Klein and Christina Baumgartner in Innsbruck between 21 and 26 January 2015.

Firstly, I want to thank to them for organizing and hosting a great sprint and for being such friendly and generous hosts. My goal for this sprint was to work on merging the Patternslib and Mockup Javascript frameworks and I’m happy to report that very good progress was made.

Long story short, the merge was successful and it’s now possible to use patterns written for either project within a single framework. Before I expand on how this was achieved during this sprint, I’ll first provide some necessary background information to place things into context and to explain why this work was necessary.

What is Patternslib?

Patternslib’s goal is to allow website designers to build rich interactive prototypes without having to write any Javascript. Instead the designer simply writes normal HTML and then adds dynamism and interactivity by adding special HTML classes and data-attributes to selected elements.

For example, here is a Pattern which injects a list of recent blog posts into a element in this page:

The declarative markup for this pattern looks as follows:

<section id="alpine-blog-injected">
    <a href="#portlet-recent-blogs"
       class="pat-inject" data-pat-inject="target: #alpine-blog-injected">
        Click here to show recent blog posts
    </a>
</section>

The dynamic behavior comes from the fact that I have the Patternslib Javascript library loaded in this page.

On pageload, Patternslib scans the DOM looking for elements which declare any of the already registered patterns. If it finds such an element, it invokes a Javacript module associated with that pattern.

In the above example, the <a> element declares that it wants the Inject pattern applied to it, by virtue of it having the pat-inject HTML class.

Patterns are configured with specific HTML5 data properties. In the example above, it is the data-pat-inject property which specifies that the target for injection is the #alpine-blog-injected element in the current page and the content being injected is specified by the href attribute of the <a> element. In this case, the href points to an anchor inside the current page, but it might just as well be a link to another page which will then get injected via Ajax.

More information on how to configure pat-inject can be found on the patternslib website.

Each pattern has a corresponding Javascript module which is often just a wrapper around an existing Javascript library (such as Parsley, Select2 or Pickadate) that properly defines the configuration options for the pattern and properly invokes or applies the 3rd party library.

So, in order to do all this, Patternslib can be broken down into the folowing components:

A registry which lists all the available patterns.
A scanner which scans the DOM to identify declared patterns.
A parser which parses matched DOM elements for configuration settings.
The individual Javascript modules which implement the different patterns.

What is Mockup?

Mockup was inspired by, and originally based upon, Patternslib and was meant to bring the power of declarative interaction patterns to Plone.

When Mockup started, Patternslib was undergoing significant refactoring and development and it was decided that Mockup should fork and go its own way.

What this means is that the 4 different components mentioned above, were all changed and due to these changes the Mockup project diverged from Patternslib and started developing in a similar but different direction.

So what’s the problem?

While Mockup was being developed for the upcoming Plone 5 release, we at Syslab continued using and improving Patternslib in our projects.

Syslab built an intranet for the Star Alliance, which was based on a prototype design by Cornelis Kolbach, the conceptual creator of Patternslib. This design became the inspiration and blueprint for the Plone Intranet Consortium (PIC), which consists of 12 companies working together in a consortium to build an intranet solution on top of Plone.

So, the PIC are building a product using Patternslib, while Plone 5 itself is being built with Mockup, an incompatible fork of Patternslib.

This was clearly a bad situation because we now had:

Two incompatible Javascript frameworks being used with Plone.

Not only were the two frameworks incompatible in the sense that patterns written for the one don’t work on the other, but they could also not be used on the same page since the two implementations would compete with one another in invoking Javascript to act upon the same DOM elements.
Duplication of effort

The same or similar patterns were being developed for both frameworks, and when one framework had a pattern which the other wanted, it could only be used after being modified such that it couldn’t be used in the original framework anymore.
A splitting of the available workforce.

Developers were either working on Mockup or Patternslib, but almost never on both, which meant that the expertise and experience of developers wasn’t being shared between the two projects.

How could this be fixed?

To solve the 3 main problems mentioned above, we needed to merge the common elements of Mockup (specifically the registry, scanner and parser) back into Patternslib.

This will allow developers from both projects to work on the same codebase and enable us to use patterns from both projects together.

At the Alpine City Sprint in Innsbruck, I worked on achieving these goals.

Changes brought in by Mockup

After the fork, Mockup introduced various changes and features which set it apart from Patternslib.

In order to merge Mockup back into Patternslib, I studied these changes and with the help of others came up with strategies on what needed to be done.

Here are some differences and what was done about them:

Mockup allows patterns to also be configured via JSON, whereas Patternslib used a keyword:argument; format

A week before the sprint I added JSON parsing ability to the Patternslib parser, thereby resolving this difference.
“Leaves first” parsing versus “root first” parsing

Mockup parses the DOM from the outside in (“root first”), while Patternslib parses the DOM from the inside out (“leaves first”).

According to Rok Garbas, the creator of Mockup, the outside-in parsing was done because it reduced complexity in the scanner and the individual patterns.

Wichert Akkerman who originally wrote the Patternslib scanner however provided a good reason why he chose “leaves first” DOM scanning:

If I remember correctly there are several patterns that rely on any changes in child nodes to have already been made.This is true for several reasons: 1) a pattern may want to attach event handlers to child nodes, which will break of those child nodes are later replaced, and 2) child nodes changing size might impact any measurements made earlier.

Indeed, while refactoring the Mockup code during the merge, I ran into just such a case where a pattern couldn’t yet be initialized because patterns inside it weren’t yet initialized. By turning around the order of DOM scanning, this problem was resolved and the code in that pattern could be simplified.

So, now after the merge, scanning is “leaves-first” for Mockup patterns as well.
Mockup patterns are classes extended from a Base object

Patternslib patterns on the other hand are simple JavaScript objects without constructors.

The Patternslib patterns are conceptually very simple and more explicit.

However, what I like about the Mockup approach is that you have a separate instance with its own private closure for each DOM element for which it is invoked.

After merging, we now effectively have two different methods for writing patterns for Patternslib. The original “vanilla” way, and the Mockup way.
The Mockup parser was completely rewritten

The Mockup parser looks nothing like the Patternslib one and also supports one less configuration syntax (the so-called “shorthand” notation).

This was one piece of code which could not be merged within the time available at the Alpine City Sprint.

So currently we still have two different argument parsers. The Patternslib parser needs to be configured much more expicitly, while the Mockup one is more implicit and makes more assumptions.

Merging these two parsers will probably have to be done at some future sprint.

There are some other more minor differences, such as that every Mockup pattern is automatically registered as a jQuery plugin, but merging these was comparatively easier and I’ll won’t go into further detail on them.

What I did during the Alpine Sprint

So, in summary:

I refactored the Patternslib registry, scanner and some core utils to let them handle Mockup patterns as well. Luckily the eventual patch for this was quite small and readable.
I changed the Mockup Base pattern so that patterns derived from it now rely on the registry and scanner from Patternslib.
I fixed lots of tests and also wrote some new tests.

This whole task would have been much more difficult and error prone if either Patternslib or Mockup had fewer tests. The Mockup team deserves praise for taking testing very seriously and this allowed me to refactor and merge with much more confidence.

What else is there to still be done?

Being able to use patterns from both projects and merging most of the forked code was a big win, but there are still various things that need to be done to make the merge complete and viable.

I’ve ranked them from what I think is most important to least important.

1. Update the documentation

Currently documentation is scattered and silo’d in various places (the Patternslib website, the Plone Intranet developer docs and the Mockup developer docs).

The Mockup docs are now out of date ater this merge and need to be brought up to date on these recent changes.

The Patternslib docs are less of a problem because they don’t have to deal with Mockup (which can now be seen as an enhancement suite for it), but they can definitely still be improved upon, specifically with an eye on Mockup developers who will start relying on them.

The Plone Intranet consortium also has a useful walkthrough explaining how to create a Patternslib pattern from scratch..

2. Devise a way to also use the Patternslib parser for Mockup patterns

As mentioned, the Mockup patterns still use their own argument parser.

Letting them use the Patternslib’s parser will either require extending the Mockup Base pattern to configure the Patternslib parser on a pattern’s behalf or instead doing it explicitly in each of the individual patterns.

3. Decide on coding and configuration standards

Unfortunately the coding standards between the two projects differ significantly.

The Mockup developers use camelCase as declarative argument names while Patternslib uses dash-separated names.
The Mockup source code uses 2 spaces for indentation, Patternslib uses 4 spaces.

4. Remove unnecessary, duplicated patterns

Both projects have patterns for modals, injection, select2 and file upload. These should be merged to get rid of duplication.

5. Move generic patterns (e.g. pat-pickadate, pat-moment etc.) out of Mockup

Mockup has some generic patterns which might also be useful as Patternslib core patterns, in which case they should ideally be moved there.

Conclusion

The sprint was a big success and I’m very happy that all the work has already been merged into the master branches of the matternslib, mockup and mockup-core repositories.

Core code from Mockup is now succesfully merged back into Patternslib and we can use patterns from both projects in the same site.

I had a lot of fun working with talented and motivated software developers and had a lot of opportunities to improve my German.

Thanks again Jens and Christine for organising a great sprint and I look forward to doing such a sprint again!

The Solution to Jaron Lanier's Siren Servers

Tue, 06 Jan 2015 00:00:00 GMT

In his book [Jaron Lanier](https://en.wikipedia.org/ wiki/Jaron_Lanier) claims that the information on the web is undervalued and underpriced.

According to him this is hollowing out the middle class because people aren’t being sufficiently valued for their information-producing work, while a new rich elite is making vast sums of money by collecting and analysing this cheap information.

In many cases the information being collected and analysed is given freely to companies by the users themselves in return for email, calendaring and social networking features.

This results in information assymmetry (e.g. who knows what about whom) and therefore a large disparity in power and wealth. Google knows everything about you and yet you know nothing of value about Google.

Siren Servers

These online services are hosted on servers, which Lanier has termed Siren Servers¹, after the Sirens of Greek Mythology. The Sirens are beautiful but dangerous creatures who lure nearby sailors with their enchanting music and voices, thereby letting them shipwreck on the rocky coast of their island.

These Siren Servers lure us into opening up our lives, so that they might freely gather data from us, on a truly massive scale, which they then analyse and profit from.

The results of this analysis is ”kept secret and used to manipulate the rest of the world to advantage.”

So much value is extracted from this data, that these Siren Servers are valued in the billions of dollars.

Lanier claims that the users of these servers are shortchanged, in that they don’t get their data’s worth in return. Not to mention the broader societal damage wrought by the surveillance-as-business-model engendered by this approach as well as the decimation of the middle class as software takes over many industries.

Siren Servers reduce risk for themselves, but in the process increase the risk for all the other smaller players. Uber’s business model for example is all about shifting risks onto its drivers. They would not be profitable if they had to carry the costs of commercial insurance, licensing, vehicle maintenance and oversight of drivers.

”The total amount of risk in the market as a whole stays the same, perhaps, but it’s not distributed evenly. Instead the smaller players take on more risk while the player with the biggest computer takes on less.”

Lanier’s solution to the emergence of Siren Servers is to “properly account” for the wealth created by people online. In other words, people need to be paid for what they do online, no matter how small and seemingly insignificant.

”If information age accounting were complete and honest, as much information as possible would be valued in economic terms. If, however “raw” information, or information that hasn’t yet been routed by those who run the most central computers, isn’t valued, then a massive disenfranchisement will take place.”

The decimation of the middle class

Lanier says this process started in the creative industry, with musicians and photographers impacted quite early on, and that it’s now spreading to other occupations, such as travel agents, estate agents and journalists.

Lanier writes:

”Making information free is survivable so long as only limited numbers of people are disenfranchised. As much as it pains me to say so, we can survive if we only destroy the middle classes of musicians, journalists and photographers. What is not survivable is the additional destruction of the middle classes in transportation, manufacturing, energy, office work, education and health care. And all that destruction will come surely if the dominant idea of an information economy isn’t improved.”

In other words, until we reconfigure the economic system to value raw data by compensating the people that generate it, we will have massive inequality.

Not everybody is of course as pessimistic or draws these same conclusions. Steve Albini, producer of Nirvana and Pixies albums and himself a touring musician, says in this keynote that the Internet has solved many problems for musicians and audiences alike and that things are actually better now than before.

Finance got networked in the wrong way

As an example of the destruction being wrought by the Siren Servers, Lanier asserts that the Great Recession was in part caused by their existence in the financial sector.

”Consider the expansion of the financial sector prior to the Great Recession. It’s not as if that sector was accomplishing any more than it ever had. If it’s product is to manage risk, it clearly did a terrible job. It expanded purely because of its top positions on networks. Moral hazard has never met a more efficient amplifier than a digital network. The more influential digital networks become, the more potential moral hazard we’ll see, unless we change the architecture.”

It’s clear that Lanier believes that this problem can be solved by reconfiguring and improving the architecture of our digital networks.

Siren Servers and the rise of surveillance as a business model

As has become increasingly clear in the past few years, the dominant business model in Silicon Valley is one based upon surveillance.

Surveillance is the mechanism by which the Siren Servers gather much of the data they analyse and profit from.

According to Lanier, many Silicon Valley techno-libertarians handwave any concerns about corporate surveillance.

”Surveillance by the technical few on the less technical many can be tolerated for now because of hopes for an endgame on which everything will become transparent to everyone. Network entrepeneurs and cyber-activists alike seem to imagine that today’s elite network servers in positions of information supremacy will eventually become eternally benign or just dissolve.”

However, what good reason is there for owners of these immensely powerful and valuable Siren Servers to one day willingly become transparent and open up in order for this power imbalance to be resolved?

On the supposed emergence of some kind of sharing/socialist utopia arising out of the aggressive right-wing libertarian practices of Silicon Valley, Lanier mocks:

”Free Google tools and free Twitter are leading to a world where everything is free because people share, but isn’t it great that we can corner billions of dollars by gathering data no on else has?” If everything will be free, why are we trying to corner anything? Are our fortunes only temporary? Will they become moot when we’re done?”

These ”network servers in positions of information supremacy” hide their code and algorithms behind patents, IP-laws and copyright.

A whole legal facade has been erected precisely to prevent the hopeful scenario of “elite network servers” dissolving into an utopian abundance.

Lanier’s proposed solutions

Cyber-keynesianism

Lanier describes his proposed solution as a sort of Cyber-Keynesianism. Based upon the idea from the economist J.M. Keynes that “stimulus” might be able to kick an economy out of a rut.

Lanier provides a qualitative explanation of the stimulus theory which I haven’t heard before.

On a multi-dimensional mathematical landscape (as could be plotted based on the parameters of an economy), there are peaks and valleys. Let’s assume that peaks are “optimal” economical configurations, where the greater good is best served, and valleys are horror scenarios (depressions, recession, collapse etc.).

Not all peaks are of equal hight, and when you are on a peak, you are usually surrounded by valleys. There might be a higher peak (i.e. better configuration) available, but it’s not clear how to get there, since moving there might mean moving through valleys.

The idea behind stimulus is to provide the impetus, a kick if you will, to make this transition to a higher peak.

Technical Solution

Laniers solution on a technical level is to use a network with bidirectional links, instead of unidirectional links as we currently have with the web. The architecture of this network is inspired by Project Xanadu already founded by Ted Nelson in 1960.

Whenever someone publishes something on the web they would be informed if someone else links to their work, due to the bidirectionality of the links. This would supposedly put a stop to copyright infringement and also allow content creators to correctly identify and bill the consumers of their media.

Of course, this would only work if there were no anonymity on the web.

Lanier further proposes a single, public, digitally networked marketplace where every participant is tied to his personal real-life identity, and where anything anyone creates online has to be paid for.

My Criticism

I thoroughly enjoyed reading Lanier’s analyses, and his descriptions of the mechanisms behind Siren Servers in particular. This was for me the best part of the book.

However, he often hand-waves obvious counter arguments or advances his own arguments in a non-rigorous way, often relying on personal anecdotes.

Digital networks’ supposed role in hollowing out of the middle class

Considering his central thesis, that digital networking is hollowing out the middle class in developed societies, I’m not sure how much blame for that can be assigned to digital networking. Other plausible factors include demographic change, globalisation and the money printing policies of Central Banks which are fueling asset price bubbles while real wages remain stagnant, thereby making wage-earners poorer.

My expectation is that no particular factor can be singled out, and that the causes are multiple, varied and intertwined in complex and difficult to understand ways, precisely because the economy is a complex, chaotic system (in a mathematical sense).

Lanier fails to meaningfully address Free and Open Source software

Lanier only very superficially addresses the role of free and open source software (FOSS) and doesn’t even mention (let alone extensively cover or deconstruct) the ideology of Free Software at all.

As a quick recap to people who don’t know the difference between “Free Software” and “Open Source software”:

Both groups concern themselves with creating software where the source code is visible, as opposed to software where you cannot inspect the source code, and are therefore unable to modify or know what it is really doing.

“Free Software” (as in freedom, not price) however stresses a moral and ethical dimension of software development and usage. It basically boils down to the fact that non-free software creates a dangerous and immoral power imbalance, slanted in favor of the creator of the non-free software and against the user of that software.²

Open Source software on the other hand ignores the moral dimension completely and takes a much more expedient approach. It simply asserts that developing software in the “open”, in other words, with the source exposed, is a superior form of software development which will result in better code.

In the index of this book, there is no entry for “Free Software” or “Open Source Software”. There is one entry “open source applications”.

Lanier doesn’t even attempt to address the viewpoint of the Free Software movement, which has been warning about these issues (structural power imbalance, Siren Servers, ubiquitous online surveillance) for decades. This is in my opinion a glaring omission from the book.

Lanier might have coined the catchy phrase Siren Servers, but his analysis of their role and negative effects on users and society might have well come from someone from the Free Software movement. See for example the article Who does that server really serve?.

As the saying goes, ”There is no cloud, only other people’s computers”³

Lanier of course, comes to a completely different conclusion on what needs to be done to resolve the situation, and appears to hold proponents of Free and Open Source Software (dimissively calling them “racals”, “openness crusaders” and Pirate Party types), not only in contempt, but partly responsible for the emergence of Siren Servers and rise of surveillance as business model.

It is clear that if Lanier were to meaningfully address all ideological opponents of his vision for the future, then he would have had to address the ideas behind Free Software in a methodical and rigorous manner and not with dismissive hand-waving.

The fact of the matter is that Free Software does provide an alternative solution to the current mess of online surveillance, increasing risk and fragilization, extreme power imbalances and centralization into Siren Servers.

The solution is to ensure that the software you use is free. If software being used by a service is free and issued under a license that ensures its freeness](https://www.gnu.org/licenses/why-affero-gpl.html), then that server will not have the ability to eventually become a Siren Server.

The reason for this is that users will know exactly what the software does with their data and will have the ability to switch to a different provider (while taking all their data with them), or to host it themselves, at a moment’s notice.

This means that the power balance is no longer slanted in favor of the service provider, and users are much more in control of their data and content.

Lanier’s hypocrisy

Throughout the book, Lanier criticises in particular Google and Facebook for their creepy surveillance business models and their usage of Siren Servers.

Lanier himself works for Microsoft, a company that is feverishly competing with Google and Facebook in surveilling users and in establishing their own Siren Servers.

However, Lanier of course does not at all address this conflict of interest or that fact that his employer is just as complicit as the companies he calls out. This apparent hypocrisy severely detracts from the integrity of his book.

Lanier’s superficial and misconstrued criticisms of “openness”

Lanier’s refusal to meaningfully engage with the different ideas and philosophies behind what he calls “openness” is for me one of the most disappointing aspects of the book.

Wikipedia

He deceptively equates Wikipedia and Facebook, saying that both are a culmination of the “information should be free” meme and that both contribute to the devaluation of user created content. In a previous book, he even calls Wikipedia “Digital Maoism”.

Wikipedia is concerned with creating a digital Commons. By voluntarily contributing to the commons (as Wikipedia contributors do), you are not devaluing your work, you are making it priceless, in both the literal and figurative senses of the word.

Conversely, the information on Facebook is not free and is not part of a commons.

I find it good to know that there are some things in this world on which one cannot put a price tag, despite the efforts of people like Lanier.

As the saying goes. ”A cynic is someone who knows the price of everything and the value of nothing.”

Wikileaks

Lanier incorrectly claims that Wikileaks is concerned with abolishing all privacy. Anyone who has made even a half-arsed effort to understand what Wikileaks is about would know that they stand for transparency of the powerful and their institutions, in order to keep them democratically accountable, and that they don’t advocate the elimination of all personal privacy.

Of course, this is a common mischaracterization of this organization, often done to intentionally discredit them. Either Lanier is also trying to discredit them or he couldn’t be bothered to properly research their stated goals.

Linux

Lanier claims: ”A linux always begets a Google” as if it’s is somehow responsible for the emergence of Siren Servers. This doesn’t explain Siren Servers built upon proprietary software, such as StackOverflow which uses Microsoft.

Inasmuch as one could argue that Siren Servers are the result of open source software (note the distinction) it is exactly due to open source’s gutting of the moral aspect out of free software.

Lanier wants to remove anonymity on the web

Lanier criticises tech companies for their creepy surveillance business models, and yet his proposal for “fixing” the internet would eradicate all privacy online and have all human interactions facilitated via a single, public market place. It would be a surveillance and privacy nightmare.

Concerning anonymity on the web, Lanier says this is the result of the “pot-smoking liberals” and “paranoid conservatives” who originally built the system and who thought anonymity was “cool”. Such a perfect example of his dismissive, non-rigorous and arbitrary reductionism which completely fails to meaningfully confront the potential benefits or dangers of anonymity on the web.

Lanier completely fails to address the fact that anonymity is often the only defence people may have against corrupt and malevolent state power. Whether this applies to activists during revolutions or to conscientous whistleblowers, doing away with privacy would be throwing away one of the few ways of ensuring free speech online.

Monetizing all human interactions debases them

By wanting to turn the web into a giant marketplace where everyone operates with an exposed identity and every digital creation, no matter how trivial, needs to be paid for, Lanier wants to further the already ongoing monetization of the public commons. As Charles Eisenstein writes sacred economics, monetizing all human interactions actually debases them.

Lanier calls his approach “humanitarian”; it is anything but. Instead, by monetizing and thereby commodifying every online human interaction, whether it’s a joke, a compliment or a statement of support, these interactions are stripped of their inherent sacred quality of humanity.

If you know that someone who sends you a message of support or love will receive a micropayment for that message, could the original intent of the message not be called into doubt?

My proposed solutions

It’s about freedom, not cost

Lanier’s proposed solution ultimately has to do with costs and trying to find a way to get people paid for anything they write and then publish online, no matter how trivial.

However, in so doing he comes up with a proposed solution that does away with all privacy and anonymity.

His proposal of using a network with bidirectional links might threaten the Siren Server status of services such as Facebook and Google. However, it’s not clear how such a network will solve the problem of Uber or AirBNB becoming Siren Servers, thereby pushing out the middle class jobs of taxi drivers and guest house operators.

It’s not only about the network architecture, as Lanier suggests. What’s more important is Software and Computing Freedom.

If you don’t have the freedom to study, use and modify the software that you use, then you will be systematically exploited and preyed upon, no matter what the network architecture looks like.

Free software specifically, is defined by these four freedoms:

The freedom to run the program as you wish, for any purpose
The freedom to read the source code and to modify it as you wish.
The freedom to redistribute original copies so you can help your neighbor.
The freedom to distribute your own modified copies to others.

Being able to read the source code is essential for software to be free. An organisation is however only compelled to share their source code if they intend to distribute compiled copies as well. If the software is used privately, i.e. in-house, then they are under no such obligation.

Open Source on the other hand, also gives the right to read the source code, but many open source licenses do not ensure the last two freedoms, and therefore offer weaker protection.

There is however a large overlap between the two groups. Most open source software is also free software, and vice versa.

Free software coupled with free and open protocols means that users are not beholden to specific service providers. It’s exactly this aspect of it which is why the dominant tech companies avoid free software like the plague and instead embrace the watered down notion of “open source”. Their goal is to maintain strict vendor lock-in, making it difficult for users to switch to other service providers by holding their data and online personas ransom. Something which they could not do if their software respected the above four freedoms.

Who pays for fee software?

When discussing free software, the question often arises how one should make money with software if you cannot enforce artificial scarcity, vendor lock-in, holding user’s data ransom or spying on them.

This is perhaps a topic for another blog post. In short, it definitely is possible to make money writing freedom respecting software and the more people insist on using only free software, the more economic opportunities will arise.

Software is better seen as a living, changing and evolving ecosystem, than as a static object. It needs constant care and attention, to keep it relevant, accurate, applicable and functioning. This care and attention is a service which can and should be paid for.

Who pays for this service? The users do. Whether they are governments, universities, corporations or private individuals.

How do they pay for it? Organisations sign service and support contracts. They also commission the writing of new features or entirely new applications.

Private individuals can pay for free software through crowdfunding, bounties, and donations.⁴

Public institutions and foundations can issue grants or fund the development of free software for usage in government and publically owned enterprises.

If all governments decided to only fund and use free software, it would create a massive amount of investment in the sector. Additionally, money spent on developing free software can be invested in a country’s own programmer workforce, instead of it being sent overseas to oversized foreign corporations.

However, those companies that currently spy on us, hold our data ransom and restrict our digital freedoms, won’t change by themselves. It’s up to the users to take matters into their own hands, to demand freedom and to support those companies, organizations and people who develop free software.

Decentralization

The problems Lanier identify are largely caused by data-assymetry (who has data on whom) and the resulting siren servers.

An architectural approach to solving this problem, which complements free software, is decentralization.

Decentralization refers to a network structure (called topology) where each node in the network can connect directly with any other node, without the intervention of a middle man (such as Facebook or Google).

In the network topologies pictured here, the fully connected topology is the most decentralized, while the star network is the most centralized.

Creating large decentralized networks is more difficult than creating centralized ones, which is why the web has come to rely on centralized Siren Servers to such a large degree.

However, much exciting work is being done on decentralizing the web and putting people back in control.

Consider Finance, and Lanier’s assertion that Siren Servers contributed to the financial crisis preceding the Great Recession.

It’s more difficult to have a financial Siren Server when financial data is decentralized and therefore effectively available to everyone.

This is already the case with decentralized cryptocurrencies such as Bitcoin. The bitcoin protocol is an open and free protocol, the original bitcoin wallet is free software and the transaction information contained in the blockchain is available to all.

An improvement upon Bitcoin would be to allow truly anonymous transactions, and this appears to be on the horizon.

Bitcoin and cryptocurrencies are however only one application which can be built on top of the revolutionary distributed consensus blockchain technology upon which it rests.

There’s already work on distributed cloud hosting, distributed microblogging and a myriad of other services running in a decentralized (non-Siren Server) manner.

Federation

An older idea than blockchain-based decentralization is federation. Think of the way in which you can send an email from a Gmail account to someone with a Yahoo account, or to someone who hosts their own mail server.

Why can we send email to people with different service providers, but we cannot send chat messages from a Google account to an iMessage account, or between Whatsapp and Yahoo messenger?

This ability of email is the result of something called federation. The ability of servers from different providers and with different implementations, to communicate and relay messages between one another.

Email is federated but facebook messages are not, because you cannot send a facebook message to someone outside of facebook.

There is no technical reason why instant messaging cannot be federated. In fact, XMPP, the protocol on which many of the messaging services are originally based, allows federation.

The reason we don’t have federation in messaging is because it’s not in the interest of Siren Servers. It would actively undermine their dominance.

If any of the big email providers, Google, Yahoo, Microsoft or Apple, could get away with only allowing emails within their own system, they would have done it by now. The proof is in the fact that they actively prevent federation in their messaging apps. The reason they cannot do it with email is because federated email achieved critical mass before any single company could kill it off.

Unfortunately, this was not the case with instant messaging. Therefore, if you actively want to work against Siren Servers, you should only use instant messaging servers which support federation, such as XMPP.

You can sign up for a free XMPP chat account here. And if you’d like to chat with me, add me as a contact:

Conclusion

As you can see, I believe in a completely different remedy than the one which Lanier suggests. One that’s based on a methodology, legal framework and software which is already being used instead of something which Wired magazine in 1995 called the longest-running vaporware project in the history of computing.

I thoroughly enjoyed Lanier’s analyses and his unique perspective on things which I think made the book worth reading.

However, his non-rigorous approach, his dismissiveness or ignorance of alternative narratives and approaches to resolving the issues he mentions, as well as his hypocritical omission of his own employer’s complicity, left me disappointed and uninspired.

Footnotes:

Introduction to Free Software and the Liberation of Cyberspace

Bruce Sterling calls them “The Stacks”, vertically integrated social media. See Bruce Sterling At SXSW 2013: The Best Quotes.↩
If you’re interested in more detail about Free Software, watch this presentation by Richard Stallman:↩
The earliest reference to this that I could find, is this article ↩
See Kickstarter, IndieGogo, Startjoin, Patreon, Bountysource, Gratipay and Snowdrift.↩

Book review: Zen and the art of motorcycle maintenance

Sat, 03 Jan 2015 00:00:00 GMT

This book, a best-seller written in 1974, is regarded as a classic and one of the most important and influential books of the last century.

Superficially, it’s about the protagonist and his troubled son who go on a motorcycle trip across America.

During their rides he expounds philosophically on the two different ways in which people subjectively perceive technology and then attempts to describe a way to reconcile this dualism.

These two approaches to technology the author calls classic and romantic modes of thought. The classic approach is technical and analytic. It is concerned with underlying form and purpose.

In the language of the 1960s and ’70s, the classic approach is “square”.

The romantic approach is intuitive, idealistic and symbolic. It is therefore “groovy” instead of “square”.

The book is in some way, an attempt to reconcile the schism that appeared in American society in the 1960s with the emergence of the romantic hippie movement, which rejected the dominant classic mainstream narrative.

However, this dichotomy extends further than the social and political milieu of 1960s and 1970s America and the author traces its origins all the way back to ancient Greece and the analytical approaches of Socrates, Plato and Aristotle. One can also see this dichotomy in software development; between the classic, analytical developers or engineers and the romantic designers.

Differing approaches to esthetics and beauty

The author mentions a perceived death force, apparently inherent in technology, but which is actually something which emerges when one approaches technology from a strictly romantic viewpoint without any appreciation for the classic understanding of underlying form and function. Out of a classic understanding of technology, a different esthetic value arises, one that appreciates the beauty of the optimal combination of form and function.

Programmers talk about code that is elegant. Not elegant in a visual or artistic sense, but elegant in the sense that is it is clear, succinct and functional to a degree that it cannot be improved upon. This is the classical esthetic ideal.

To a romantic person, a complex schematic might look complicated and unappealing, while to the classic, analytical minded person it might be a thing of beauty.

The author reconciles the romantic and classic modes of thought by pointing towards unanalyzed, non-dualistic and “ultimate” reality.

The author asserts that non-dualistic direct experience is the source of Quality in thoughts, acts and creations. Quality which encompasses both the romantic and classic approaches.

The book ultimately reveals that the protagonist himself has a dualistic split within his own mind, which he needs to reconcile.

Another layer to the story

Besides the superficial story of the motorcycle trip, the book is also about a man who experienced non-dualistic, cosmic consciousness; pessimistically interpreted by his society as a pathology (i.e. mental illness or insanity), and therefore in need of “correction”.

This alterered state of consciousness, considered an illness by psychiatrists, is triggered by an analytical, philosophical exploration of the nature of Quality, during which this man, through analysis, scales the heights of abstraction which his mind can attain, all the way up to non-dualistic experience.

Through this, he realises that non-dualistic experience, that which we experience immediately before the rational mind starts analyzing and categorizing, cannot be defined. And yet, Western analytical thought is not able to meaningfully deal with something which cannot be defined.

Practical lessons on quality

Ultimately the author reasons that Quality, in word and deed arises out of a state of being in contact with direct, immediate experience while being completely immersed in what you are doing or working on.

This is the Zen of motorcycle maintenance.

People also talk about being “in the zone”, and in Psychology there is the notion of ”flow“.

In short, one should strive to experience and maintain inner peace while doing the work. The end result and end goal should also be inner peace. In so doing, the experience of doing the work will have quality and the work itself will have quality.

Conclusion

I loved this book. Since reading it, I have a completely new notion of quality, a new appreciation for the limits of analytics and reason and for the ultimately ineffable nature of direct experience.

The book itself is a work of quality, and the author states in an afterword that his motivation for writing the book was because he deemed it a higher quality endeavor than not writing it.

I thank him for that, and because of it, I am now inspired to maintain inner peace while I work, to strive towards inner peace as my ultimate goal, and to thereby achieve quality in what I do.

Antifragile Software Ecosystems

Tue, 14 Jan 2014 00:00:00 GMT

I recently finished reading the book Antifragile, by Nassim N. Taleb, in which he discusses the idea of antifragility, the property that allows something to ultimately improve and grow stronger when subjected to volatility and stressors.

I highly recommend Taleb’s books, they have a tendency to throw conventional wisdom on its head and make you think differently of our world, in which randomness, volatility, opacity and non-linearity takes precedence.

As a software developer, I kept wondering how the insights gained from this book apply to the world of software. Especially given the fact that software is very fragile. The answer lies in the communities or ecosystems that form around software. Many of them are fragile, but some are antifragile.

If you’re also a software developer, you’ll want to be in the second category.

What is antifragile?

Taleb splits everything into three categories, fragile, robust and antifragile.

The initial chapters explain in detail why robust is not the opposite of fragile, and why the neologism “antifragile” is needed to describe the opposite of fragility.

If fragile systems are sensitive to stress and volatility, and need to be kept “safe” from harm, lest they suffer or break down, then antifragile systems are ones which ultimately benefit from randomness and stress.

In Greek mythological terms, one can think of Medusa, which grows another two snakes on her head for each one that’s cut off. That’s antifragile.

So fragile and antifragile can be considered opposites, e.g. positive and negative, while robust is simply neutral. It suffers little harm from volatility but also gets little to no benefit.

Evolution is antifragile

Some good examples of antifragile systems are found in nature and ecology. For evolution to work (i.e. for a species to adapt to its randomly changing environments), random mutations are required between individuals in that species.

A random mutation in a member of a species holds the possibility of making that member better adapted to its environment. By virtue of it being better adapted, that member will on average live longer and produce more offspring, which will be likely to have inherited the same advantageous mutation. The process continues, propagating the beneficial mutation throughout the species, in so doing, the species evolves and becomes more fit and adapted to its environment.

Nature is able to discard less well-adapted specimens and keep the better-adapted ones. An ability which Taleb refers to as optionality. It has the option to keep the “good” specimens and discard the “bad” ones.

The opposite of an option is an obligation. Nature is under no obligation to keep the poorly adapted members of a species, and if it were, it wouldn’t be antifragile anymore.

So, although individual members within a collection (like a species) are fragile to certain stressors, the overall effect on the collective level is antifragility.

Software is fragile

Unmaintained and unsupported software degrades when exposed to the stressors of time and use.

The more software you install on your computer, the higher the chance that your machine will eventually become slow and prone to crashes and bugs.

The word “bitrot” is sometimes used to refer to the gradual degradation of a software program over time and is related to the idea of entropy.

Large software systems that are composed of multiple interacting subsystems, are especially vulnerable to bitrot, due to the fact that a given subsystem cannot anticipate all the bugs or changes that might be introduced to other subsystems over time.

The more a piece of software is used, the more variance there is in how it’s being used. For example in the order, type and amount of inputs it receives as well as in the hardware its being installed on.

Lets assume that a very specific interaction with the software, which the programmer never foresaw, will cause the software program to crash and become permanently degraded.

Given enough time, someone will eventually interact with the software in this specific way, thereby permanently degrading the functionality of the software program (for example due to data corruption).

Are software businesses also fragile?

Businesses are generally fragile. They require a certain level of stability to operate and are sensitive to volatility (like regulatory changes, lawsuites or changing consumer tastes).

An antifragile software business however would be one that does well whenever some metric related to the software experiences increased volatility.

Remember:

Something that’s antifragile wants more volatility, because the benefit from increased volatility outweighs the harm.
Similarly, you are fragile to a specific source of volatility when the potential downsides outweigh any upsides.

So, let’s take “number of security flaws” as the metric.

The fortunes of a company that does security audits would improve every time there is a publicised security flaw in the software itself, or the software is compromized by an attacker.

They would therefore be antifragile to the “bad news” of exposed security flaws.

Perhaps with an upper bound. If there are too many security flaws, the reputation of the software itself might suffer irrevocably, and with it that of the company.

This illustrates the point that antifragile systems can handle volatility and stressors up to a point. Given enough volatility and stress, the entire planet can die out.

Conversely, a company whose business model depends on the software being stable, and on attacks not happening, would be fragile to the software having security flaws.

A community of fragile members can itself be antifragile

So, as mentioned earlier, an ecosystem can be antifragile, even if the individual units that make up that system are fragile.

It is however important that the units themselves are relatively redundant and that the fragility is not systemic inside the entire ecosystem.

To put it in more technical terms, units inside the system need to be “hot-swappable” and their interactions with one another should be “loosely coupled”. This is necessary to prevent complex and hard dependencies within the system, which in the case of breakdown of one unit would cause cascading failures and damage rippling through the entire system.

Antifragile software ecosystems

So, we now get to the crux of this article.

Software itself might be fragile, as well as businesses built around that software, but the larger ecosystem that encompasses the software and businesses, can be antifragile.

A software ecosystem that is distributed over many small and relatively self-contained businesses, each providing services around a common shared resource of Free and Open Source software (FOSS), coupled with a low barrier to entry for new businesses, is antifragile.

Conversely, a software ecosystem that is centralized around one large provider (often playing the role of gatekeeper or rent-seeker), artificially restricting access to the software, attempting to stifle any competition with its own services, is fragile and sensitive to large negative Black Swans (difficult to predict but extreme events).

I’ll elaborate on why this is by identifying the requirements for an antifragile software ecosystem.

Requirements for an antifragile software ecosystem

Distributization

Centralized systems provide a large single point of failure that can cause the collapse of the entire ecosystem. Distributed systems can take shocks much better and are able to route around failures and breakdowns.

Redundancy

When one unit breaks down, others need to be able to pick up the slack. For that to happen, they need to collectively be able to perform the functions of the defunct one. For example, if a software community relied solely on one unit in that community for supplying hosting services, then the entire community would break down if that one unit collapsed.

Optionality

The individual members in an antifragile ecosystem need to be able to modify and adapt.

For a software ecosystem, this means the ability to modify and adapt the software itself. Only through being able to modify the software in a distributed, bottom-up way, can the ecosystem withstand stressors in an antifragile way, by making use of optionality.

Optionality refers to having the option to take (or do) something or not. As mentioned earlier, it’s how evolution is able to “design” extremely sophisticated organisms, by keeping the “good” ones and discarding the bad ones, and it applies to software ecosystems as well.

For example, in the Plone CMS (content management sytem) community, add-on modules are developed in a redundant and distributed way. There are multiple blog add-ons, commenting add-ons, newsletter add-ons and so on. Through a process of optionality and selection, the good ones rise to the top while the weaker ones are eventually discarded.

Why the requirement for Free and Open Source software?

FOSS facilitates all three of the above requirements. Communities and ecosystems that develop around FOSS are by nature distributed, redundant and provide optionality.

Distributization and redundancy come from the lack of gatekeepers, “rights-holders” and rent-seekers. Anyone can start using FOSS and selling services around it from anywhere in the world. No license, special qualification or permission required.

Optionality comes from the fact that anyone can do trial-and-error tinkering with the software and keep their improvements. They can also choose the best or most appropriate changes and modifications made by other people.

Without optionality a software ecosystem will not be antifragile, and without FOSS, a software ecosystem will not have optionality.

Additionally, software that is Free (as in Libre) displays qualities much more akin to information than non-Free software, which due to it’s source opacity functions more as a black-box application. As Taleb explains in his book, information is antifragile. You can’t easily kill it, and by trying to suppress it you have to acknowledge its existence and refer to it.

Lastly, the lower barrier to entry fosters competition.

In order to differentiate themselves, individual developers or businesses need to compete on quality and/or price. It’s survival of the fittest, as opposed to the rent-seeking, vendor lock-in and monopoly so often seen in proprietary software ecosystems.

In the book, Taleb alludes to Silicon Valley’s fail fast, be foolish model as being antifragile as opposed to a more centralized environment with inherent systemic risks, such as the New York banking system (his example).

Silicon Valley might provide an antifragile method for producing businesses, but the software produced by these businesses don’t necessarily result in antifragile ecosystems.

Look at the software businesses that have arisen out of Sillicon Valley. Facebook and Google are both large and centralized. Increasing centralization and size results in increasing fragility of the web.

If Google’s servers were to go down (or be compromized or blocked by a government), a very large part of the web would cease to function. The chance of Google’s servers going down are of course very small, it’s a negative Black Swan event. Something which society nevertheless needs to shield itself against.

Antifragile against what?

Having mapped out the requirements for an antifragile software ecosystem, it makes sense to consider the kinds of stressors such an ecosystem will be antifragile against.

This section is a bit more speculative, mostly because we cannot know the exact nature and extend of the stressors a system will be exposed to. Luckily we don’t need to know the stressors to be able to make use of antifragility.

It’s also important to know that stressors are information. In fact, stressors often appear through the disclosure of “negative” information. When negative information is revealed, it can cause shocks to software ecosystems. The fragile ones will suffer and might even break down, while the antifragile ones will ultimately benefit.

Here are some examples of stressors a software ecosystem can be antifragile against:

Malicious Hacking (Cracking)

FOSS is more resistant to cracking than closed source software. Additionally, distributed systems are much harder to target and infect.

Increased awareness and fear of cracking, makes FOSS, its ecosystems and communities more attractive to software users (individuals, companies, governments etc.) and therefore antifragile to the “bad” news.

Surveillance

A big part of surveillance is actually also cracking, so the same rules apply here.

It’s much more difficult for eavesdroppers to embed backdoors in software with freely available and visible source code.

Encryption software is especially vulnerable and can’t be trusted if its closed source.

News of security flaws spread quickly throughout the community and fixes are generally pushed out almost immediately. You won’t easily have the problem of a FOSS community member privately disclosing security flaws to an intelligence agency (so they might use if for surveillance) while hiding it from the users.

Again as surveillance and privacy fears increase, FOSS becomes more attractive, making the communities and ecosystems around it more antifragile.

Negative economic circumstances

Many governments are experiencing budgetary constraints and employing austerity. Governments are able to make significant savings when they switch to FOSS ecosytems. Not mentioning the increased autonomy and control of their software and resulting digital formats.

Other stressors

There are of course many more stressors that might ultimately benefit distributed FOSS ecosystems, for example censorship and economic boicot, but for the sake of brevity I won’t go into detail for each one.

Conclusion

I’ve mapped out the requirements for a software ecosystem (and community) to be antifragile, and have alluded to Plone as an example of such an ecosystem.

The Plone software, like other Free software (Drupal, Wordpress, Apache, Firefox, Libre Office, GNU/Linux etc.), is a technological commons. It belongs to humanity as a whole.

Everything I wrote also applies to the ecosystems around cryptocurrencies, like Bitcoin.

There are usually vibrant communities of developers, packagers, integrators, evangelists and users around these pieces of Free software, and they usually conform to the requirements mentioned above. Members are distributed all over the world, there is redundancy in development skills and support offerings and the source code is Free (as in beer and in speech).

Converse.js 0.7 released: Now supports Off-the-record encryption

Sun, 10 Nov 2013 00:00:00 GMT

I’m happy to announce that the 0.7 release of Converse.js now supports Off-the-record (OTR) encrypted messaging.

The OTR protocol not only encrypts your messages, it provides ways to verify the identity of the person you are talking to, plausible deniability and perfect forward secrecy by generating new encryption keys for each conversation.

How this works is actually fascinating. The encryption keys are generated via a Diffie-Helman exchange. On youtube there is a video with a very simple and intuitive explanation on how the Diffie-Helman key exchange works. For more details, you can also read the OTR version 3 protocol spec.

I was able to add OTR support for converse.js by using Arlo Breault’s excellent otr.js library, which is also used in Cryptocat and Diaspora.

Many thanks go out to him for this.

Note:

Arlo cautions that the otr.js library has not yet been properly vetted by security researchers and shouldn’t be used in life or death situations.

This word of caution applies doubly so to its integration into converse.js!

I’m not a professional cryptographer and this should be seen as an experimental feature.

Caveat emptor!

Screencast

Note: For detailed fullscreen, make sure that video quality is set to HD, by clicking the gear icon on the video player.

On Javascript Cryptography

Implementing Cryptography in Javascript is relatively controversial and in 2011 Matasano security wrote a scathing criticism called Javascript Cryptography Considered Harmful.

I read that article before I started working on OTR support for converse.js and I’d like to go through some of its criticisms step by step to explain how they might apply to converse.js and what I’ve done to mitigate them.

If you’re going to read on, it would make sense to go read that article first.

Secure delivery of Javascript to browsers being a chicken-egg problem

Their criticism appears to apply to the case where developers use Javascript cryptography to implement secure communications with a server instead of relying on a standard protocol such as SSL/TLS.

The OTR crypto is not trying to compete with or replace SSL/TLS.

Sending data in the clear with normal HTTP is not secure and should be avoided. Therefore, if you are going to use OTR with converse.js, always make sure that the site users HTTPS (i.e. SSL/TLS).

Ideally you would also want a mechanism to sign the Javascript and then verify it’s authenticity once it’s been downloaded.

This can be done with code that runs in browser extensions (like Cryptocat does), but (AFAIK) unfortunately not with browser-agnostic javascript (like converse.js).

This looks to me like the achilles heel for browser javascript crypto.

Besides the webserver serving the Javascript, it’s also important that the BOSH service you use employs SSL/TLS, as well as the XMPP server the BOSH service connects to.

The XMPP community is on a drive to secure all the major public XMPP servers. See the Manifesto created by Peter Saint-Andre and signed by many in the XMPP community.

On xmpp.net they let you test the security of servers on the Jabber/XMPP network.

Besides using javascript served via TLS, you can also download the converse.js tarball to your computer, unpack the files and access them directly. The index.html page will open up a page identical to the one on conversejs.org.

Browser Javascript being hostile to cryptography

The prevalence of content-controlled code.

If the website fetches javascript from thirdparty websites (for example to serve advertisements), then you not only have to trust the server to serve non-malicious code, but also all those 3rd party servers.

The solution is to not use such a site for secure communications. Clear your cache regularly, Use hosts you trust, or as I’ve mentioned above, bypass the webserver entirely by using converse.js from your own filesystem.

The malleability of the Javascript runtime.

Javascript is indeed very malleable. Methods and attributes on objects can be replaced during run-time (see Duck Typing and Monkey Patching),

This makes it easy to inject malicious code into Javascript.

However, in order to do this, the attacker’s code needs access to those Javascript objects, and fortunately Javascript provides a way of hiding sensitive objects inside closures, thereby preventing an attacker from accessing and manipulating them.

Simply put, a closure defines a limited scope for objects, variables and functions which when used correctly, cannot be accessed from any other Javascript code.

Any variables, objects or functions defined locally inside another (parent) function (with the var) keyword, are not accessible outside of that parent function.

If the parent functions is anonoymous (is not referenced anywhere) or the child function’s runtime outlives the parent’s, you have a private, enclosed scope with data that cannot be accessed from outside that scope.

All the data in converse.js is encapsulated inside such closure.

The lack of systems programming primitives needed to implement crypto.

Matasano claims that Javascript lacks a secure random number generator RNG.

Without having authoritative knowledge on whether their claim is valid, I’ll refer you to this blog post by Nadim Kobeissi (author of Cryptocat), where he claims:

This is in fact not the case. window.crypto.getRandomValues() is a secure random number generator available in JavaScript, and supported by major web browsers.

His blog post is well worth reading, and he also addresses some other criticisms for which I didn’t have anything to add.

Conclusion

OTR encryption in converse.js provides increased security and privacy for your chats. Messages are encrypted and not logged or cached

Any Jabber/XMPP services that snoop on your communications by logging your messages (like for example Facebook does) will only have access to the encrypted ciphertext, making it useless for surveillance or exploitation.

Features coming future releases

The current implementation of OTR doesn’t have any encryption policy support. With that, I mean the ability to set a policy such as Always encrypt messages.

Currently you need to always manually enable encryption.

Distributed services and self-hosting

There are more and more Free and Open Source (FOSS) self-hosted web-applications becoming available as alternatives to so-called centralized “cloud” solutions.

See for example roundcube, kolab, ownCloud, MailPile, friendica and buddycloud.

Converse.js can be integrated into any of these applications, and since you should ideally host them yourself, you’ll have more control on what Javascript is actually being served.

Feedback, patches, donations

All of the work I did on OTR integration was in my free time and free of charge. A gift and my personal contribution towards the movement for providing free communication solutions that are not founded upon surveillance and commercial exploitation of personal data.

All the code is open source and contributions are always welcome, be it in audits, features, bugfixes, documentation, words of encouragement or tips.

Converse.js: XMPP instant messaging with Javascript

Sat, 02 Mar 2013 00:00:00 GMT

It is now possible to use converse.js, the Javascript from collective.xmpp.chat, in a standalone way, to communicate with any public Jabber account.

Lately I’ve been spending time refactoring converse.js the Javascript library used by collective.xmpp.chat the instant messaging add-on for the Plone CMS. My goal is to make it usable on it’s own, requiring nothing else except an XMPP server to communicate with.

This would enable any website owner to add instant messaging functionality to their website, and due to the federated nature of XMPP, users could chat to any other public XMPP account (once they have been accepted as a contact).

One thing that previously prevented you from using converse.js on its own, was the fact that it made XHR calls to the Plone backend to fetch user data. To fix this, I added vCard support, to converse.js but also to Plone by adding it to collective.xmpp.core.

Last week I reached a significant milestone on the path to this goal, and I’d like to take a moment and share with you.

It is now possible to use converse.js (in a static HTML page) to communicate with Jabber accounts on any public server.

In the demo below, I illustrate this by chatting to a Google user and to a Jabber.org user. In the converse.js page, I’m authenticated with a Jappix.com account and I also use their connection manager to connect to their XMPP server. If you’re doing XMPP via HTTP (i.e in the browser), you’ll need a connection manager as a bridge to your intended XMPP server. Thanks a lot to the Jappix guys for making their connection manager public!

Note: For detailed fullscreen, make sure that video quality is set to HD, by clicking the gear icon on the video player.

If you are interested in the project, please contribute by forking the code on github.

Thanks a lot to Alec Ghica and David Ichim from Eau de Web who’ve made various improvements to both collective.xmpp.chat and converse.js over the past months.

Report on the Beer and Wine Sprint in Cape Town

Tue, 29 Jan 2013 00:00:00 GMT

The Beer and Wine sprint wrapped up on Sunday and it’s now a good time to report on what was South Africa’s first ever core Plone sprint. This was a joint sprint between Cape Town and Munich, which in hindsight, was a very good idea. The two groups of sprinters had daily meetings via Google hangouts, and the structure and support we received from Munich helped a lot. Connecting with the sprinters on another continent brought home the point that we are a global community and that no core Plone work is done in isolation. The daily hangouts also motivated us to get cracking so that we could have something to report on!

South Africa is geographically isolated from the places where most Plone development takes place (notably Europe and North America) , which results in the local Plone community not getting as much exposure to sprints and community events as would have been possible otherwise. The goal of this sprint was to help change that, and to provide an opportunity for us to contribute back to the Plone community from which we’ve received so much.

Most of the people who attended the sprint were first-timers, and it was a joy to see the enthusiasm with which people started working. Eric Steele and David Glick flew down to join us here at the tip of Africa, and provided much needed experience and support.

About 14 people met at sprint venue on Thursday morning and after introductions and a quick hello to the Europeans via the wormhole, we formed groups and started sprinting.

Mike Metcalfe and Campbell Mckellar-Bassset (rockfruit) started working on porting CMFPlone tests from Products.PloneTestCase to plone.app.testing. They kept at it with dogged determination for the next four days of the sprint and made good progress. Izak Burger and Jurgen Blignaut tackled some TinyMCE bugs, while Rudi Ackhurst, Witek, Johan Beyers and myself started working on Plip #13260. We did this via Dojo sessions, which we held for most of the sprint. Other people did tutorials and proofreading of documentation.

It appears as if the sprinting bug has bitten at least some of the people who attended, and there’s already been talk of doing this again some time. I personally had an amazing experience and learned a lot while organising this event.

To the bittereinders who sprinted every single day (Thursday to Sunday), thanks for your energy and support! This would have not been possible without you. We laid a solid foundation for future core Plone contributions from our local group.

Lastly, thanks a million to everyone involved in making this sprint a reality, Eric, David and all the people who attended, our sponsors the Praekelt and Plone Foundations, the guys in Munich and all those who helped with support and advice. And thanks to Liz Leddy who planted the initial seed in my head that this was not only possible, but a actually great idea!

P.S Credit for 3 of the photos goes to David Glick

Slides and Video for my talk on XMPP+Plone at Plone Conf 2012

Fri, 02 Nov 2012 00:00:00 GMT

The video of the talk I gave at in Arnhem at the 2012 Plone Conf has been uploaded to Youtube. Check it out if you are interested in integrating XMPP with Plone.

Thanks a lot to the person who uploaded this video! After seeing Cillian de Roiste spending much of his free time uploading the videos of the Munich Plone Konf, I realise how much effort this is!

Follow-up info on XMPP-stuff after the conf

In the talk, I mention collaborative editing and showed demos of it (with TinyMCE). Unfortunately soon afterwards I came upon a significant flaw in the current approach. If you are interested in the collaborative editing possibilities for Plone, please read Collaborative editing of HTML with TinyMCE not going to happen soon.

Also since this talk, I’ve been quite busy improving collective.xmpp.chat to get it production ready. I’ve made an initial release to Pypi, which was actually supposed to be a 0.1 alpha release (but which I messed up somehow). Please consider collective.xmpp.chat to still be alpha software and not yet production ready! The most notable issue at the moment is poor support for having multiple tabs open. I’m aware of this and will work on a solution soon. Update: Fixed in the latest release: http://plone.org/products/collective.xmpp.chat/releases/0.2a2

I’m also in the process of extracting the Javascript files from collective.xmpp.chat and putting them in their own repository, called converse.js. The idea behind this is to completely decouple the JS from Plone and in doing so allow one to implement/deploy something like collective.xmpp.chat for other non-Plone backends.

That’s it for now. Thanks for the support and encouragement from everyone in the Plone community. You guys rock!

Real-time, collaborative applications in Plone by jcbrand

Real-time, collaborative applications in Plone from jcbrand

Collaborative editing of HTML with TinyMCE not going to happen soon

Mon, 22 Oct 2012 00:00:00 GMT

TL:DR: Collaborative editing doesn’t work well with XML/HTML. Right now, the solution seems to be to use Markdown instead.

During my talk on XMPP applications in Plone, someone asked whether I was aware of any issues with regards to collaborative editing in TinyMCE.

I don’t exactly remember my reply, but it came down to “no”, I wasn’t aware of any problems with it. Just the next day however, at the sprints, I encountered a fundamental problem with regards to collaborative editing of Rich Text through TinyMCE.

The problem lies in the fact that the Diff-Match-Patch is meant for plain text only. It has no concept of structured content such as the a DOM-tree and can therefore create problems when one tries to patch HTML content. It seems that mainly two things can happen. Firstly, because your HTML nodes are not atomic, they can be broken by the patching algorithm inserting text inside them. Secondly, the patching can result in HTML nodes appearing in the wrong order. (See what the author of Diff-Match-Patch had to say about it on the project page and his excellent explanation on stackoverflow).

Ideally one should use a patching algorithm that is tree-based and which will maintain the integrity of the DOM-tree. When using the plain text algorithm, there are however two approaches to try and mitigate the above issues.

Strip all HTML tags, diff-match-patch the resulting plain text and re-insert the original tags.
Replace all tags with Unicode chars outside of the used range (thereby making them atomic), apply diff-match-patch and then restore all tags.

The problem with the first approach is that your markup is now immutable. So any markup changes a collaborator makes (e.g bold, italics, heading or other formatting) will be discarded. I think this is a dealbreaker and rules out this approach.

The problem with the second approach is that even though your HTML nodes are now atomic, they might still appear in the wrong order.

Because of this, I think trying to do collaborative editing via an HTML-based Wysiwyg editor is not the way to go. Instead, a Markdown editor (such as Epic Editor) seems to be the way to go. Markdown is plain text, and therefore doesn’t suffer from the problems mentioned above.

I think I have created the impression at the Plone conf that production-ready collaborative editing via TinyMCE will soon be a reality, and for that I would like to apologize.

Instant Messaging for Plone with javascript and XMPP

Mon, 30 Jul 2012 00:00:00 GMT

Late last year Yiorgis Gozadinos who at the time was still working for Jarn, released a video showing his work in integrating XMPP (eXtended Messaging and Presence Protocol) into Zope/Plone.

I had some experience in creating a chat client for Plone before and was inspired by the possibility of porting it to XMPP. In the end it wasn’t so much of a port but rather a complete rewrite.

Today, I’m relieved and happy that it’s finally in a state where I can show it to you. I’ve been spending most of my free time the last two months writing collective.xmpp.chat.

It’s been both frustrating and fun (mostly fun) and a very valuable learning experience. Like that one week where I rewrote the entire codebase to make use of backbone.js. Aaah, fun times.

Some features already implemented:

Single user chat Roster support (adding/removing contacts) (XEP-0144) Setting presence status (online, offline, away, busy) Custom status messages Multi-user chatrooms (XEP-0045) Chat windows stay open upon page reload Offline storage of messages using HTML5 localstorage (still needs to be secured) Message actions (/me) Setting the topic of a chatroom (/topic)

Because almost all the code is written in Javascript (converse.js is about 1500 LOC), you can even stop the Zope server and still continue to chat. Unfortunately I forgot to show this in the screencast.

If anyone has any questions about the code, how to install, configure and use it, please don’t hesitate to contact me. The add-on includes a buildout.cfg file that will set up Zope/Plone, Nginx and ejabberd for you. But please beware, the code is still in a pre-alpha state.

And then lastly, I plan to give a talk at the upcoming Plone Conference in Arnhem. This talk will focus on collective.xmpp.chat, the underlying technologies and how to write XMPP-enabled add-ons.

If you consider this an interesting topic, please go and vote for this talk to ensure that it gets accepted.

Screencast

Note: If you have trouble reading the text, click on the gear icon and set the video to High Definition.

Plone and Dexterity: Enable behaviors per content type instance

Tue, 03 Jan 2012 00:00:00 GMT

plone.behaviors are great, but because they are stored in the FTI, they are enabled for all instances of a content type. This blogpost will show how you can enable a plone.behavior for a specific instance.

Note: This blogpost assumes that you are already familiar with a lot of the Zope/Plone programming technologies and concepts. Every time I mention a specific add-on or technology, I've added a link to documentation.

UPDATE: There is now a add-on for Plone based upon the approach taken in this blog post, please see: [collective.instancebehavior](https://pypi.python.org/pypi/collective.instancebehavior).

Behaviors, provided by the plone.behavior package, provide a very useful way of extending the functionality of a Dexterity content type. Unfortunately, the fact that a content type’s behaviors are stored in its factory contents information (FTI) inside the portal_types tool, means that they are class (or type) specific, and not instance specific. This means that behaviors can only be enabled globally, in other words only for all the instances of a specific type.

So what can we do when we want to enable behaviors only for a specific instance of a content type? At Syslab we for example had a client who wanted a basic folderish Workspace content type, that can be extended into a more featureful custom Project type.

To enable per instance behaviors, we would need a way to store those behaviors on the instance. I decided to use annotations for this purpose. Additionally we must create our own IBehaviorAssignable adapter that overrides the default one to not only look for an object’s FTI registered annotations, but also look for the behaviors stored in the current instance’s annotations.

I created this adapter in a module named behavior.py:

from myproject.types.config import INSTANCE_BEHAVIORS_KEY as KEY
from myproject.types.workspace import IWorkspace
from plone.behavior.interfaces import IBehavior from
from zope.annotation import IAnnotations
from zope.component import adapts, queryUtility plone.dexterity.behavior
import DexterityBehaviorAssignable


class DexterityInstanceBehaviorAssignable(DexterityBehaviorAssignable):
     """ Support per instance specification of plone.behavior behaviors """
     adapts(IWorkspace)

    def __init__(self, context):
        super(DexterityInstanceBehaviorAssignable, self).__init__(context)
        annotations = IAnnotations(context)
        self.instance_behaviors = annotations.get(KEY, ())

    def enumerateBehaviors(self):
        self.behaviors = self.fti.behaviors + self.instance_behaviors
        for name in self.behaviors:
            behavior = queryUtility(IBehavior, name=name)
            if behavior is not None:
                yield behavior

Now register this adapter via ZCML in configure.zcml:

<adapter factory=".behavior.DexterityInstanceBehaviorAssignable" >

We know what we want to use as a storing mechanism (Annotations) for per instance behaviors, and we have a custom adapter that knows to look for them, but we still need a way to save new behaviors to a specific instance’s annotations.

In my case, I wanted to extend the Workspace content type with a Project behavior that provides additional fields. Refer to my blogpost on schema-extending a dexterity type for an explanation on how to create such a custom behavior. This was however only my specific use-case, and any behavior can be applied in this way.

To provide a mechanism for saving new per instance behaviors, I created a browser view @@enable_project, which can be called on any Workspace instance. When it gets called, my custom behavior (with full path myproduct.types.project.IProject) gets saved in the annotations of the specific Workspace.

class EnableProject(grok.View):
    grok.context(IWorkspace) grok.name('enable_project')
    grok.require('cmf.ModifyPortalContent')

    def render(self):
        context = aq_inner(self.context)
        annotations = IAnnotations(context)
        instance_behaviors = annotations.get(KEY, ())
        instance_behaviors += ('myprodcouct.types.project.IProject',)
        annotations[KEY] = instance_behaviors

        IStatusMessage(self.request).add(
            _(u"This Workpace has now been turned into a Project."),
            u"info"
        )
        return self.request.RESPONSE.redirect('/'.join(context.getPhysicalPath()))

You may want to provide a UI mechanism for enabling per instance behaviors. I decided to do this by adding a new object_buttons action in the actions.xml Generic Setup file:

<?xml version="1.0"?>
    <object name="portal_actions" meta_type="Plone Actions Tool" xmlns:i18n="<http://xml.zope.org/namespaces/i18n>">
        <object name="object_buttons" meta_type="CMF Action Category">
            <property name="title"></property>
            <object name="enable_project" meta_type="CMF Action" i18n:domain="plone">
                <property name="title" i18n:translate="">Enable project</property>
                <property name="description" i18n:translate=""></property>
                <property name="url_expr">string:\$<object_url/@@enable_project></property>
                <property name="available_expr">python:context.portal_type == 'myproject.types.workspace' and not context.is_project() or False</property>
                <property name="permissions"> <element value="Modify portal content"/> </property>
                <property name="visible">True</property>
            </object>
        </object>
    </object>

This provides a convenient way (via the actions dropdown) for the end user to turn a Workspace into a Project.

The difference between BrowserSkins and BrowserLayers

Fri, 08 Jul 2011 00:00:00 GMT

In this blogpost I’d like to shed some light on two concepts that sound similar and can be quite ambiguous and confusing when first encountered.

When we register certain Zope Components (via ZCML or Grok), we can specify a layer attribute for which we would provide an Interface. This will restrict these components to be available only when that Interface is provided by the Request object.

In ZCML it looks like this:

<browser:page
    name="foo" for="*" template="my-template.pt"
    permission="zope.Public" layer=".interfaces.IMyLayer"
/>

And with Five.Grok:

class Foo(BrowserView):
    """ """
    grok.template("my-template")
    grok.name("foo")
    grok.layer(interfaces.IMyLayer)

How would the Request provide an interface? Well, with the Zope Component Architecture, you can make any object provide any interface, by using the alsoProvides method. In this way we can use interfaces as markers, to mark or identify certain objects.

In Plone this is very handy, because it allows us to only enable or disable certain functionality, by registering it against a specific Interface and then marking the Request or not. This “marking” of the request is done when your Plone add-on product is installed or uninstalled in the configuration panel.

How this is done, depends on two different implementations, one in plone.theme (called BrowserSkins), and one in plone.browserlayer (called BrowserLayers).

Both approaches do the same thing, they mark the Request with an Interface, but they way they do it differs, and that can be bewildering to the uninitiated.

BrowserSkins

plone.theme provides a way mark the request with an interface when it’s associated with a skin installed in the portal_skins tool. Plone 4 comes with two skins, the Classic skin and the newer Sunburst. In other frameworks or environments, a skin could be called a theme.

Lets pretend we were creating a skin with a new egg product called collective.kickasstheme. If you’ve ever generated a Plone theme package with PasteScript and ZopeSkel, then you’ve now doubt seen the following.

In collective.kickasstheme.browser.interfaces.py we have this interface declared:

from plone.theme.interfaces import IDefaultPloneLayer

class IThemeSpecific(IDefaultPloneLayer):
    """Marker interface that defines a Zope 3 browser layer.
       If you need to register a viewlet only for the "Custom Theme"
       theme, this interface must be its layer (in
       kickasstheme/viewlets/configure.zcml).
    """

Registered with this ZCML:

<interface
    interface=".interfaces.IThemeSpecific"
    type="zope.publisher.interfaces.browser.IBrowserSkinType"
    name="Kickass Theme" />

And then with browser resources registered against it, for example:

<browser:resourceDirectory
    name="collective.kickasstheme.images" directory="images"
    layer=".interfaces.IThemeSpecific" />

  <browser:viewlet
      name="collective.someviewlet"
      manager="plone.app.layout.viewlets.interfaces.IPortalFooter"
      class=".viewlets.MyViewlet" layer=".interfaces.IThemeSpecific"
      permission="zope2.View" />

The skin is installed with GenericSetup, so we need to inform it about it in collective/kickasstheme/profiles/default/skins.xml

<object name="portal_skins"
    allow_any="False" cookie_persistence="False"
    default_skin="Kickass Theme">
  <-- Here comes a list of registered folders that are registered in portal_skins -->
</object>

plone.theme registers an event subscriber (also known as an event listener) for the IBeforeTraverseEvent. In this subscriber, it queries for the BrowserSkin that is installed with the current Skin in portal_skins and then uses directlyProvides, to mark the Request object with this BrowserSkin. If our skin/theme product is installed, then the IThemeSpecific interface declared in our egg collective.kickasstheme will be the BrowserLayer and will mark the Request.

You can see the code for yourself in plone/theme/layer.py and plone/theme/configure.zcml.

This means that all components (such as BrowserViews, ResourceDirectories, viewlets, etc.) registered against our BrowserSkin will now appear. As soon as we uninstall our theme egg, they will again disappear, because the BrowserSkin is again removed.

Drawbacks

BrowserSkins have two major drawbacks.

You can only have one of them active at any time. If you need more than one BrowserSkin (which is just an Interface) on the request (i.e to render portlets registered against another one) you will have to subclass that other BrowserSkin.
You also need to create a theme product (with a skin registered in portal_skins) every time you want to create your own BrowserSkin.

So, what do we do if we for example want to create a new viewlet or portlet in a non-theme product, but don’t want it to render unless the product is installed?

BrowserLayers

plone.browserlayer was developed to solve this problem.

So, now we can create any egg, it doesn’t have to be a theme egg, and the declare inside it our BrowserLayer.

We start off by creating the marker Interface that will be given to the layer attribute:

from zope.interface import Interface

class IMyProductLayer(Interface):
    """Browser layer for this particular product """

Then add a browserlayer.xml file to the GenericSetup profile directory, with the following:

<?xml version="1.0"?>
    <layers>
        <layer name="my.product" interface="my.product.interfaces.IMyProductLayer"/>
    </layers>

When we now install our egg in Plone’s control panel, our BrowserLayer will also added to plone.browserlayer’s persistent registry of “installed” layers and applied to the Request object during runtime.

All components and resources registered against this BrowserLayer will be activated and visible. Once we uninstall our egg, they will disappear again.

Benefits:

The benefits of using BrowserLayers are basically the inverse of the drawbacks of using BrowserSkins.

Any egg can have a dependency on plone.browserlayers and create BrowserLayers, not just theme eggs.
You can have multiple BrowserLayers currently on the Request.

Conclusion

Both BrowserSkins and BrowserLayers allow you to mark and unmark the Request object and thereby switch components on or off. They also allow you to override existing components, by simply registering your custom components against a specific layer. Previously, you had to override them in overrides.zcml and this could only be done once.

BrowserSkins had some drawbacks and BrowserLayers where developed to address these. That said, both are still around and both are still in use.

Lastly, please use them! It’s bad practice to register components for Plone sites that are not tied to a layer. If you do. they will be active in the Plone site, regardless of the egg being installed or not, potentially causing havoc when serving multiple Plone sites.

Schema-extending Dexterity content types with five.grok and plone.behavior

Mon, 13 Jun 2011 00:00:00 GMT

This is a quick blog post about using plone.behavior to change the schema of Plone’s Dexterity content types. Hopefully it’ll help people familiar the Archetypes’ schema-extending paradigm, to achieve the same effect with Dexterity types.

Of course, Dexterity is not Archetypes, and plone.behaviors are not schema-extenders (well, not only), but sometimes you want to do the same thing, whether you’re using Archetypes or Dexterity, and that is extend the schema of an existing content type created in a different product (that you don’t want to or cannot edit) in your own product.

Martin Aspeli has provided much more thorough and complete documentation on behaviors here. In short, behaviors can be seen as conditional adapters. The condition being that the behavior must be listed in the behaviors property of the type’s FTI (Factory Type information).

Schema-extending in Archetypes also happened through adapters, but they were not checked against the type’s FTI, and were thus not conditional.

Example:

Suppose we have a Sector type in another package, and that we want to add another field statistics_level. We don’t want to modify this other package, so we’ll use plone.behavior to “extend its schema”.

In our own package, we create sector.py and add the following code:

from zope import schema
from plone.directives import dexterity
from plone.directives import form
from plone.autoform.interfaces import IFormFieldProvider
from plone.app.dexterity.behaviors.metadata import MetadataBase
from plone.app.dexterity.behaviors.metadata
import DCFieldProperty

class IExtendedSector(form.Schema):
    """ """
    statistics_level = schema.Choice(
        title = _("label_statistics_level", default=u"Statistics Level"),
        description = _(
            "help_statistics_level",
            default=u"Level 1: Basic statistics. Level 2: More detailed statistics."
        ),
        required=True,
        vocabulary = SimpleVocabulary([
            SimpleTerm(1, title=u"1"),
            SimpleTerm(2, title=u"2"),
        ]),
        default=1
    )

The interface IExtendedSector defines the new field statistics_level that we want to add.

alsoProvides(IExtendedSector, IFormFieldProvider)

We then mark IExtendedSector with IFormFieldProvider to signal that it should be processed for form fields.

class ExtendedSector(MetadataBase):
    statistics_level = DCFieldProperty(IExtendedSector['statistics_level'])

Next we need a class that implements the MetadataBase behavior adapter and which will act as the adapter factory.

Now we need to wire this adapter in with ZCML:

<plone:behavior
    title="Extended Sector"
    description="Provide additional statistics_level field"
    provides=".sector.IExtendedSector"
    factory=".sector.ExtendedSector"
    for="plone.dexterity.interfaces.IDexterityContent" />

Lastly we need the portal_types FTI (Factory Type Information) to be updated with our behavior. Go the the ZMI, then portal_types and then click on the Dexterity type you want to extend. You will see that there is a lines field called Behaviors.

Add the interface of your new extender here, i.e collective.myproduct.sector.IExtendedSector

That’s basically it!

You probably want to add the behavior interface to a GenericSetup XML file that defines the type you are extending, to avoid having to manually add it to portal_types every time you create a new site (it’s also very easy to forget about this step).

Something like this in your type’s GS-XML file:

<!-- List of enabled behaviors -->
<property name="behaviors">
    <element value="plone.app.content.interfaces.INameFromTitle" />
    <element value="collective.myproduct.sector.IExtendedSector" />
</property>

Oh, and lastly… you’re going to need five.grok for this. Your package needs to be grokked, otherwise the plone.directives.form form hints won’t be activated.

In your configure.zcml:

<grok:grok package="." />

You can do a lot more than just the above withDexterityand plone.behavior, take a look at Martin’s documentation for more info.

New spelling and grammar checker for TinyMCE

Fri, 25 Mar 2011 00:00:00 GMT

The spellchecker “After the deadline” is now available as an add-on for TinyMCE in Plone. It’s multilingual, open-source, platform agnostic and even does grammar checking.

I’ve actually already added this add-on two months ago but didn’t get around to mentioning it. After the deadline is an open-source spelling and grammar checker that can be integrated with web based WYSIWYG editors, such as our beloved TinyMCE.

This new spelling checker provides some advantages over the existing (and still default) IESpell. Firstly, IESpell only works on Microsoft Windows. Secondly, IESpell is only free for non-commercial purposes. After the deadline however is open-source (GPL license, see here) and platform agnostic. Oh, and it’s multilingual as well! To top it off… the killer feature, for us at least, is that it has a grammar checker. Pretty impressive IMHO.

The actual thing doing the spellchecking, is a Java server app, which you should download and install on your own server. There is a public default option for testing and demonstration purposes, but it comes with no guarantees.

After the deadline is included with all Products.TinyMCE releases since 1.2.1, but it’s not the default spell checker. To enable it is at least straightforward.

Enabling AtD in TinyMCE for Plone:

Go to the Plone control panel and click on “TinyMCE Visual Editor”
Click on ‘Toolbar’ (middle left)
Make sure that ‘spellchecker’ is checked.
Click on ‘Libraries’ (top right)
Under Spellchecker plugin to use, choose After the deadline
Under AtD Service URL, choose your ATD server’s URL. (The default is their - public service)
It’s however recommended that you install your own AtD spellchecker service
See here for more details.
You should now have After the deadline enabled and have a spellcheck button in your TinyMCE editor.

If you’d like a demonstration of ATD, click here.

Plone 4 compatible release of Quills

Thu, 10 Mar 2011 00:00:00 GMT

Since I’ve started this site, I’ve been using Quills for blogging functionality in Plone. So far, it’s been perfectly fine for my relatively simple needs and I haven’t found a reason to change. Unfortunately, except for some work to port Quills to Plone 4, development and bugfixes on Quills has completely dried up. No new releases were made for more than a year and the mailing list was almost dead. I only realised this when I fixed some bugs while porting my site to Plone 4 and couldn’t find anybody to help making a new Plone 4 compatible release.

Eventually I got hold of Tim Hicks, one of the original maintainers of Quills, and I offered to help out with maintaining Quills and to make the new releases.

So, I’m happy to announce that new Plone 4 compatible releases of the Quills packages have been made. Notably Products.Quills, Products.QuillsEnabled and quills.app (all now at version 1.8a1). This new release is labeled alpha, as I’m still very new to the Quills codebase, but I’m already using it on this blog without any major problems.

If there is anyone who is still using Quills on a legacy Plone site and would like to migrate to Plone 4, just pin your Products.Quills (and Products.QuillsEnabled) version to 1.8a1 in your versions.cfg or bulidout.cfg.

Hopefully everything goes well (did I mention it’s alpha ;) but if not, feel free to let me know and I’ll try to help out.

New beta of Babble Client (and related packages) released

Wed, 19 Jan 2011 00:00:00 GMT

After spending a few months acclimatising to a new life and continent, I’m happy to be productive again. I’ve released a new beta version of Babble.Client and other packages.

In the last 6 months, my girlfriend and I went backpacking in South America and then we moved continents to go live in South Africa after having stayed in Germany for a while. To be honest, we completely underestimated the amount of effort involved in adapting and integrating into our new environment and the amount of admin that was waiting for us after having lived without any significant responsibilities for 4 and a half months while backpacking.

We had virtually no worldly possessions except for our backpacks (and some boxes in storage), and needed to get a place to stay and some office space for me.

Needless to say, it took us a while to get back on our feet and for me to get into a good working rhythm again.

New packages released:

So, I’m happy to finally be back in the swing of things and to be releasing new updates to some of the packages I help maintain.

I’ve released new versions of the packages in the Babble suite, specifically worth mentioning is the new beta version of babble.client (1.4b1) which provides support for email usernames (available in Plone 4).

Together with the client release, I had to release new versions of the messaging server: babble.server (0.3.1), actionbar.babble (0.1b1), a package which provides Babble integration for the actionbar.panel, and also a new version of collective.js.blackbird (which I use for Javascript logging). You can use babble.demo (0.2) to quickly set up a prototype environment in which to test Babble.

Improvements and new features:

I’ve gotten a few requests for new Babble client features the last months, for example the ability to broadcast chat messages and to restrict contacts to certain groups. Some of the smaller requests, I’ve implemented in my spare time (such as better URL handling in messages, and making the portlet header configurable), but others would simply take up too much of my valuable free time to implement. I would however be happy to assist anyone considering implementing these features themselves.

Some improvements I would like to eventually implement myself are better Javascript testing (for which I need to remove all DTML statements) and the ability to push messages to the client via something like comet (instead of polling).

There are going to be some PloneSA sprints coming up in the Cape Town area here in South Africa where I would hopefully be able to work on some of these areas.

As always, contributions are very welcome, thanks to Cleber J. Santos for fixing some Plone 3 compatibility bugs!

Feel free to contact me with questions or suggestions!

South American travel diary, Part 4. September 2010

Mon, 11 Oct 2010 00:00:00 GMT

3 September, Starting our hike to Choquequirao

Choquequirao is a lesser known ruined Inca city located on the Salkantay mountains. It can only be visited after a strenuous two day hike, down into the Apurímac valley and then up again on the other side of the mountain. This makes it much less visited and also more mysterious than the famous Inca ruins at Machu Picchu. So far only about 30-40% of the site has been excavated.

The tour we booked consisted of four days of hiking, two days to Choquequirao and two days back for a total of about 70 km of steep, mountaneous terrain.

We got up early in the morning and headed by bus to the small town of Cachora, less than 200km removed from Cusco but still about 4 hours of driving on the endlessly curving and car-sickness inducing mountain road.

Cachora is a quaint and very rural town, with just about every building we could identify built out of mud. We waited at the back of a small shop, watching the chickens, pigs and mules in the backyard, as Jesús, our guide and cook, prepared lunch. Inside a dark room to the left was a colony of guinea pigs, most of whom inevitably will end up on someone’s dinner plate someday. I considered eating guinea pig in Arequipa, a local delicacy, but after seeing a photo of a skinned guinea pig spread out on a plate, I decided to pass.

We were supposed to be a group of four people, but at the last minute, the other couple canceled because of food sickness. That meant it was only Manuela and I, Jesús our guide and Milano, the horse-guy, which suited us just fine. Additionally, we had a horse and a mule with us. They carried our luggage, food and the tent, as we headed off to the wild for the next four days. The English speaking guide we were promised, was of course nowhere to be found. Luckily our Spanish skills were by now good enough to understand the important bits and on occasion even make some smalltalk with Jesus, who turned out to be not just a nice guy but also fitter than a marathon runner.

We started our trek at about 12 in the afternoon, heading towards the snow-capped mountains in the distance. The trail went slowly upwards for about 10 kilometers (although this would turn out to be the most level part of the hike), before we descended on the other side of the mountain for another 9 km towards our campsite Chiquiska, near the Apurímac river. On the way we passed small farms and little huts with corn on plastic sheets lying next to them in the sun. The corn will be fermented to create Chicha, the local beer that dates back from Incan times. We stopped at one of the huts and bought a whole jug of the grainy, fizzy and little bit sour beer, for only 1 sol (about €0,30).

Jesús told me that they also have a stronger variant made out of fermented sugar cane. As luck would have it, we met a local farmer with his mule and his wife coming up the mountain. He happened to have some sugar cane chicha and generously offered it to us. We drank some of the stronger chicha, made a bit of small talk and then went our separate ways, them continuing up the mountain and us going down towards the river.

We arrived at our campsite just before dark, to discover that we would be the only people sleeping there. We were camping at an oasis near the river, surrounded by banana and papaya trees. Facilities were basic open outdoor showers (also known as hose pipes) and a hole in the ground as a toilet. We had a fresh papaya straight from the tree, before we unpacked our stuff and sat down to dinner. We were amazed at Jesús’ ability to prepare a hearty and tasty three-course dinner in the wilderness with his limited supplies and no electricity.

That night, Jesús and Milano slept out in the open on wooden benches as Manuela and I went to bed in the relative luxury of our sleeping bags in a small domed tent.

4 September, Choquequirao

For the second day, we had to cross the Apúrimac river and hike back up the mountain on the other side. This turned out to be the toughest day of the hike, and Manuela got very angry at me for all the time anticipating the end around the corner only to find another steep zig-zag going further up the mountain.

At about noon, after 5 or 6 hours of hiking upwards, we reached a relatively level clearing with a small village of less than a 50 inhabitants. The houses were made of mud and there was no electricity or running water except for a mountain stream from which the locals channeled water using hose pipes.

We ate lunch under a roof next to one of these houses and then tried to rest for a while before we would have to continue trekking another 2 hours to the lost city of Choquequirao.

Choquequirao is situated in an absolutely stunning area and its clear that the Incas had an eye for choosing spectacular locations to build some of their cities. The beauty of these locations convinced them of their sacredness, and on a small hill next to the city they made a flat and round clearing (like a heli-pad) where they held their rituals. Standing in this clearing, with a 360° view of the mountains on all sides and the river cutting through the middle, was a very special experience that made the tough hiking all the more worth it.

Jesús and I then walked down towards the terraces on the other side of the mountain to look at the llama pictures made in the rock walls. The original way down was a long series of stone steps that would have to be hell to walk back up again. Luckily there is now also a hiking path going back up.

As mentioned before, Choquequirao is only partly excavated, and it was possible to go into the bushes next to the excavated buildings and see more walls, houses and other structures hidden between the thick vegetation. There were only 4 other tourists at the site that afternoon and we could roam the whole area undisturbed and entranced by the magical surroundings.

We stayed as long as we could and then started hiking back to the previous village were we would overnight. The hike lasted well past sunset and into the night and Manuela remarked on how we started the days hike before sunrise.

That night, Jesús prepared a delicious soup that he claims is a family tradition and we lay on the grass to look at the stars as we did the previous night down by the river. They were more vivid than I’ve ever seen before, and that includes star-gazing in the desert in Namibia and near the observatory in Sutherland, South Africa.

5 September, Choquequirao

On the third day, we had to hike back down the mountain, up the other side past our old camping site to another site further up. On the way, we stopped at a small mud house and shop were Jesús and I drank two vases of locally brewed Chicha beer before continuing onwards.

That night we camped next to a small two room mud house, were a local woman and her husband lived. We had dinner inside the bigger room, which functioned as a living-room, bedroom and kitchen in one, as the wife tended to an open fire and heated up some left-overs for the just returning husband. We were humbled by the simple and secluded lives that these people live, up in the mountains and far away from the trappings of modern society.

That night we had a special show held for us by the fireflies flickering neon green in the darkness and spread out all over the mountain.

6 September, Choquequirao

We had half a day of trekking left before we reached the town of Cachorra again. We left early and hiked up the mountain in thick mist that prevented us from seeing more than a few meters ahead. It was about 6 difficult kilometers up the steep mountainside, before we reached the top and hiked another 10 level kilometers to town.

The last four days’ hike was tough but definitely worth the effort. We visited a lost city located in a spectacular location which left us amazed and inspired. In the process we learned a lot about ourselves, about rural Peru and the difficult yet simple life up in the mountains.

10 September, Traveling to Machu Picchu

After spending a few days resting in Cusco again, we prepared to leave to the famous Inca ruins of Machu Picchu. For many travelers, Machu Picchu might well be the main attraction of their South American trip. I remember being filled with wonder at the prospect of a continent filled with spectacular and mysterious ruins left by enigmatic empires such as the Incas, Aztecs and Mayans.

Machu Picchu itself was every bit as magical as I imagined it to be, but the profiteering, bad service and lies peddled to us by the agency we trusted to get us there, left quite a few fantasies shattered. Reality however sometimes has the effect of tempering a wild imagination.

Not that it would have necessarily have been better going with another agency. Cusco is filled to the brim with tour operators. Any idiot can set up a table with pictures of Machu Picchu and other attractions and start selling tours. And many do. The tours they sell are of course passed along to other operators, and in the process all accountability goes out the window.

Add to this the fact that the only sanctioned way of getting to Machu Picchu (with the exception of hiking for 2 to 3 days) is paying about US$100 for a one way train trip of little more than 100km, and we almost considered leaving the whole thing as it felt a bit too much like a rip-off.

Instead, we thought we found a way to beat the system. A small operator (one of these desk jobbies as mentioned earlier) offered two-day trips to Machu Picchu via bus for about US$115, whereas a normal tour would cost about US$180,

We left the morning very early, 4:30am to be exact, in a small tour bus excited to be on our way to the famous site. The first 3 hours driving turned out to be uneventful, until we reached the top of a mountain pass outside of Cusco. Turns out that we were bundled together with another group who booked a more adventurous tour and that they would now cycle down the mountain while we slowly follow 50m behind in the tour bus. For the next 2 and a half hours, we crawled down the mountain at about 20km per hour with the group of 20 or so cyclists ahead of us. Of course, our “tour agency” completely neglected to impart this little piece of information to us.

After coming down the mountain (with its comparatively good road), the day got progressively worse. Due to delays caused by the cyclists, bad planning and numerous roadworks on the horribly uneven and narrow gravel road through the valley, we reached the town of Santa Thereza for lunch at about 16:00pm with not a single apology or explanation made.

Eventually, after getting into a smaller bus and then afterwards also a train ride, we reached Aguas Calientes, the small tourist town close to Machu Picchu. It must be the most fake and kitsch town we have encountered on this whole trip. Perhaps ever. Everywhere are plastic Incas, kitsch paintings of muscled Incas (abvious rip-offs of 1980s fantasy art such as by Boris Vallejo), and of course someone trying to get you into their (overpriced) restaurant or hotel. Our tour guide on the other hand, couldn’t be bothered to even remove his headphones to properly talk to us.

As mentioned, we left the morning at 4:30am, and arrived in Aguas Calientes at about 8:00pm. Keep in mind that Aguas Calientes and Cusco are only about 112km (in a straight line) apart.

10 September, Machu Picchu

We got up at 3am that morning in order to be among the 400 people who would be allowed to climb Huayna Picchu, the hill next to the lost city.

Walking around the ruins of Machu Picchu made all the effort in getting there worthwhile. The ruins and their location are simply spectacular and not even our half-useless guide could spoil the occasion.

On the way back, even though we were squeezed into a small minibus, the driver decided to make some more cash on the side and drove around Santa Thereza offering a trip to Cusco. Eventually three more locals entered the minibus as well and we continued to Cusco. The drive back was even more horrible than before. Our driver drove carelessly and very dangerously along the mountain passes, so much that we repeatedly had to ask him to drive more carefully. He didn’t give a rat’s ass of course, and continued to speed and cut corners.

After about 8 hours of driving, when we got closer to Cusco, he suddenly veered off the tarmac and started driving on a rough and uneven dirt road. Turns out that there was some kind of roadblock ahead which he was trying to avoid. Why the hell would he take a big detour on a rough gravel road to avoid a roadblock? Of course he declined to tell us, so we could only guess.

Long story short, Manuela and I were so relieved to be back at our hostel after a frightening and claustrophobic journey of about 9 hours, that we went to the supermarket and bought a whole lot of comfort food. Crisps, chocolate and red-wine. We celebrated still being alive. Moral of the story, don’t take the “bus” to Machu Picchu!

11 September

We went for a massage. Like everything in Peru (and even more so in Bolivia) even though the masseuses really try hard and I can only respect their work ethic, they somehow just lack those finishing touches of professionalism. The CD player doesn’t work, they need to move the tables when a new client comes in and the one girl burnt Manuela with a hot stone. All in all, it was only €6, so still good value for money.

Manuela also received coca cream for her face. This reminded me of an earlier prediction I made to her and which I’ll mention here as well. I predict that within 5 to 10 years, a coca craze or fashion will enter the Western market for especially cosmetics and toilettries. Expect coca toothpaste, facial creams, shampoos and the like. Watch this space ;) Similar to the herbal toothpaste craze of a few years back…

13 September, Leaving to Manu National Park

We had already resigned ourselves with the idea of not going to the Amazon during this trip, when Manuela spotted a flyer in Cusco advertising a work and travel opportunity in the Amazon basin in Manu National Park. Originally we considered going to Belém in Brazil, or Iquitos in Peru, but both turned out to be very expensive and the stories of unscrupulous and non-ecofriendly tour operators didn’t impress us much either. This opportunity, presented by an agency called ManuPeruAmazon, seemed to be different however, and after chatting with them at their Cusco office, we signed up for a week of camping in the jungle to help out with volunteering work while also taking part in hikes and boatrides.

Manu National Park is an area which is so difficult to reach that the Spaniards actually never invaded it. They attempted at different times resulting in the loss of many men forcing them to eventually give up. Uncontacted tribes still roam the vast and unreachable interior.

ManuPeruAmazon is the project of Oliver Innocente, an Italian expat, and his Peruvian wife Carolina. Oliver has been living in Peru for about 20 years, most of them spent in the jungle.

After taking a 7 hour bus-ride from Cusco into Manu National park, we arrived at a small town where Oliver was waiting for us. His right hand was bandaged after been stung or bitten by something, most probably a spider. His right thumb, where the bite took place, was completely red and swollen up to more than twice it’s normal size, while his fingers and wrist were also still red and swollen. On the spot where the bite took place, was a large gash with raw flesh peeling out. And this was already 8 days after it happened. This being the jungle, Oliver didn’t have access to medication, not that he wanted any really, and he made exclusive usage of local medicinal plants.

We drove for another half an hour to a small river port town and Oliver showed us the biggest pea I’ve ever seen. The pips are covered in a white fluff, which was quite tasty.

From the port we took a motorized wooden boat on the Madre del Dios, an important tributary of the Amazon river, downstream into the jungle and towards our camping place. It was an amazing experience.

The campsite itself was literally 4 small domed tents and two reed and grass structures to provide shade. For the next 5 days we would have to wash in the river and digg a small hole every time we needed a toilet. This was exactly the experience I was hoping for, something more basic and away from the other tourists and tour guides who treat them like wallet owning cattle.

Oliver was given this particular plot of land (more than a 100 hectares) by the Peruvian government only two months ago, to facilitate its rehabilitation after being damaged earlier by a large landslide. He already had grand plans for reintroducing different plant species, creating a botanical center and also facilities for students and scientists to come and study the jungle. About 60 meters up, there was another campsite, with a thatched roof where Carolina would prepare food. We were convinced relatively early that Oliver was genuinely concerned with protecting the local environment and that he cared deeply for the jungle. He spoke with great respect of the local natives who still live in the Amazon and about their beliefs and knowledge of the environment.

This first nigth at the campsite we saw little glowing green eyes, moving between the plants and the darkness. These irridescent green dots were on the backs of so-called click beetles, named after the clicking sound they make when you capture them. This, coupled with the everpresent symphony of the jungle, countless other beetles, crickets, critters, bugs and birds, created a truly magical first night in the Amazon.

14 September, Working and hiking in the Amazon

On this day we got up at about 5 in the morning and went to the clay walls where blue headed parrots licked clay to ingest minerals in order to counter the toxins of some of the unripe fruits they eat.

Our campsite was right next to the river and there were still pools of water near the tents. Numerous butterflies of many different shapes, colours and sizes asssembled near these pools. Some were very large and yellow, others were adorned with intricate patterns of a beautiful dark blue, while another kind was a kind of unearthly neon green. They contributed a whole bouquet of different colours to the jungles’s palette of infinite shades of green.

The path towards the topmost campsite was steep and at certain spots difficult to climb. One of the first things that needed to be done, was to build steps on these steep parts and Tony and I volunteered to start working on this. At the upper campsite, there was also a small nursery, made out of halved bamboo reeds, where Manuela and Sofia helped to clean the reeds and plant new seeds.

Using shovels in the damp sand, we relatively quickly created the initial steps, but in the jungle humidity and hot sun we sweated like never before.

After doing volunteer work in the morning, we ate lunch, relaxed for a whlie and then took the boat downriver to start a four hour hike starting on the other riverbank. We hiked for about an hour and a half, through swamps, under gigantic leaves (called Elephant ears) and under very tall trees where we saw some squirrel monkeys. Eventually we reached a small lake bordered by an imposing wall of thick impenetrable jungle up to 40 meters high. We took a so-called “catamaran” (basically just a wooden barge) on the river and for the first time, we were so quiet, slowly floating on the still water, that we could really appreciate the countless birds on the lake, undisturbed by our presence. On the one side of the river, a Capybara was relaxing and everywhere where the Huatzin (German “Schopfhuhn”) and other birds.

We arrived back at the campsite when it was already dark and becoming clear that a thunderstorm would soon be breaking out. For the last half an hour we saw impressive displays of lightning interspersed with loud ominous thunder, betraying the massive size of the clouds moving above us. The wind was so strong, that first Tony’s and Sofia’s and then our tent collapsed and we hastily had to pack our stuff and hike away from the river bank to the upper campsite to avoid the danger of a rising river during the night.

As we reached the upper campsite, the wind picked up in strength and it started to rain. The strong wind teared holes into the sides of the large tent housing our dining area. The winds were by now so strong that we were at risk of falling trees and more than once we had to run outside and stand in the relative safety of a clearing. The branches of some trees were now breaking off, and once I had a narrow escape as a large sharped edged branch came flying past me as I came out of the tent. Eventually the wind started to quiet down, while the rain continued through the night and into the next morning.

15 September, Tabacco ceremony

It is quite interesting to experience the rich, lush and fertile jungle, while simultaneously also it’s hostility. Sandflies and mosquitos are an everpresent nuisance, biting the skin and causing itches and sores. One tree’s bark causes irritation on the skin while many others are covered in dangerous thorns. All the worms we encountered are covered in large hairy spikes that are not to be touched, while poisenous plants and fruits abound. Yet, we drank water from a local stream, without the need of even a purifier, while the tap water of most of South America is off limits.

Previously Thomas, a polish volunteer who was already at the campsite when we arrived on the first day, went out to pick some wild jungle tobacco and let it dry out in the sun. The local people use the tobacco in a ceremony as a medicinal plant, during which the participants drink boiled tobacco water, together with about 5 litres of water, until they throw up, thereby cleaning and purifying their stomachs. Thomas offered to prepare the tobacco for us and all four of us decided to take part in such a ceremony. It was an interesting experience and each of us threw up in the end. I felt weak and tired directly after the ceremony, but a little while later I regained my strenght and headed down to continue working on the steps.

Coming down the steps, I saw a big Tayra otter, directly in front of me. It snatched what seemed to be a banana left there by someone and then disappeared into the bushes. It was amazing, every time leaving the camp, you could discover animals, birds or insects of which you never had an idea before,

16 September

The following night, we moved our tent for the third time in three days. This time, next to a very tall tree near the edge of the campsite. Coming down this tree was a long path of leaf-cutter ants, each one carrying cutling of leaf in its pincers. The path extended down onto the ground and then away into the jungle for about 15 meters before it disappeared into the dense undergrowth and I couldn’t follow it anymore. This was a very nice surprise for me, having grown up playing with (torturing) the large red ants on my grandfathers farm and since developing a fascination for these industrious little creatures. The Amazon jungle seems to be an ant lovers paradise.

During our hike two days ago, Victor, our Peruvian guide, showed us a specific tree filled with very aggressive fire ants. These ants live in symbiosis with this tree and actively protect it from any invaders, for example such as the leaf-cutter ants. The other trees in the jungle are fair game for all kinds of insects, ants and mammals, but this specific tree is left unscathed.

Then there are the large (2 to 3 centimeters!) black solitary ants. They do not belong to any colony and are quite dangerous, with a bite that causes a nasty fever. In addition there were even larger reddish ants, whose backsides the natives cook in oil, and all kinds of flying ants.

17 September, Leaving to Cuzco

Our last morning in the jungle was spent carrying stones to cover the steps we built. Afterwards we all went for a fun and refreshing swim in the the river, letting the current take us to another riverbank downstream. Five minutes out of the river and we were again drenched in sweat as we packed our bags for the long trip back to Cuzco.

The bus smelled like sweat and coca leaves, and after arriving at the first roadblock, we understood why. There was a woman in in traditional clothes carrying a large bag, and when the police officer entered the bus she stuffed it under the large jumper she was wearing. As he passed her, she pulled it out and threw it across two rows to a friend on another seat, who hid it under his seat. The police officer took a quick look and left without any arrests, but more drama was to come as we entered Cusco.

The second roadblock, about 4 o’clock in the morning, contained many more police officers and they were also quite a bit more aggressive. Four officers entered the bus and started searching through people’s bags. The woman who previously managed to hide her bag from the police wasn’t so lucky this time. Three officers stood over her and demanded to see the bag she was trying to hide. She just flatly refused however, and when one officer tried to take the bag, a pulling match broke out between her and the police. For what seemed like 20 seconds, the police officer tried to pulll the bag away from her as she clung to it for dear life. Ultimately, it was an quite entertaining if futile effort on her part to prevent them searching through her bag, and eventually she was arrested.

When we eventually left the bus at the bus station, about 5 o’clock in the morning, I noticed that the whole floor was covered in coca leaves. Apparantly, Peruvians are allowed to transport a certain quote of coca leaves on them, but the problem on the bus was that the people from the jungle were trying to transport coca above this quota.

18 September, Reggae party in Cusco

As this was our last night together with Tony and Sofia, we decided to celebrate and go out. We partied the whole night through in a club where a Reggae band played all the classics and time flew by.

We were definitely sad to have to split up with our new friends again as the four of us got along very well which can be rare while traveling.

Something for other travelers to take note of in Cuzco is a small vegetarian restaurant in Calle Ruinas named La Encuentro in Cusco. They offer the best lunch deal in town between the many tourist rip-offs (salad buffet and 2 courses of food plus a drink made, all made of organic ingredients for 6 Soles).

20 September, Leaving to Chile

For our trip to Tacna, on the border with Chile, we finally managed to book a Cruz del Sur bus, recognised as the best bus service in Peru. Not only that, but we also managed to get the mythical full cama bus with fully reclining seats. We’ve almost given up hope of capturing one of these elusive buses in South America, despite persistent rumours that they do actually exist.

During our stay in Copacabana, we went to a bus agency with large TurPeru (another good bus company) posters outside and booked what we thought was a full cama, TurPeru bus from Copacabana to Arequipa. We got however a crappy local bus with semi-reclining seats and a terrible driver schooled in the proud Peruvian tradition of bad bus drivers.

This time, things were different however. We arrived at the terminal, and there it was. An actual Cruz del Sur full-cama bus waiting for us. This was the most luxurious bus we took so far, with plasma screen televisions and large spacious leather seats, pillows, blankets, food, drinks and a clean bathroom. The bus is even monitored via GPS to prevent speeding, which is a welcome restriction to your average Peruvian driver’s natural impulse.

21 September, Tacna Peru to Arica in Chile

We drove 9 hours to Arequipa and then switched to another, less luxurious bus to Tacna. The landscape outside of Tacna is completely barren. This must be the place where they shot the Martian landscape scenes in movies like Total Recall. There is absolutely nothing growing in this part of the Atacama desert and it was quite a contrast to the lust Amazonian jungle in Manu national park. The air that finally manages to cross the high Andes is by then completely depleted of water. This results in the Atacama desert, the driest desert in the world.

Arriving in Tacna, we took the local bus to Arica for 10 Soles per person. Half the price that the Taxi drivers (apparantly Collectivo) at the bus station wanted. The bus was fine and crossing the border to Chile without incident as well.

Coming into Arica, we were stunned by the amount of Chilean flags we saw. Almost every single house and many cars had a Chilean flag (sometimes two) hanging on a mast. We weren’t sure what to make of this spectacle.

Arica has a pedestrian precinct on Avenue 21 de Mayo and it was interesting to see all the fashionable clothes shops, department stores and other consumerist havens which people in many countries take for granted but were quite scarce in Peru and basically non-existent in Bolivia. Almost all the shops were decorated with Chilean colours and flags and it was here that we finally realised that Chile was busy celebrating their 200 year independence anniversary.

21 September, Tacna Peru to Arica in Chile

Just outside our hotel is a mural with four graves, one whose cross throws the shadow of a swastika. One of the graves bears the inscription Augusto Pinochet, 1915-2006. Fuck, a reference to Chile’s brutal former right-wing dictator, whose regime was responsible for the deaths and dissapearances of tens of thousands of dissident Chileans. Shortly after coming to power after a military coup in 1979, he rounded up all the leftists and dissidents he could find and killed and tortured them in the local Santiago football stadium.

Incidentally, during the 1980s, he also had narrow relations with Israel and the Apartheid South African government. Chile is still coming to grips with the pain caused by his regime, and it was interesting to see this already on our first day here.

23 September, Arriving in San Pedro de Atacama, Chile

We were quite disappointed with San Pedro de Atacama. The town itself looks quite nice, with desert adobe style buildings and a bit of an alternative vibe, but this was all spoiled by the aggressive tour operators, accommodation, restaurant and bar promoters and sky high prices. The moment we got out of the bus, we were surrounded by 5 or 6 different people trying to sell us their accommodation. For the rest of our stay in the town, we would not be able to go outside without having to fight off various different sellers and promotors. Definitely not our idea of a nice time. The Lonely Planet mentions that San Pedro de Atacama is a town in which the tourism industry developed too rapidly (with all the negative aspects that this brings), and that is exactly how it appeared to us as well.

It was sad and a bit strange to see almost no authenticity in the town. We encountered almost no locals who weren’t trying to cash in on tourism. There were for example no school children walking home, or any other activity by the locals that might represent some kind of normalcy unaffected by tourism, except the sunday church service.

25 September, Bicycle ride to the valley of the moon

After an aborted attempt at cycling the 12km out of town towards the Valle de la Luna (valley of the moon) the previous day (we left too late to catch the sunset), we decided to try again one more time. To make matters worse the previous day, my bicycle broke and we had to walk back to town. So, instead of going to KMZ, were we were before, we went to another and even cheaper shop, where we rented the bikes for half a day at C$3000 (€4,50).

We biked out of town for about 5 kilometers, until the entrance to the valley of the moon, from where it was another 7 kilometers on a relatively flat gravel road that curves to the right and then ends after a ramp up and into the canyons. We weren’t the only ones going there, but just about the only ones going by bike. Quite a lot of buses and cars passed us by as we huffed and puffed cycling against a strong headwind.

We eventually reached the lookout point and watched the sunset, still being relatively stressed after a negative hostel experience earlier in the day. The landscape is quite pretty and we felt that renting a pair of bikes to get there was probably our best option. San Pedro de Atacama offers quite a few scenic places to which the many tour agencies would be quite happy to drive you, but the tours (as with most things in the town) are all overpriced in our opinion.

Concerning our negative experience earlier in the day, suffice to say that we would NOT recommend staying at Corvatsch hostel. The staff were all rude and unfriendly and there was a dark cloud of negativity hanging over the place from the moment we arrived. The guy who runs the place is hands down the biggest asshole we met in Chile (people in general actually being quite friendly and nice) who even physically threatened a fellow roommate of ours after he objected to his female friend being shouted at. Eventually, our whole dorm of 6 people demanded our money back, packed our stuff and left for another hostel.

26 September, Back to Bolivia and Salar de Uyuni

During our last stay in Bolivia, we were prevented from going to the famous Salar de Uyuni, the largest salt flat in the world, by discontented miners and other Bolivians from the Potosi district. They had blockaded the roads outside of Potosi as well as the small town of Uyuni with stones and dynamite, effectively imprisoning the whole town populations and preventing anyone else from getting in. The story made international headlines because a group of about a hundred tourists were stranded at the Uyuni airport while the food stores of the town ran low.

We waited a few days extra at Sucre, hoping that the dispute might be resolved soon, but eventually realised the gravity of the situation and went to La Paz instead. You might expect that as the situation worsened, the tourists in Bolivia might be dissuaded from trying to get into Uyuni, but for some of them, it did exactly the opposite. People paid outrageous sums of money for private jeep transfers all the way from La Paz, to try and circumvent the roadblocks, just about the same time as the stranded Uyuni tours tried to escape via small chartered flights. At least one time they were unsuccessful, with protesters laying rocks and dynamite on the airport runway.

So, here we were about a month later and things seemed to be back to normal in Uyuni. This time we would enter from the Chilean border (close to San Pedro de Atacama), and drive through the Bolivian Altiplano (high plateau) desert with it’s harsh cold weather towards the salt flat itself.

On the first day, we drove with the Licancabur volcano in the background, towards sulfur spewing geysers and hotsprings where we took a dip in the 35° water. We then continued past green and turquise coloured lakes towards the spectacular Laguna Colorada (red lake), where we would overnight at a house nearby.

For the whole day the wind was blowing with incredible force, blasting sand into your face and eyes, while temperatures hovered just above freezing point. Nevertheless, Manuela and I decided to take a walk towards a lookout point near the red lake. The wind was behind our backs making things a bit easier as we walked through the small tough gold-coloured bushes (basically the only vegetation in the area) past a herd of Vicunyas (cousins of the Llama) on the one side and pink flamingoes on the other side.

Walking back towards our lodgings was a different story however. The wind was blowing against us with full force, freezing our exposed skin and hurting our eyes. From time to time we would turn around and walk backwards for a while, before we would get tired of the slow pace and make another attempt at trudging against the wind. This might have been more frustrating, if not for the fact that we were enchanted by the harsh but beautiful landscape around us. Parts of Laguna Colorada’s waters were a truly blood-coloured red, creating an impressive contrast to the gold coloured wind-swept hills.

27 September, Salar de Uyuni

All the tour operators in Salar de Uyuni make use of Toyota Land Cruisers, as they are the only vehicles able to handle the rough terrain coupled with the extreme height above sea-level (well above 3500m up to more than 5000m).

On the second day our old Land Cruiser shook and bumped its way past more coloured lakes, vicunyas and pink flamingos. We had a flat tire just after lunch, and then another one near the end of the day, which left us stranded and hoping that another tour group would soon pass and help us out.

In the end, we were very lucky and only had to wait about 15 minutes before another Land Cruiser arrived from whom we could get a spare wheel. We slept the night in a salt hotel, a bit of a gimmick, but still kinda cool.

Flavio, our excellent guide and driver, spent most of the evening patching and fixing the two burst tires. Before we embarked on the Uyuni tour we were a bit concerned, having heard numerous stories of drunk and sometimes abusive drivers. Luckily this didn’t turn out to be a problem for us and we were very impressed with the one that we had.

We booked the tour through a Chilean tour company called Tierra Mistica, and could recommend them to other travelers.

28 September, Salar de Uyuni

On the final day of the Salar de Uyuni tour we finally reached the actual salt flat. The salar is actually the remnant of an ancient prehistoric inland sea and is almost completely flat. In fact, it’s probably the best driving surface (if not actually a road) in all of Bolivia!

In the middle of the salar lies Incahuasi (meaning house of the Inca’s), an outcropping refered to as the “island”. The island is covered in large cacti, and still has the remains of the ancient coral reefs that grew there when the whole place was under water. It must have been an awesome place for scuba diving.

You can easily lose your perspective in the Salar, making it possible to take all kinds of silly photos. I broke off a piece of salt from the ground and put it in my mouth. It was salty, but not as extreme as a clump of table salt would have been.

29 September Tupiza, Bolivia

We arrived in Tupiza, the previous night after hitching a ride with a driver from Tupiza tours who also just finished a Salar de Uyuni tour. He dropped us off at Hostal Horizon Colorado and after some unsuccessful haggling we settled on a private room with three beds, a television, private bathroom with mirror and a waste basket (almost always lacking) for less than half the price we paid for the shitty 6 bed dorm in San Pedro de Atacama (about €7,40).

Next to our hostel is a salon de belezza (beauty salon) and, in typical Bolivian fashion, with a nameboard that looks like it was designed (I use the word quite liberally) using Microsoft paint and some standard fonts together with a few images of women collected from the web. One of the “women” actually being a picture of a Barbie doll. Whatever happened to subtlety?

From the getgo we felt quite happy and at home in Tupiza and despite our earlier reservations, we were enjoying it to be in Bolivia again. So far we’ve managed to avoid any buses and the people even seem a bit friendlier than before. Either because our Spanish skills are improving or perhaps because Tupiza is just a friendlier town. It’s a pleasure walking around town again, visiting the local markets with their wide variety of fresh produce, cheap clothes from China, offerings to Pachamama and almost anything else that the townsfolk might need.

It’s also one of the more authentic towns we’ve come across so far. Especially compared to San Pedro de Atacama, where it feels as if every single inhabitant is greedily contemplating how to make (more) money out of tourism. In Tupiza, the locals go about their daily routines (like selling llama fetuses for example), largely oblivious (or at least tolerating) the odd gringo to come across their path. So far we’ve spotted about 12 other gringos after criscrossing the entire town. Gone are the kitsh tourist shops, restaurant and bar promoters pestering you to enter their establishments and of course the swarm of tourists who would otherwise with their presence enable all these things to happen. It’s like a breath of fresh air.

30 September Horseback riding in Tupiza, Bolivia

Tupiza really has some amazing surroundings, perfect for donning a cowboy hat and heading out on a horse. We rented two Argentinian horses and set off with our guide on a 7 hour ride out into the area.

Turns out, our “guide” was a thirteen year old boy. Perhaps this was not strictly legal, and it would be unthinkable in Germany, but he seemed to know the area and how to ride a horse and in Bolivia that already counts for a lot.

1 October, Leaving for Córdoba, Argentina

After our horseback-riding adventure of the previous day we were ready to leave to Córdoba in Argentina. This would mean 3 hours in a local bus until the Bolivian-Argentinian border, and then another 18 hours in an Argentinian bus until Córdoba.

We went to the bus station two days before to find the bus schedule and perhaps so long book a bus to the border, but when we arrived there we saw an agency for a bus company called Balut, whith large photos of a big gold bus with the letters Cama Gold written across it. The other bus agencies had paintings of buses if anything at all. Not exactly something that instills confidence.

We entered the little Balut office and asked the lady some questions. “Yes the bus is full-cama (meaning luxurious and fully reclining) and looks like the gold one on the picture”, “Yes, lunch and dinner will be served on the bus.”, “The bus will leave from the border at 19:00 and will take you through border controls”, and so on. Wow, this seems great!

Well, aftrer taking another dilapedated old local Bolivian bus to the Villazon, we were still optimistic and looking forward to the gold coloured luxury full-cama bus that will be taking us from there to Córdoba.

We enter the Balut office in Villazon, see more pictures of gold Balut buses crossing Argentina and ask for our tickets. Turns out however that just about everything we were promised before was bullshit. The bus isn’t full-cama, isn’t gold and looks nothing like on the pictures. It will leave later than originally told and there won’t be any meals served on the bus. And lastly, it doesn’t take us through border controls, we have to walk there (or take a taxi) ourselves. Ah, good old Bolivia, that was again, what people warned us of before!

October, Cordoba in Argentina

It’s alread October by now and we have since left Bolivia for Cordoba and Buenos Aires in Argentina.

Cordoba is a fascinating and culturally rich city with seven (7!) universities. We spent a lot of time just walking through the city center, checking out the bakeries, coffee shops, malls and other amenities that we’ve since learned to appreciate.

Roberto Allende and the guys from Menttes invited me out for dinner and I was served the largest steak I’ve ever seen. Covered with two fried eggs for good measure.

Roberto’s lovely wife and their 2 month old son also accompanied us, giving Manuela the chance to hold and play with the baby, while we discussed frameworks.

The next night we met up with the Javier Mansilla (whom I met earlier at the Budapest Plone conference) and his colleagues at Machinalis and enjoyed some Picadas with Argentinian beer and more wine.

We agreed that the Plone ties between Argentina and South Africa need to be strengthened and I intend to return home with just that message! Thanks for the friendly hospitality from all the Plonistas in Cordoba!

October, Buenos Aires Argentina and then off to Cape Town!

We had high hopes for Buenos Aires after hearing so many good stories from friends and travelers and I’m glad to say that we weren’t disappointed in the least. Tango in the streets, more good steak, friendly people, good infrastructure, great nightlife, and an old and interesting city center; Buenos Aires really has a lot to offer.

The Argentinians close all the shops around lunch until about 16:00 to allow for a fiesta, which took a bit getting used to as we tended to start going out to town at around lunch each day. Because of this, the Argentinians also go out much later. We would arrive at a restaurant at 21:30 and be the only customers there. Then, at about 22:30 the locals arrive and at 23:00 it’s full! In Germany many restaurants would have by then already closed the kitchen.

We have three days left in Argentina before flying to Cape Town and starting a new chapter of our lives in South Africa. I am still enjoying Buenos Aires, its culture of Gauchos (Cowboys), Tango and late night dining, but also filled with excitement at the prospect of going home for the first time in two years and at seeing old friends and family again.

From myself and Manuela, who had contributed lots of insights, corrections and stories to this blog, I would like to say, thanks for reading and “Adiós Chicos!”

South American travel diary, Part 3. Aug 2010

Sun, 08 Aug 2010 00:00:00 GMT

In August, we continued traveling through Bolivia and onwards to Peru. Along the way we saw dinosaur tracks, a military parade celebrating Bolivia’s independence, had some really good food in Sucre, mountain biked down the world’s most dangerous road in La Paz and visited Inca ruins at the Island of the Sun.

1 August, Traveling from Samaipata to Sucre

We began the month traveling by night bus from the lovely small town of Samaipata to Sucre, the constitutional capital of Bolivia.

We stopped in the middle of the night next to a church in a small unknown town, and everybody streamed out of the bus. Obviously this was a restroom break, but in Bolivia that implies something else than it would in most other countries.

Most passengers didn’t even walk five meters before unfastening and pissing right there in the street between the bus and the church. A proper restroom was or even a kios was nowhere to be found.

About 30 meters away, a family or group of friends were sitting around a table in a little restaurant. For them, it must have been a sight to behold; an entire bus full of people (men and woman) appearing in their field of view, pissing in and around the street.

With an area of 30 meters around the bus covered in dozens of streams of urine, everyone promptly returned to their seats and the bus continued to Sucre as if nothing funny ever happened.

In hindsight, we were lucky that the bus stopped at all. We have been warned that the busses travel for hours and hours without stopping for breaks and one traveler even told us that she suspected people of urinating in their seats.

Little did we know that urine (and urinating) would become a recurring theme in Bolivia. Men and boys (and sometimes woman) would piss at the side of a traffic filled street in the middle of town and in broad daylight. That awful smell is omnipresent and a not so gentle reminder that everything outdoors is fair game to be used as a lavatory, and not just by the roaming packs of dogs.

2 August, Sucre: Exploring Cólon

Manuela, our friends Tony and Sofiaand myself arrived early the next morning in Sucre and took a taxi to our hostel.

We headed off to the nearby Christobal Colón street, to find a cheap place for lunch. There were a few Plato Familiar restaurants in the area, with not much separating them concerning outward aesthetics, so we decided with our empty guts and chose one at random. The eatery was family owned and the children were also involved, either taking orders or carrying plates of food. The menu is set. Everyone eats the same lunch, with the advantage being that it is incredibly cheap. Lunch consisted of a soup (described by Sofia as reminiscent of Minstroné), a main dish consisting of grilled meat with rice and a small salad, and an orange for desert. All for 9 Bolivianos per person, about €1.

3 August, Sucre’s markets

The city center of Sucre is actually quite pretty, with white colonial buildings backdropped by large mountains in the distance.

What is really remarkable and noticable about Sucre, is the everpresent juxtaposition of cultures and generations. The old and the new clash on almost every street. I would walk down the street passing an old woman in traditional clothes selling nuts, and then a few meters later pass an internet shop filled with young kids playing World of Warcraft.

Travelers looking for something more authentic than a South American varitation of western consumerism, will inevitably enjoy Bolivia. The woman in particular all wear traditional outfits consisting of Bowler hats, braided hair down to their hips decorated with tassles at the ends and colourfully weaved ponchos, scarves and bags. Parts of the outfit, the hat in particular, can be attributed to Spanish influences, but the end effect is unquestionably and endearingly native.

Going shopping in Sucre (and Bolivia in general) is an experience in and of itself. Supermarkets are few and far between, and Bolivians instead buy and sell almost anything you might need at the local market or on the street. The main market in Sucre is quite big and well ordered. The woman selling cosmetics, soap and beauty products are all clumped together in one area.

Another area is reserved for stalls with food products (pasta, rice, canned food, sauces etc.), yet another section contains meat sellers, then fruit sellers, cakes, sweets, toys and on and on. The whole area bustles with activity and shopping seems to be a much more social experience, with the inevitable small talk and bartering as one moves from stall to stall to buy the day’s or week’s essentials. You cannot walk through the market without people calling you over to try and sell their warres. Their sales call is usually a three times repetition of the particular item they want to sell. Such as Mandarina!, Mandarina!, Mandarina! when selling fruit, or even La Paz!, La Paz!, La Paz!, when selling tickets at the bus station.

The walls of the one courtyard inside the market building are lined up with small fruit stalls, with women behind them selling smoothies and fruit salads. I ate the best tasting fruit salad of my life at one of these stalls (for about € 0.50). We’ve seen similar fruit salad stalls in La Paz, but they weren’t as good. Highly recommended for anyone visiting Sucre.

In addition to the local market, there are also turistico markets, selling bags, clothes, jewelry and mementos to the tourists. One of the travelers favourites, is the chompa, the Bolivian equivalent of a jersey or jumper (from where the local name comes), complete with llama patters and often being associated and worn by the Bolivian president Evo Morales, much in the same way that Nelson Mandela wore his colourful shirts. Other loved items are colorful bags and handbags, leather wallets, caps and Incan/Peruvian beanies. Almost every backpacker will buy at least one traditional clothes item, which the older people all still wear, while many youngsters try their best to look urban and “western”.

Another oddity in Sucre (at least for Bolivia), is the businessman wearing the western style suit and tie. This is quite a contrast to the countryside, where the people live a life of subsistance farming with their donkeys, chickens and cows.

4 August, Sucre’s dinosaur tracks

We made a visit to the dinosaur tracks outside of Sucre, home of the largest collection of dinosaur footprints in the world. The footprints were discovered in 1994 by the workers at the cement quarry operating in the area. Up to 5000 prints from about 250 different dinosaurs are embedded in a gigantic, near-vertical slab of limestone rock.

Unfortunately the visit was a major disappointment, even for a former dinosaur nut like me. It seems that the company owning the quarry prevented the construction of a proper dinosaur park or museum. The limestone rock containing the footprints is seperated from the actual museum by a large part of the quarry, including a road carrying its trucks, and this section is off limits to any visitors. The small dinosaur museum is located on a hill about 50 meters away, from where you can squint to try and see the footprints.

I couldn’t help but wonder whether this was not the result of bribing and corruption which enabled a big business (the cement company) to prevent the establishment of a proper museum (next to the actual footprints) of what is obviously a national archeological treasure. The museum itself looked nice from the outside and it seems as if is was created or funded by people with good intentions, who wanted to create a center for education and the preservation of the prints, but the fact that it was so far removed from the actual site was just depressing. In the end, after hearing from another disappointed Spanish tourist, that the museum itself was not worth the entrance fee, and also out of principle, we decided not to enter.

Tony and I tried to sneak down the hill and closer to the footprints to get a closer view or at least a nicer picture, but we were quickly spotted by the guards and told to leave. Closer to the tracks, we found a small brick building and a sign welcoming visitors to the Dinosaur museum, confirming our suspicions that the original musuem was planned to be closer to the actual prints. Too bad.

5 August, Sucre: My birthday, the mirador and the best steak in a very long time

One of the advantages of Sucre, is the high number of good and yet very affordable restuarants in the area. Today was my birthday, and Manuela and I went with Tony and Sofia to a café up a hill where one could have a view of the whole city. Tony and I bought some coca leaves at the market the previous day, and we were munching on balls of leaves pressed into our cheeks as we walked up the hill. After about an hour of having the leaves in your mouth, you should add a catalyst (either plant ash or bicarbonate soda) and then put the ball back into your mouth. The end effect is a mild stimulant, like a strong cup of coffee, but not much more.

Afterwards we went to a French restaurant near the city center. We drank a bottle of Aranjuez red wine and decided to eat potatoes with caviar and the steak with red wine and mushroom sauce. This turned out to be the absolute culinary highlight of our time in Bolivia, and the best steak in ages, all for less than €5 per person.

Bolivia’s anniversary of independence is on the 6th of August (circa 1825), and on the night of the 5th, there was a large military parade right under the balcony from where we were sitting. It was a very nice coincidence (Glücksfall), and we felt fortunate to be able to witness it from above, since the streets were incredibly crowded. The parade was quite militaristic, something which none of us has ever really witnessed. An impressive variety of military units with different outfits and weapons marched past our balcony, including a group of paramilitaries who rode on jeeps wearing full camouflage, while pointing semi-automatic machine guns at the spectators. Quite a spectacle from above.

7 August, Sucre: Going out to Florin

After having had a fun night of food and drink 3 nights before at Florin, a dutch owned bar and restaurant, we decided to go back for seconds. Coincidentally a pop band from La Paz named Dr. Jet was playing that night and we were entertained with covers of Spanish pop songs as well as Lady Gaga, Shaquira, Robbie Williams and others.

It was interesting to see that not a single person there was wearing any traditional peace of clothing or even jewelry. Everyone was dressed completely western, with one guy making sure that his Hilfiger jersey was draped over his shoulder exactly right to allow one to see the label. Just as the preppies do in the business school in Mannheim Germany and probably all over the world.

Sucre in general was a culinary pleasure especially considerung the prices. It’s also the chocolate capital of Bolivia and we more than once went to a local cafe where we drank hot chocolate and ate chocolate covered fruit, for about € 2,50 in total. In the end, we stayed much longer in the city than we expected, as we were hoping for the road to Potosi to clear. It was being barricaded and blocked for weeks by unhappy protestors, trapping panicky tourists inside the city of Potosi and preventing anyone else from going in. In the end the situation took too long to normalise and we decided to continue onwards to La Paz.

10 August Arriving in La Paz

The overnight bus ride to La Paz was quite terrible and nerve-racking. Just the day before we read news of 72 people that died (and 196 injured!) only in the month of February 2010 in bus accidents in Bolivia, and now we were sitting in such a bus, speeding along the side of a mountain, with a steep drop off at the one side. Loud Peruvian/Bolivian music blared through the shitty bus speakers, creating more distortion than anything melodic, while we tried to enjoy the cultural experience and not imagine rolling down to our deaths in the valley below.

The bus trip was again overnight and upon arriving in La Paz and checking into our hostel, we decided to make use of the rest of the day by walking to the nearest lookout point Mirador Killi Killi.

La Paz is really high, the highest capital (administrative) city in the world, and we quickly ran out of breath every time we walked up a hill.

11 August, La Paz’s witches market

We went to the Plaza de Armas in the city center on our way to the famous Witches’ market. The main square is again quite pretty, surrounded by colonial buildings and well policed. One of the buildings still contains bullet holes from a gun fight between the police and army some years ago. The square is full of pidgeons and local women sell seeds to feed them, which was popular among both the locals and the gringos.

We continued further, past a few more markets before eventually reaching the wiches’ market. This is probably the most touristic area of La Paz, but still fascinating. In between the numerous touristicos stalls, there were witches’ stalls, selling all kinds of plants, potions, offerings, totems and even dried up baby llamas or llama fetuses, dried frogs and dead Armadillos.

Referring to these people as witches is in my opinion quite misleading and also unfortunate, because aside from the “magic potions” which are all bullshit and imported from the east anyway, these people were just selling the traditional gifts (such as the baby llamas) to be offered to Pachamama, the earth goddess. To refer to them as witches is to impose (often negative) western stereotypes onto their traditional customs and practices. Many native Americans in Bolivia and Peru still adhere to their traditional and native customs and practices, which include making burnt offerings to Pachamama.

13 August, Mountain biking on the death road

Mountain biking on the death road, outside of La Paz, is one of the main tourist attractions in the area. It’s called the “death road”, or “the worlds most dangerous road” because of the alarming amount of fatal car accidents that have occured there over the years, and not because of the amount of mountain biking accidents, although some tourists have also died. The road is dangerous, because it is only wide enough for one vehicle at a time, slopes downwards from a height of 4700m to 1200m and is flanked by the side of a mountain one the one side, and a steep drop into the canyon below on the other side.

What few people seem to mention however, is that the death road is also beautiful. It is inside a cloud forest and the surrounding landscape and vegetation is simply stunning.

The cloudforest seems to be one of nature’s most psychedelic places. White mists from the clouds fill the air, as if from a giant smoke machine, while new fractal shaped fern leaves uncoil and unfold according to a genetic blueprint that is hundreds of millions of years old. Now and then you find colourful flowers; pink, purple, orange and red, some big and shaped like a Jester’s hat and others small, hanging like brightly coloured clocks. Between them fly colourful butterflies. Bright spots in a canopy of a million different shades of green. All the rocks and fallen branches are covered in moss and vines twist and turn around the trees and bushes. Streams flow in the cracks and ravines, sometimes falling dozens of meters over the edge of a cliff, down to a pool below. The whole place seems to breathe; alive, vibrant and blessed with the abundance of water.

Biking the death road was an incredible thrill and well worth the price. We started early the morning at the top of the mountain La Cumbre at 4700m. So high, that it was icy cold and actually snowing. The first part was on a tarred road, still being used by cars and trucks (as opposed to the actual death road which has a newer tarred replacement). After a while we reached the actual death road (a gravel road) and the beginning of the cloud forest. From here on, it was still another 63km and 3500m down to the vally below. An unforgettable combination of adrenalin sport and nature.

15 August, Last day in La Paz and traveling to Copacabana

After a few days in La Paz, we left for our final Bolivian destination. The town of Copacabana on the coast of Lake Titicaca.

On our final day in La Paz, we shared some impressions of this fascinating city and we remarked on how amateur things in Bolivia sometimes appear. In some areas of La Paz, the billboards are actually the prettiest things in view. Sometimes, we would go to a restaurant, and it would feel as if you were the first customers they have ever had.

Upon our arrival in Copacabana, we went to such a restaurant. We wanted to leave, feeling that it might be a mistake to eat at this particular restaurant, seeing that they had no other customers yet, horrible decoration, a dirty table cloth, and a strange menu, but the whole family of about three generations were looking at us excitedly (at least by Bolivian standards, which is with almost no expression at all), so that we felt compelled to stay and stimulate the local economy. Big mistake. We ordered a bottle of Kohlberg red wine, which by had now secured its place as the best value for money in Bolivia, but this time for more than double the retail price. Ten seconds later the mom and her small daughter ran out of the restaurant and Manuela makes a bet with me that they are going out to buy the Kohlberg. Sure as hell, a while later she returns with a bottle shaped object hidden beneath her poncho.

We also ordered two trouts, one with butter and one with tomato, known to be one of the local specialities. After a while the trouts arrive, both looking identical. Fried in fat, lying on a handful of chips, with a bit of dry rice on the side. The only difference is that my trout is completely covered in Ketchup while Manuela has some butter (or perhaps margerine) smeared over hers. Depressed at being fleeced with our shitty trouts and (expensive) cheap bottle of wine, we ate without much conversation, although we occasionally made jokes and laughed at the situation.

It’s remarkable how many Bolivians try to use tourism as a means for creating better lives for themselves and how hard they sometimes have to work for it. Once we were in a restaurant, run by a married couple who worked from the morning to the night, while their small child played behind the counter. The mother makes all the food and the father takes the orders. Later, we would go to a restaurant run entirely by a single woman. She would take the orders, cook the food and serve it, for seven tables! We were glad to support these hardworking local people, and were therefore also prepared to sometimes lower our standards a little.

16 Copacabana

In Copacabana we visited the local cathedral with its famous Virginita of Copacabana, a statue (almost doll) of the holy mother Mary, created by a relative of the famous Inca king Atahualpa. The Incas worshipped mother earth through the goddess Pachamama, and this practice was later synthesised with the Catholic veneration of Maria, as introduced by the Spanish.

Copacabana is surrounded by three hills, and we wanted to climb up one of them to see the nice view. As it happened, this hill was also a type of pilgrimage, with a place for burnt offerings at the top. Along the path to the top, there were stone monuments, with scenes from Jesus’ capture, lashings and crucifixion, culminating in the final scenes of his death and resurrection at the top of the hill. Midway up towards the top of the hill, a shaman was holding a ceremony for a young couple, burning their offerings of candy, fake money and other paraphernelia on an altar.

We took a breather and went to the edge to look out at the sun over lake Titicaca ahead of us. Just then, an old man walks past us, pulls out his wiener and pisses right in front of Manuela and I. We took the (not so) subtle hint, and left the area to continue upwards.

Walking up the hill, I remarked that this must be the dirtyist, most littered hill I have ever climbed. Paper, plastic, beer bottles and everything in between was scattered along the whole way, making the experience much less pleasant.

At the top of the hill, more people where making offerings to smaller representations of the Virginita (basically a doll in a glass case), who represents the mother Mary, but also Pachamama. The locals believe that (among other things) you must offer something representative of that which you want to receive. So many people would burn fake money, in the hopes of receiving real money.

The craziest of all however, was the stalls at the top selling small houses, cars and other offerings such as confetti and sweets. Presumably, if you want a new car of a new house, you buy such a small representation, and burn it as an offering to the Virginita.

Another strange practice, that we also in a catacomb beneath the church, was the custom of buying candles, burning them and then using the wax to create the outline of the object that you want. As mentioned, at the top of the hill were stone monuments with paintings of Jesus’s crucifixion and resurrection behind dirty glass cases. These stone monuments were completely covered in these wax figures, making them very oily and dirty.

It’s relatively well known (or should be) that Catholicism is a synthesis of early christian sects and some pagan European practices, which makes the fact that there is an ongoing synthesis of Catholicism with the indigenous American/Incan beliefs as found in Bolivia, ever more fascinating.

17 August, Isla del Sol

Apparantly, according to the Incas, lake Titicaca was the navel of the world, the place where the first Incas were created and also where the sun and the moon came from.

We took the 8:30 ferry to Isla del Sol (Island of the sun) where according to Incan legend, the Incan god Viracocha, the first Incan Manco Capac and his sister (Incan royalty married their sisters) met and where the reign of the Incas began. Nearby is also the smaller Isla del la Luna (Island of the moon), where the ruins of a pre-hispanic Incan monastery that housed virgins to be sacrificed on the sun island, can still be seen. It is said that the sun and the moon came out of lake Titicaca, and that the sun, being jealous of the moon’s bright light, threw dust into its face, thereby covering its light and causing its grey white colour.

We stopped at the north end of the island, and after a quick stop at a small museum where you could see Incan and pre-Incan artifacts recovered from the depths of Titicaca, we headed off to the ruins at the north end of the Island. We walked to the Piedra Sagrada (holy rock), before continuing to a rock outcropping containing the face of the Incan god Virococha. The Incan’s (and perhaps even pre-Incas) recognised an elongated angry looking face in the rock, considered it to be from a deity and built an altar in front of it on a piece of rock that is so perfectly situated in front of the face that it’s almost eery. We were introduced to a local Curandero Shaman, who still conducts ceremonies at the altar, asking for good harvests or other favours. He explained to us that virgins were sacrificed there in pre-hispanic times, later to be replaced by llamas, and also that there is an energy in the name Virococha and in the names of other deities.

In what is such an incredible coincidence (and blind luck) that it’s almost uncanny, the image of the first conquistador to meet the Incas in the area, Francisco Pizzarro, reminded the Incas of their deity Virococha. An elongated, angry face with a beard. The Incan ruler, Atahualpa killed his half-brother (and heir to the Incan empire) a while before, and the Incas were already tense and scared of what they considered a taboo act by Atahualpa. So when Pizzarro (deemed representative or incarnation of Virococha) arrived, his Incan army of 40 000 men were so terrified of their deity’s wrath, that they fled before the 140 men strong conquistadorean army. This provided Pizarro with a victory that would otherwise have been much less likely.

The effect that the Inca’s prophesies and mythologies had in their own downfall and subjugation towards the Spanish, gave me food for thought concerning the ever popular Christian apocalyptic prophesies and their power in actualising some of the events they “prophesize” simply by the power of enough people believing in them (as the Incas believed in their own mythology).

Apocalyptic musings aside, we visited the Incan palace nearby, where Atahualpa first met Pizarro and offered some of his gold in turn for his life. Pizarro instead decided that he wanted all of the gold and then killed Atahualpa.

After visiting the ruins, we started walking along the ridge of the Island towards the south side which took us the whole day, ending at a restaurant with a spectacular view of the sunset over lake Titicaca.

19 August, Traveling to Arequipa, Peru

We took the early morning bus to Arequipa in Peru, a 9 hour journey. Along the way we saw impressive hilled landscapes with herds of llamas and a lake with pink flamingoes. Unfortunately the bus driver drove like a maniac, speeding, honking incessantly and making the whole bus sway from side to side, causing Manuela and I to become car sick. The unpleasant bus journey was however lightened up a bit by a guy who sang while playing a peruvian flute and a Jukelele.

So far Peru seems very much like Bolivia. As in Bolivia, many people paint the walls of their houses with political advertisements for their preferred candidate for the local governor. Sometimes a large rock in an outcropping in the middle of nowhere would be painted with a political party’s emblen, name and slogan on a white background. I am much more aware of the local politics in Bolivia and Peru than during our time in Argentina and Brazil.

21 August, Arequipa

After staying in our third so-called “party hostel”, which are very popular among backpackers, we’re quite disappointed with the level of sophistication and maturity of many of the backpackers we encounter. A lot of them treat the experience the same way teenage adolescents would treat a school trip, flirting and jockeying for attention or popularity, drinking every night before throwing up and then bragging about it the next day, and generally acting as if they are still somewhere in Europe, Australia or North America. One highlight was a drunk couple coming back to our 6-bed dorm after partying until about 3:30 am, and then proceeding to have very noisy sex, in the process waking everybody up and all without even the slightest appology.

This is of course a gross generalisation, and we have met many open-minded, knowledge hungry and genuinely interesting travelers along the way, but usually not in “party hostels”.

22 August, Arequipa

Coming in to Arequipa, we saw a lot of poor neighbourhoods with dirty unkempt buildings. The city center is quite the opposite however, with lots of renovations being done in the streets around the main square. Arequipa is surrounded by three big volcanoes, and the colonial buildings in and around the square have been built with the white “sillar” volcanic rock, lending it it’s reputation as the “white city”.

In the afternoon we went to the local “Santuary” museum, where the frozen remains of “Juanita”, a twelve year old Incan ritual sacrifice victim is on display. The Incas sacrificed children, seen as pure and unspoilt, to win the gods’ favour whenever they fell on hard times. In the 1400s, the Incan empire was plagued by an erupting Volcano (near Arequipa) followed up by a drought. Juanita was the daughter of a family of very high social standing, who was taken from Cusco, all the way up to a very high Volcanic mountain to be sacrificed. The Incas believed the mountains were the realms of the gods and the weather patterns and tribulations they faced going up there, were the gods response to their presence.

After her death, Juanita was wrapped in fine cloth and buried with a collection of symbolic and sacred objects. For the next 500 odd years, she remained frozen in the -20°c temperature, untill she was discovered in the thinning ice of yet another volcanic eruption in 1995.

The musuem and the staff were very professional and well worth a visit. A far cry from what we had become accustomed to during our time in Bolivia.

23 August, Mountain biking at Volcano Misti

After having so much fun mountain biking down the death road in La Paz, we decided to have another mountain biking adventure at the Misti Volcano. We got picked up early in the morning and drove out to the foot of the Volcano, getting nice views of Arequipa as we left the city limits. The outskirts are not so soft on the eye (to put it euphemistically) as the city center. Peruvians stream into the city looking for a better life and construct shabby looking brick buildings on the outskirts. Often the walls are left unrendered and unplastered, leaving the houses looking ugly and unfinished.

New communities made up of these buildings appear quite quickly at the edges of the city, where there is no water, sewage or electricity infrastructure. After some time the new residents start to demand basic services and eventually the government provides them, together with property rights. In this way, the city grows in a haphazard and disorganized way and making the outskirts very unattractive. It must be said however, that these people at least build brick structures, so it’s still a few steps ahead of squatters and shanty towns of which we have seen surprisingly little in South America.

A little bit up at the foot of the Misti, we got stopped and geared up to bike down to the bottom. It was quite a bit more difficult and technical than in La Paz, but still an incredible amount of fun! We then drove to another small rural town and mountain biked there as well, before returning to town for lunch at Mistica café in Calle San Francisco. Secret tip: The best lunch we had in town and for only 8 soles each.

24 August, Trekking the Colca canyon

The Colca Canyon, outside of Arequipa, is reportedly the deepest canyon in the world, twice as deep as the Grand Canyon and averages a depth of 3400 meters for a length of 100 km.

We booked a tour for two days and left at 3:00 am the first day to the town of Cabanaconde to start hiking down the canyon. A while before we stopped at a well known Condor lookout point for breakfast and another opportunity to spot some of these majestic birds. What awaited us there completely surpassed our expectations. Condors were floating on the rising thermals right in front of the viewpoint and sometimes less than 5 meters away! About 8 condors seemed to be in the area, seemingly unperturbed by the dozens of tourists snapping away with their cameras.

The hike down to the canyon took the rest of the day, about 7 hours in total and 14 km in length. We spent the night at the so called (and well deserved) “Oasis”, with a pristine rock pool of fresh 22° mountain water. The accommodation consisted of huts with thatched roofs and mud walls and floors. Basic but part of the fun and atmosphere.

25 August, Trekking the Colca canyon

On the last day of the Colca canyon treĸ, we woke up at about 4am to hike back up and out of the canyon. It took us about 2 hours and 45 minutes (3 hours being the average) to complete the 6km long and 1km high hike out of the canyon. It was quite a challenge because it was so steep, but also good training for the hike to Choquequiraothat we would later attempt. Part of the hiking trail up and out of the canyon can be seen in the photo below:

The rest of the day was mostly spent driving in the bus and watching the scenery, including a majestic view of a nearby valley covered in terraces (seen below) some perhaps dating from the Incan times, before visiting some hot springs where we could relax our aching muscles.

27 August, Leaving to Cusco

At the end of the month, we left for the famous city of Cusco, were we’ll be visiting two lost Incan cities. Machu Picchu and the lesser known Chocequirao.

That we will leave for next month however, and for the next blog entry. :)

P.S. Think twice before using Escandinavia tour agency in Arequipa, Peru

We booked the mountain biking and Colca Canyon activities through a local tour company (Escandinavia) located in the main square (Placa de Armas) in Arequipa. The agency is run by a Peruvian named Edgar, who speaks good English. We were quite happy with the tours he booked for us, but unfortunately things changed for the worse later when he made an erronous bus booking for us, for which he lost an entire day. Afterwards we caught him lying to us about having someone to pick us up in Cuzco, and he also didn´t pay (as promised) the deposit of a booking we had to cancel because of his mistake. The fact that he made a mistake in booking our bus could be overlooked, but after he lied to us and also neglected to fulfill his promises, we feel it’s necessary to warn others. The staff at the Point hostel in Arequipa also later informed us that - after some disappointments - they do not make use of his services anymore.

South American travel diary, Part 2. July 2010

Sat, 07 Aug 2010 00:00:00 GMT

We continue on our travels through South America, including visting the Pantanal in Brazil, Iguacu falls in Argentina, Ciudad del Este in Paraguay and then off to Samaipata in Bolivia.

4 July, Canoa Quebrada

We got a room without breakfast included, so we decided to use the opportunity to sleep late instead of being woken by an alarm clock. At about 12 am we went down to the local (very basic) bar/restaurant for some Churrascuro (grilled meat and side dishes) at R$ 10 per person (about €4.50). It’s been 6 weeks without any stomach problems or diarrhea. This includes eating from street stalls, bus stations and from makeshift “home” restaurants (sometimes in people’s living rooms).

We then went to the main beach, quickly saw that it was too crowded for us, turned left and started walking to a collection of electric windmills off in the distance. Canoa’s beach is quite long, a local estimated about 20 km in total, with the main beach occupying only about 150m, so we had a long stretch of deserted beach to explore.

I expressed my desire to to go and stand under one of the windturbines so we continued walking along the coast into their direction. After about almost two hours of walking in the midday sun we concluded that these things were still deceptively far away, and that we must get some rest soon. To make matters worse, we only had 500ml of water with us. I eventually found a new target to make all the walking worthwhile and we headed off to a sanddune nearby to look for some shade.

Canoa Quebrada is very reminiscent of Swakopmund in Namibia. It’s very hot and dry with a long stretch of coast flanked with white sanddunes. Very pretty in it’s own way. We were unfortunately quite sad to see plastic litter across the entire stretch of beach that we walked. To make matters worse, we found several dead fishes and puddles of brown and clearly polluted water. One of the big touristic selling points in Canoa is the buggy rides to the sanddunes. Every 15 minutes a buggy with smiling people would pass us by as we walked along the coast and practically the entire stretch of beach was turned into a buggy highway covered in tire tracks. As a result we were a bit disappointed with Canoa Quebrada. It seems the residents still need to develop a sense of responsibility for conserving the beaches fragile ecosystem and for keeping it clean.

6 July Overnighting at Fortaleza airport

Our plane from Fortaleza to Campo Grande left at 6:00 in the morning, so we made the decision to stay the night at the airport.

At about 12:00 at night, we made a little campsite under the stairs of the viewing hall and got ready to tuck in. Manuela successfully slept there for about an hour, but about half an hour after I joined, a security guard came and politely asked us to leave. That was pretty much our only chance for sleeping that night, so for the remainder we sat on some (probably purposefully) uncomfortable chairs listening to music and trying to catch some minutes of sleep before being woken up by the inevitable pins and needles in the butt cheek and leg that bore the brunt of the body’s weight.

7 July, Campo Grande

We arrived in Campo Grande at about 12 in the afternoon. Hungry and exhausted. We met a Danish couple and the four of us were quickly roped in by a smooth talking Pantanal tour salesman named Gil.

Gil fired off his sales pitch at our fragile, sleep deprived minds. We were all wary of being ripped off and trying to be intelligent, discerning tourists, but eventually the prospect of going downtown alone with our backpacks to look for a better deal looked less and less appealing. Gil continued, showing us pictures of a Pantanal lodge with reclining beach chairs, swimming pool, dormitories and private rooms and then when our defenses finally broke, went ahead and sold us a package that excluded all of the above. Without explicitly telling us of course.

9 July, Santa Clara lodge, Pantanal

The first morning of our stay in the Pantanal, we had to be ready at 4 o’clock to go on an early morning game drive.

The Pantanal is 90% private owned, so we had to restrict ourselves mostly to the big public gravel road that moves between the farms, which felt less immersive in the wild than the game drives I’ve been on in South Africa. This was one of a few times that the thought occurred that it’s quite a shame that there is no big national park in the Pantanal, such as Etosha in Namibia or Kruger Park in South Africa.

For brevity, I’ll refrain from overly describing this amazing place, and instead the inquisitive reader to the wikipedia page. Some interesting facts that I can’t help mentioning though, are that the entire Pantanal covers an area the size of France and contains about 35 million caimans (alligators) and a stunning collection of birds and other wildlife. Pantanal means swamp, and during the rainy (summer) season, almost the entire place gets covered in water. During the dry months, only a limited amount of swamps remain though, with lots of dry bushveld in between.

Our guide for the early morning trip was exceptional. In the dim light of dawn he spotted birds in trees and even small spiders in the road. Sometimes it took me half a minute to spot a bird that he recognised minutes before in a moving vehicle.

Overall we spent about 6 hours in the bush, which included watching a spectacular sunrise over a lake, followed up by a hike for about 2 hours.

We saw quite a few animals, a local raccoon, small bush deer, large deer, some more caimans, black howler monkeys, a large and scary tarantula like spider and a variety of birds.

The afternoon was spent horse-riding. What could be said about that? Well, it was great, I could get used to a lifestyle of regular horse-riding and I felt a special kinship with my specific horse, even though it was a bit of a tourist whore. It made some strange grunting and burping noises during the ride, which prompted various theories from our group as to their origin, randing from the mundane, to sexual to esoteric.

10 July, Santa Clara lodge, Pantanal

The nights we slept in hammocks in the open air (surrounded with mosquito nets), and were woken up that morning by the sounds of the wild. Loudest of all was the call of a bird called the Chacochachalaka, whose name is an onomatopedic attempt to recreate this creature’s cacophonic call but which ultimately fails. The noise is usually created by at least two birds in tandem. The one starts making a more high pitched call, then immediately followed by a lower pitched reply. The aural effect is a bit similar to a recording of a chicken being grabbed by the neck and swung around a few times, with the pitch lowered a few octaves.

For the morning activity, we went on a boat ride on the local river, a tributary of the Paraguay river. We got a chance to see even more Caimans, some Capivaras and another large variety of birds.

The afternoon was spent piranha fishing with little peaces of beef, a surprisingly easy and fun activity. Manuela had one of the biggest catches of the day, I caught about 5 piranhas, with 3 being let go for being too small to do anything useful with.

Later, while we were cleaning the fish, our guide cut out the tiny heart of a dead piranha. To my amazement, it was still beating! Our guide remarked that perhaps that’s why the piranhas are such nasty fish, they have very small hearts. I held the beating heart in my palm for a few minutes, considering wether I should eat it, but eventually threw it away. The piranhas have very sharp teeth, capabale of biting off a finger, even after they’re dead (due to convulsions), but the concept of people being attacked and killed by piranhas is a myth. We swam in the caiman and piranha infested waters and lived to tell the tale. Piranhas will only attack if you are bleeding and caimans eat only fish. So even though both animals are still dangerous and can definitely injure a person badly, it’s still quite safe to swim in the river if you maintain a distance and avoid any bleeding.

11 July, Santa Clara lodge, Pantanal

Early that morning, in addition to the now familiar cry of the Chacochachalakas, we were woken up by the ominous ruckus made by a troupe of Black Howler Monkeys. It was however a pleasure to lie in a hammock in the early morning ours listening to the sounds of nature, and even the less musical animal cries are a huge improvement over the sounds of construction workers and city traffic.

12 July, Santa Clara lodge, Pantanal

We spent the morning taking on a two and a half hour hike, during which our guide explained many of the medicinal properties of the local vegetation. Examples included the Yellow Trumpet Tree, whose bitter sap is a remedy for inflöamation. A water lily from which a hallucinogenic tea could be made, the Acori Palm, from whose seeds you could press drops of water to use as eyedrops. The Acori Palm was quite abundand in the area and seemed to be an important part of the local ecosystem. The leaves are fed on by the cattle during the raining season, when the grass is covered in water. The coconuts are eaten by armadillos to file their teeth and eaten by the Macaws, who can crack them open with their powerful beaks. From the hard shell covered fruit you can make a drinking cup, a plate or a horn with which to make a Jaguar call. Inside the hard shell of the fruit is a yellow flour (in addition to some worms) which is eaten by the natives. We were also introduced to Tuco, a bamboo used for making bows, and Caraguata, a bromeliad whose leaves are used for fiber. At this point I stopped taking notes and decided to concentrate on enjoying the hike instead.

Overall, the Pantanal was an enriching but not entirely stressfree and relaxing experience. Firstly, there was an unpleasant run-in with a very rude and unprofessional guide on the first day. Then, the huge amount of acquired mosquito bites coupled with steep temperature changes between searing hot in the afternoon and chillingly cold at night in addition to the everpresent and all-covering dust, meant that I was quite relieved to be leaving. On this trip so far, we have been happiest when we were doing our own thing on our own time. Taking part in a packaged tour of the Pantanal was just the opposite of that. The activities were fun and worth the effort and money, but there were just a few too many unpleasantries, none the least the way we were treated by some of the lodge staff, that make me remember the experience with mixed feelings.

13 July, Bonito

We arrived in Bonito (meaning beautiful) the night before, after a long and very bumpy ride on what was probably the most half-hearted attempt at creating a temporary gravel road in order to facilitate repairs on the main road. In other words, this gravel road was awful, and I had visions of our poor little minibus breaking down with us having to spend the night at the side of the road in the middle of nowhere.

Instead, all went well and we were now in the popular town of Bonito, renowned for it’s offering of beautiful caves and rivers and a whole range of activities associated with them. There was just one problem, the hostel where we stayed and all the activities it included, were terribly expensive.

We therefore opted for the poor man’s alternative, rented bicycles and rode 7km to Banleário Municipal, where we could snorkel in the river. The bicycle trip out of town was a pleasure, providing another chance to take in and appreciate the landscape. Banleário Municipal was a pleasant surprise, next with a very beautiful and clear turqoise coloured river, swarming with fishes. We snorkeled for about two hours until it started raining, when we huddled under a shelter with two Brazillian caretakers discussing (in broken Portuegese) Brazillian weather/festivals/beer/people/etc. compared to their German equivalents. We were again struck by the friendliness of most of the Brazillians we encountered.

Bonito is definitely one of the richest most developed towns we’ve encountered so far, with quite a few very stylish and modern houses around town. Perhaps that’s why the activities were so expensive. It could also however just be an indicator of the wealth gap between the poorer North and richer South of Brazil.

14 July

After one day in Bonito, we left deviated from our original plan and decided to leave for Foz do Iguacu, a Brazilian border town near the Iguacu waterfalls. As we headed more and more south, the landscape and the climate continued to change, getting greener and colder. Some areas even reminded us of the south of Germany, with neatly kept green fields dotted with patches of forest vegetation (rainforest as compared to Germany’s temperate forest).

We had to change buses twice and waited about two hours in the middle of the night for one of the later ones, which meant that I was again very tired upon arrival.

We decided to take it slow, had lunch with a nice older Danish couple who (very patiently) were waiting with us the previous night for their own (even more delayed) bus. We swapped some travel stories and they told us about their adventures in Peru. Afterwards, we left for the Argentinian border town of Puerto Iguazu, to find some accommodation for the night.

Upon arriving in Argentina, we quickly detected a different vibe, culture and look than we were getting used to in Brazil. The shops seemed to be better stocked and with more specialties.

15, 16 July, Iguacu Waterfalls

We spent two days visiting the Parque Nacional Iguazu, in which the spectacular Iguazu waterfalls can be found. The largest of which, is the Garganta el Diablo, or “Devil’s throat”.

The magnificence of the this waterfall is beyond description. Enormous amounts of water continuously flowing over the edge and into a white abyss. The white mist making it difficult if not impossible to see the water below. When the eye looks on to the water pouring over the edge, it tries in vain to find something in the ever changing sceneto focus on, giving rise to a mesmerizing effect. I felt as if I could have stared at the falling water for hours.

There are 269 other waterfalls, extending to both sides of this monster, creating a wide vista so impressive and beautiful that no picture or photo could ever do it any justice. We know, not just because we tried to capture it with our small 100€ camera, but also because we could not find any postcard, poster or picture elsewhere that captured their splendour. Simply put, they are much too big to adequately fit in any picture while still maintaining a perspective of their true size.

On the second day, it was very cold and raining, which dampened our enthusiasm a bit, but after seeing a blind person visiting the park the previous day, things were placed in perspective and I was grateful for having the good health and fortune to see these impressive waterfals.

18 July, Tres fronteiras park

After our rainy excursion to the waterfalls, I fell ill for about two days in which we stayed inside at the hostel. The small ten person dormitory was particularly suffocating for Manuela who didn’t like the hostel to begin with.

On the second (and our last) day, I however started to feel better and we decided to walk to the “Tres Fronteiras” park, were you could see Brazil, Paraguay and Argentina all in one view. The park itself was free and our spirits were lifted after getting out of the hostel for a change.

19 July, Ciudad del Este

Ciudad del Este is a chaotic Paraguayan frontier town on the border with Brazil and Argentina and the second biggest city in Paraguay. Its entire economy seemingly depends on its large markets of (mostly cheap Asian) consumer goods which are bought by Brazillians looking to save money by avoiding customs and sneaking the goods back through the border. Often the goods are then resold on the Brasillian side.

Many people who visit Ciudad del Este for a day don’t even bother to get their passports stamped and the buses and taxis who cross the border from Brazil wont stop at the passport controls unless you specifically ask them. Upon nearing the border, we however decided not to tempt fate, and asked our bus driver to let us out to get our passports stamped.

Upon entering the city via the “Friendship bridge”, we immediately saw the large markets covering all available sidewalk space. Many of the buildings between the sidewalks are also filled to the brim with small markets and shops. We didn’t experience the Paraguayans as being too aggressive salespeople however and usually didn’t feel badgered to buy crap we didn’t want.

It was raining during our excursion, and this only highlighted the neglect and disrepair of the city itself. Large potholes were filled with brown water, and some buildings seem to never have been painted. By chance, we passed a mall (more like a narrow 5 story department store than a mall) called “Mona Lisa” which we recognised from large billboards in Brazil, urging shoppers to go to Ciudad del Este and shop there. We decided to take a look.

Inside we were quite surprised. Fancy and expensive cologne and fragrants, new Ipads and Iphones, imported Swiss chocolate, Sushi and a whole assortment of other luxury items. The mall itself could be located in any big city in the world, but the contrast with the disheveled, dirty and poor spectacle going on outside its walls was striking.

At the end of the day we took a minibus taxi back across the bridge to Brazil. The one guy sitting next to Manuela looked quite nervous as we neared the Brazillian border but as the taxi driver made a point of speeding around the border gates he relaxed visibly and made a Catholic cross with his right hand. Everyone

was therefore a bit surprised when the two gringos in the back asked to be let out to go and stamp their passports again. The driver pulled over

right there at the side of the road, ignoring the hooting from the irritated drivers behind us and let us out, before speeding away again.

On the same day we took a long distance bus back up to Corumba on the border with Bolivia.

21 July, Puerto Quijarro

We spent a day in Puerto Quijarro, a small dusty Bolivian frontier town on the border with Brazil. After having spent almost all my cash in obtaining a visa I wasn’t aware I needed, I had to find a place to withdraw some more Bolivianos. Neither of the two cash machines in the town accepted my card however. Again the problem seemed to be that I had a Mastercard. In fact, I had two, which feels like having two left feet at a dancing contest. In South America, Visa is accepted almost everywhere, Mastercard however is a different story. In the end we had to take a taxi to the nearby town of Puerto Suarez, to use their ATM, which did accept my card.

As a result of this, we didn’t have a chance to buy train tickets to Santa Cruz for the day (you can only buy tickets in person and for the same day), and ended up being the only two guests staying in one of the nicest, most spacious hostels we’ve ever been to, Tamengo. It used to be the local public swimming pool, before the current owner bought it and turned it into a hostel. The town of Quijarro was so poor that it seemed unlikely that the municipal would have had the funds left to maintain a public swimming pool.

22 July, Train from Puerto Quijarro to Santa Cruz

We woke up early and made sure to get tickets for the train later in the day. We would be traveling on what is known as the “death train”, apparently because it was used to transport yellow fever victims in the past.

The train shaked and dipped along at a snails pace. On a small television at the front of the compartment, Spanish songs from the 1970s were playing, sung, among others, by an overly zealous woman with kitsch, bright make-up and a priest with a guitar. Early in the day I heard an old 90s techno song blaring from some car’s speaker in a dusty road populated with chickens, pigs and countless stray dogs. So far, being in Bolivia has presented its strange moments.

23 July, Traveling from Santa Cruz to Samaipata

We arrived in Santa Cruz, after a harrowing 17 hour train ride. The compartment bumped, stuttered and shaked, and occasionally there would be very loud bangs coming from below as we not only swayed in a sideways motion, but also vertically, with the bottom of our carriage banging on the tracks below. On occasion the train swayed so much that I would instinctively prepare for the inevitable tipping over of our carriage. In the end, the inevitable seemed to be avoided, or perhaps just delayed for a later time.

One of the first things noticable about Santa Cruz, as we zipped away from the bus station in a taxi, was that there were no high rise buildings, and everywhere seemed desperately poor, although we went past a car dealership selling luxury 4x4s, including a Porsche Cayenne. Nevertheless, Bolivia is one of the poorest places I have ever visited, outside of some pockets in Southern Africa. Coming into Santa Cruz with the train, we saw an unplanned sprawl of shoddy structures, bisected with dirt roads. Rubbish is strewn everywhere, and in between chickens, pigs, cattle and people roam listlessly looking for something to do or eat. The children seem happy though, having fun playing in the dirt. Bolivias population is mostly indigenous, and for some reason their babies are incredibly cute, with dark elf-like eyes.

We spent about 3 hours in Santa Cruz and then took a minibus taxi to Samaipata (30 Bolivianos each), but after 20 minutes of alternating acceleration, then braking, then snaking through traffic, our driver sped over a speedbump and the car broke down. We had to all get out at the side of the road and wait for a replacement minibus to arrive.

The new driver was older with big, very thick glasses. I wasn’t sure whether this should instill confidence or fear in his ability to see objects in the road. Every now and then he would unceremoniously stop the car and jump out to take a piss at the side of the road in full view of the occupants. Later we were all almost killed when he narrowly avoided being driven over by a large oncoming truck. All the time without any communication from his side or the slightest apology. When we arrived in Samaipata, he simply drove past and we had to tell him to stop and insist that he turn back and drop us off. Now suddenly he was very chatty with the other occupants, no doubt complaining about the gringos in the back.

As he sped away, we were however more relieved than anything else, and looking forward to a new week of adventures.

A cliché in many people’s minds is that indigenous populations live closer to the land, more in harmony with nature and have more respect for the earth than the greedy, exploitative westerners. Sadly, this is not the picture we were presented with in Santa Cruz and on the way to Samaipata. Many people seem to lack any sense of responsibility for caring for their environment, painfully evident by the fact that the whole countryside seems to be covered in rubbish. Plastic bottles, wrappers and the like are simply thrown on the ground, with no regards for the terrible visual and environmental pollution it causes.

24 July, Samaipata

Samaipata, a town of 3000 inhabitants in an “ellbow of the Andes” with no bank branch, ATM, supermarket or dedicated internet connection, is the provincial capital of the province Florida and even boasts a court with 2 apparently unbribable judges. Unless you pay them in cash.

There is however a wonderful fresh food and meat market that more than makes up for the lack of a supermarket. The apples and pears in particular were delicious. There’s also a small European expat community (particularly Germans and Austrians) who provide tours, good restaurants, cafés and even a very good French bakery. Not to be outdone, there was also a very good local restaurant that we would often frequent during our stay.

Samaipata is located in a valley, with the rainforests of Amboro National park on the one side and a much dryer semi-arid climate on the other, which of course makes for a lot of diversity and interesting scenery.

On that morning, we met up with Tony and Sofia, whom we first met in Rio de Janeiro and stayed in contact with since then, and together decided to take a hike to a lookout point on one of the hills nearby. Long story short, we took a wrong turn and instead ended up at the local “Zoo”. More an animal refuge run by an expat Swiss wonman, with a selection of touchy howler monkeys, spider monkeys, a three legged wild cat, some dogs, wild pigs, tortoises, macaws, parrots and a selection of other critters. Our visit more than made up for not finding our lookout point and we were glad to contribute in a small way towards protecting the abandoned and traumatised animals.

26 July, Horse-riding

We returned to the zoo to rent some horses and rode up a nearby hill with a lookout point called “Mirador El Paradiso”. One of the dogs of the zoo went with us and he was later joined by another dog from a nearby farm. They accompanied us all the way up to the top of the hill, with the one dog making sure that any and all wildlife were thoroughly chased away before our arrival, by running out ahead and barking loudly.

The lookout point provided a very nice vista of the surrounding hills and valleys and the horses could even graze a bit at the top. The whole trip took two and a half hours at about €13. Money well spent.

27 July, Bella Vista

One of the oldest tour agencies in town (they might actually be the oldest) is called RoadRunners, started by a loud but very friendly Austrian named Olaf about 15 years ago. I was initially attracted to their shop when I saw that they had a book exchange with a broad selection, and in the end Olaf was nice enough to lend me a thick biography on Che Guevara. Che’s last guerilla mission was in Bolivia, and before he was eventually captured and killed somewhere in the area, his men went down to Samaipata to rob a bank and retrieve some asthma medication for Che. One local guard was killed and in the excitement the guerillas fled from the scene and the shocked villagers without remembering to get any medication. Che was condemned to suffer from acute asthma for the rest of the campaign.

We decided to take a tour called “Bella Vista” (Pretty view) with RoadRunners and it turned out to be one of the highlights of our lengthy stay in Samaipata. We drove out of town for about an hour and then proceeded to hike up a high mountain (dubbed the “cathedral” by RoadRunners) with a flat top from which we would have a spectacular 360° view of the surroundings. The hike was not too taxing, but no picnic either and we sometimes had to walk in single file on a ridge too thin for two people to stand side by side, flanked by steep drops on either side. Not a good idea for people with a fear of heights.

We were blessed with good weather and the views were spectacular, getting better and better as we hiked to the top. From above you could see in all directions, up to the furthestmost edges of the Bolivian Andes, about 80 km away. We hiked down on the opposite side of the mountain we hiked up upon, eventually ending in a ravine covered in a forest with ferns, bromeliads, orchids and even two impressive tree ferns.

29 July, Condor Hike

The majestic (but truly ugly) Condor, with a wingspan of up to 3 meters, is the national symbol of Bolivia. It can cover a distance of hundreds of kilometers in a day, conserving its energy by gliding on air currents. Near Samaipata is a cliff with a small inaccessible pool and waterfall, which attracts Condors from as far as Argentina and Peru, due to it’s secludedness and the fact that Condors cannot ascend straight from the ground up, but instead from the side of a cliff where the air currents can hold them up.

We hooked up with local guide Saúl, (a trained Biologist and owner of Tucandera tours) who accompanied Tony, Sofia, Manuela and I on a 16 km trek over the mountains to a lookout point where we could view the condors swooping down to the waterfall to bathe and drink.

We first took a taxi to a waterfall, called “La Pajcha”, and then to a point nearby were we started hiking up a mountain. Sometimes we would arrive, sweating and out of breath at a certain area, to find it littered with cow dung. This was something we already noticed on our Bella Vista hike and we were continuously amazed at the mountain climbing skills of the local cattle.

Saúl showed us some local fauna, including some aromatic flowers “Vira Vira” to make tea from (he was nice enough to pick some for us and we continued to make tea for the following two weeks) and a plant with which the local indigenous woman induce abortions (and risk their own lives or nervous systems). We saw condors quite early on, including one very rare (for the area) tropical condor, but we could not have known the spectacle that awaited us when we reached the cliff. There were about 6 condors visible at the top of the waterfall, and we made turns using Saúl’s binoculars to get a better view. Every few minutes, a solitary or pair of condors would appear on the horizon, to swoop down into the canyon and land on the cliff on top of the waterfall. It was an amazing site to behold, and, for lack of an expensive camera, we had to be content with taking it all in being in the moment. We saw about 25 condors in total and the day turned out to be one of the highlights of our entire South American trip.

We were warned beforehand that too many tourists might disturb the birds, and we made sure not to get too close, even declining to go to another lookout when our guide offered. In the end, we stayed a good distance, saw two dozen condors and had the time of our lives staring out into the expanse of the large condor filled canyon in front of us.

30 July, El Fuerte

The original Samaipata, a pre-Incan settlement, was located next to a large slab of sandrock. The sandrock was carved out at the top and at the sides, in what appears to be for ritualistic purposes. The Incas later arrived and continued to carve out symbolic shapes with Jaguars and Condors, and three twenty four meter long channels with symbolic snake carvings, used for liquid sacrifices. In the middle is a round circle with alternating seats for high priests and spirits. At the sides are niches that were covered in gold and contained urns with the mummified remains of important people.

Today, Samaipata is located in a valley, and this original settlement is now only known as “El Fuerte” (the fort). El Fuerte was the eastern most fronteir town of the Incan empire and therefore of particular importance to them, (much more important than Machu Picchu apparently) as can be seen by the length of the snake canal, which is much longer than a similar one in Cusco. Theories abound as to the significance of the site. Some years ago, an Austrian scientist studied the site, and found water rising out of holes in the rock (though to be used for sticking in poles for building structures). The theory arose that the sandrock might in fact contain limerock, that acts as a sponge and collects rainwater through the sandrock, only to release it again at a later stage. Upon witnessing this phenonemon, the original inhabitants considered the site sacred, and built a temple to pray and sacrifice to the spirits to provide them with water from the rock.

We again took the tour with Olaf from Roadrunners, who told us lots of amazing stories about the Incas, the Spanish conquistadores and an incredible coincidence that made an army of 40 000 warriors flee from 160 conquistadores headed by Pisarro.

Unfortunately the sandrock of El Fuerte is very soft and subject to a lot of erosion. So far the authorities have not done anything to combat this, and if left unchecked, some of the carved out features might be already be gone in 15 to 20 years.

31 July, Cloud forest in Amboro National Park

Samaipata is located close to the Amboro National Park, an area of about 100km by 120km with a large section in the middle completely off limits to visitors. Located inside Amboro at the top of a mountain range is a cold rainforest, the so-called cloud forest. The name arising from the fact that it’s so high that it’s mostly covered in clouds.

This was our last activity in Samaipata, and we went out one last time with Olaf to go and hike up to the cloud forest. Three weeks earlier, there was heavy snow in the area (for the first time in 50 years) and the cloud forest suffered a lot as a result, with many branches breaking, trees falling down and the big tree ferns losing their leaves. The hiking path was also partly destroyed and we would have to do some machete work here and there to clear a path.

It was however still an amazing experience walking through a forest covered in clouds, seeing trees covered in ferns, moss and lycans and smelling the mist in the air.

We can definitely recomend RoadRunners to anyone intending to visit Samaipata. Olaf and the gang were very helpful, friendly and honest and after three outings with Olaf we were all a bit sad to say goodbye.

1 August, Finishing Che Guevara

This was our last day in Samaipata before leaving for Sucre and I spent most of the day finishing reading the biography of Che Guevara that Olaf lent me. I was finished with about two hours to spare before our bus arrived and was glad to have received some insights into the man whose famous image adorns Cuban money, posters, flags and the t-shirts of clueless teenagers and students everywhere in the western world.

Che definitely had some admirable qualities, but he was also a fanatic Marxist and a cold-blooded killer and guerilla fighter. I find it the hight of irony that his image adorns the t-shirts of youths from the very same capitalist countries he so despised. Just as I have once before realised with regards to the selling of religious relics, I was again aware how almost nothing is sacred or taboo when it comes to selling on the open market of capitalism, even if it is the image of a man who died fighting against that very same idea.

South American travel diary, Part 1. June 2010

Tue, 06 Jul 2010 00:00:00 GMT

This is a roundup of the first month of my shared travels through Brazil and South America, in the form of a journal or diary. I tried to write down the highlights of almost each day, as a means of remembering special events and to share with the people who care. This post turned out to be quite long, with the personally more interesting and intuitive bits appearing near the end, as I became more relaxed and immersed in the traveling experience. Feel free to skip boring parts or to just look at the pictures ;)

30/31 May, Sao Paulo

The people at Olah! hostel were very helpful and accommodating. They picked us up at the airport and had to wait about two hours for us, due to a delayed flight. Then they even offered to take us to the supermarket at 12 o’ clock at night.

In general Brazillians seem friendly and eager to help. We felt as though we stood out as travelers but during our first day walking in Sao Paulo, three different people approached us asking questions (probably for directions) in Portuguese, so perhaps we blended in better than imagined.

1 June

We went to Museu de Arte de S.Paulo (MASP), which provided free entrance on Tuesdays. They had an exhibition of collages from the German artist Max Ernst with a well written breakdown of his life and interpretations of the different pieces. We made our own interpretations of his work, which largely consisted of critiques of the abuse of power by the elite (represented by the French bourgoise), but also of men over woman.

We left the museum inspired by Ernst’s creativity, his well-traveled and bohemian lifestyle indicating that creative lifestyle design is nothing new. Some people have been traveling the world and making a living doing what they love for a long time already.

2 June

The sight of vagrants and destitutes lying on the dirty sidewalk and public spaces in the center of Sao Paulo was quite jarring, and something I was not used to seeing anymore since spending a year and half in Germany. They were in all likelyhood just sleeping, but one couldn’t be sure. One might have been lying there dead with people just passing by unaware.

We went to the famous Galeria do Rock, a 6 story mall dedicated to Hard Rock and Heavy Metal culture, where we met Aggnes, who works with Sepultura and is a volunteer for the Sea Sheperd organisation.

Up at the top of the Banespa building, one gets an impression of how big this city really is. Buildings and skyskrapers dot the landscape extending as far as the eye can see.

June

On Thursday we met my friend Witek (who coincidentally happens to be in Sao Paulo) and his Brazilian friend Selena. We went together to Parque Ibirapuera, the biggest park in Sao Paulo. It was a day of relaxing in the park and drinking coconut water while Witek and I discussed some of the parallels between South Africa and Brazil. Later Selena took us in her car for a trip around the city, explaining the sites, presenting it to us in a completely different light.

4 June, Rio de Janeiro

We take the bus, Expresso do Sul, to Rio de Janeiro. We opt for Executivo, the mid range option and are pleasantly surprised with a very modern and comfortable interior.

Upon arrival in Rio, we are a bit surprised with how poor and unkempt the neighbourhoods look. We drive past favela after favela and everywhere are scribbles and grafiti. A lot of the grafiti in Sao Paulo were really very impressive murals, but our first impression of Rio is that it’s mostly gang signs and scribbles. Every inch seems to be tagged. Sometimes places that are seemingly impossible to reach are marked, such as the 3rd stories of buildings and on the sides of bridges.

Later at the hostel we had Caipirinha night, lots of fun and a good time to meet other travelers. I spent some time with an American traveler of (North-)Korean descent, discussing the expressiveness of the German language. He was suitably impressed ;)

5 June

We took a walk to Botafogo and Flamengo beach with Niall, an Irishman we met the night before. The weather is pretty crap, windy and rainy, but we’re all in a comparatively good mood and it’s fun to explore Rio for the first time.

On Botafogo beach, Brazillians are playing American football (which was a surprise) and an old guy is doing amazing tricks with a soccerball.

We again drink coconut water, agua de coco, this time with a straw straight out of a cooled coconut.

Our backpackers was nice at first glance, but we soon realize that we are two of almost 40 people squeezed into the place.

6 June

Today is Sunday and we plan to go to the Hippie market in Ipanema. First we walk to Copacabana beach. It’s still a bit windy but it’s nice and warm and there are lots of people on the beach.

So far Rio seemed quite poor and unkempt, but in Copacabana beach it’s like a world apart and we can see why it’s so famous. The official uniform for Brazillian men seems to be boardshorts and nothing else. The women wear hotpants with small tops or small dresses.

The Hippie market was a very nice surprise. There were lots of beatiful handmade leatherware, such as deocrated world maps, bowls, belts and armbands. Also jewellery and dresses. We saw very little kitsch tourist trinkets, and the market seemed to live up to its Hippie name.

7 June

We want to walk to Tijucaforest, but don’t reach it and instead go to the Botanic gardens. The botanic gardens were 5 Reais to enter, which in our opinion didn’t make it worthwhile, as it was relatively boring. The whole of the not very big park was filled with workers cleaning up and building paths, to the point of absurdity. We felt that the park already had too many paths and that it was just like any other park, and not what we expected from botanical gardens.

That night we bought what we thought was red wine. Upon drinking it, we realised though that it was very sweet and must have been some mix. It tasted like a mix of red wine and Fanta grape. Weird…

8 June

We took a one day Rio tour, and although it was nice and sunny the whole day, the Christ Redeemer statue was engulfed in a big fluffy cloud. We went up regardless but the view were not nearly as spectacular as they would be on a clear day.

We also went to the charming old neighbourhood of Santa Theresa and later to Lapa, were we met the artist who covered a whole flight of stairs in tiles collected from all over the world. Lots from Germany and a few from South Africa as well, unfortunately all of them quite kitsch.

9 June, Itacaré

We leave to Itacaré on Sao Geraldo busline. I made an effort to stay awake and take in the landscape. Deforestation was evident everywhere and for me quite sad. Nonetheless, the landscape was striking and pretty.

10 June

We arrive in Itacaréafter a 24 hour bus trip. Itacaré is a small town on the coast of Bahia, with 14 beatiful, pristine beaches popular among hippies and surfers. Each one has its own character, with some only reachable after hiking through the forest, revealing a secluded tropical paradise, filled with trees coconuts and cute white crabs.

The houses are small and modest, but gone are the scribbles and grafiti which were so commonplace in Rio. Everywhere locals are working, digging, mixing sement in the street and painting the sidewalk stones red white and blue. Lines with confetti and Brazillian flags span the narrow cobblestone streets and a festive atmosphere is in the air. In June the inhabitants of Itacaré have a month long Sao Joao religious festival, not to mention the World cup getting underway.

Children are everywhere, playing in the streets and whenever there is a patch of grass free, they will be playing football. The neighbourhood were we stay is a bit more rough than the touristy main street, but we feel safe and there is a definite community spirit in the air. The people seem to live outdoors and with one another, not closed off in their own houses as people in suburbs tend to do.

As a South African, it was very noticable how mixed Brazil is and how easily and comfortably the different people and races interact with one another. Mixed race couples are common and I hope that South Africans could one day take a leaf out of Brazils book and also learn to live so relaxed and comfortably with one another.

12 June

I watched a Brazilian dating show tonight, and it was quite a funny experience. The contestants start kissing within 5 minutes of meeting each other, and every now and then, everyone is rounded up and made to show off their dancing skills, with especially the men going over the top, showing their abs and striking rediculous poses.

One thing that was remarkable tongight, was that although Brazil is such a ethnically diverse culture, when watching television you would think that it is mostly white with some brown sprinkled here and there. Television presenters are on average much more paler than the faces seen in the streets and towns.

13 June

Everyone in the hostel, about 8 people, started partying with a few beers at the hostel and we then went out to town for a jol. The Itacaré nightlife was great, with the alternative/Hippie vibe and influence omnipresent. We ended up at a place on the beach where a Reggae band played all the classics. Capirinhas were everywhere and it was a great atmosphere and ambiance for Manuela and I discuss our trip and how it was going so far. I felt a sense of euphoria, and again realised that life can be as exiting or boring as you want it to be, and that the possibilities for excitement and fulfillment are endless.

14 June

Today we got up early to walk to the end of town and then hiked for some 50 minutes through Atlantic rainforest to get to Prainha (little beach), one of the prettiest beaches in the area.

On the way we passed a small waterfall and a smaller rocky beach with a beatiful view.

Prainha was practically deserted except for two other couples and some surfers, including a Brazillian couple we met two nights before.

It was a lovely way to spend the day in the Bahian nature and at a pristine beach, and best of all, it did’t cost us a cent to get there.

That night, we suggested that we have a barbeque at the hostel, and Marialena, the cleaning lady at the hostel, sent a young boy to buy some fresh fish for us. She showed me how to clean and prepare them and then I grilled them with some meat, potatoes and aubergine all together on the fire.

Afterwards we chatted the night away about life, traveling and Brazil. We received a lot of insight from Niels, a Portuguese speaking Dutchman, about the politics, laws, police and culture in Brasil.

16 June

This night we went to Jungle bar to watch the local Capoeira school showing off their moves. It was an unforgettable experience. Everybody was in peak physical condition, and some of the older guys were quick as hell. The teacher is a guy called Jamaika, apparently in the late 40s, but still looking very fit.

I couldn’t help but think of the slaves who in their meagre spare time created and practiced the art of Capoeira. The slave masters were wary of anything that might have made the slaves a threat, so Capeoiera was disguised to on the face of it appear as a dance, perpetually accompanied with distinctly African singing and music, with the more dangerous moves only practiced when they were alone. Often I was reminded of breakdancing, although this martial art is much much older.

Today, the descendents of those original slaves still practice their martial art, and the descendents and relatives of the slave masters are honoured to become their understudies.

17 June

We traveled today with Fausto, Niels and Helena to the more secluded beaches of Havaizinho (little Hawai) and Engenhoca. You need to walk for about half an hour through the forest to reach Havaizinho, and then another 20 minutes to reach Engenhoca. Both beaches are beautiful with Havaizinha being our favourite. Engenhoca was a pleasure to swim in, with nice waves but very little undercurrent. We ended the day back at Itacaré on a hill at La Choncha beach watching the sunset. Just another day in paradise. :)

19 June

We had an amazing time in Itacaré, spending 9 days in total. Steve at Buddys was an awesome host and the staff (especially Daniel) were incredibly friendly, making us feel completely at home. We will leave with heavy hearts but fond memories.

20 June, Morro de Sao Paulo

We woke up early to catch the bus from Itacaré to Valença. The bus departs numerous times each day, but Brazil was playing in a World cup match that day and we’ve heard stories of the whole country coming to a standstill during football matches, so we decided to play it safe.

From Valença we took the speedboat (pronounced “speegy boachy” by the locals) ferry to the Island of Morro de Sao Paulo.

The water was opaque with such smooth waves that they reminded me of the simulated waves in a computer game. I’ve been reading about quantum mechanics and the fact that certain physical attributes are left undetermined (uncomputed?) until some observer chooses to observe them. Lately the view of the universe as a quantum computer seems to make more and more sense to me. Things are left uncomputed as quantum waves of probability, until (and if) the first observer chooses to observe them.

Upon arriving at Morro de Sao Paulo, we passed through a lovely arched colonial gate. In the afternoon in winter the sun can be seen setting through its opening.

Morro de Sao Paulo has no cars (none that I’m aware of). That being said, the first thing you are being greeted with upon entering through the gate (aside from the official charging a R$10 land tax) is some guy offering a taxi. The taxi it turns out, is a wheelbarrow, in which they load your baggage and then push for you through the sandy streets, stopping at the Pousadas while you haggle with the owners.

21 June

Morro de Sao Paulo is nice, the weather is perfect and it has it’s own charm, but it’s quite a bit more touristy than Itacaré. Everyone wants a piece of the pie, and you sometimes feel objectified as a walking wallet.

22 June

We took a trip today to another beach on the island, where we smeared some magic mud on us. All information we got was “Remédico” (pronounced: Hermaigico, ‘ai’ as in ‘fair’) and “Medinicional”, so were still not sure what it’s good for. My skin did however feel quite smooth afterwards.

We then left to the pristine island of Ponta de Corraes, apparantly with a 70km stretch of untouched coastline. Unfortunately the terrible polluting effect of plastic was abundantly clear in the large amount of bottles, caps and other garbage washed up on the shore.

23 June

We tried to hike to a fountain on the island, took a wrong turn and ended up crossing the whole island, until we reached the beach on the other side. We hiked a bit through the jungle, and upon reaching a small village, the locals enthusiastically greeted us and even offered us some of their food.

At night there was a Sao João celebration, with a live reggae band and groups in colourful dresses doing traditional dances.

Earlier, during the day, Manuela found a ripe, intact coconut, which we took home and placed in the fridge. That night we drank the chilled coconut water and ate coconut flesh from a smalled “seco” (dry) coconut.

24 June

We left before noon on a catamarán to Salvador. It’s almost impossible to not get seasick while sitting inside, but if you go stand outside in the front and stare out into the endless sea, then the nausea goes away and the fun of sailing takes over. I tried to imagine the trials and hardships the Rennaisance era European sailors had to overcome, crossing the oceans in search of a traderoute to India.

It was a holiday today, and Salvador’s city center was empty. We went straight to the airport and spent 5 hours waiting for our flight to Natal.

We were hosted by a couchsurfer named Cyrus, who was even kind enough to pick us up at the airport.

Friday 25 June, Natal

We went walking in the area of Alecrím, Natal, and shown the town square of the original Portuguese settlement. Cyrus told us that the Portuguese generally created their settlements in the same fashion. A government building, followed by a jail and then a church, all three surrounding a town square. Converted Indians would then build their houses around the town square and extending outwards.

We spent the afternoon with a group of friends watching the Brazil, Portugal game ending in a disappointing 0-0 draw. Afterwards we all sat around the tree, with the group spontaneously breaking out into percussion and song. For me it was culturally eye opening, to see a completely racially mixed group of friends naturally erupting into a distinctively African flavoured song. These people have completely embraced the African heritage of the slaves and made it part of their Brazillian culture. It is the complete opposite of South Africa, were the different cultures were forcefully kept seperate, never developing a common sense of culture or togetherness and where white people still often reject anything perceived as “black” African, while patting themselves on the back for being so “African” themselves. This is however, slowly changing in SA, with people becoming more integrated. Brazil is just decades (or centuries) ahead.

We then went to the beach and Fortaleza dos Reis Magos (“Fort of the three wise men”), which was the initial Porguese construction in Natal. The fort was built in the 18th century, later captured by the Dutch, and then recaptured by Portuguese and Native Indian soldiers.

That night, we went to a gathering organised by Cyros’ percussion group. The party was inside a mostly bare building, with a bar in the corner serving beer, water and cola. The drummers play on halved plastic water and trash barrels. One guy was playing on an empty gas canister. Everyone was dancing and a girl in oriental dress danced in the middle, reminiscent of belly dancing. Many of the songs were religious songs of the Afro-Brazillian religion Candomblé.

After about 15 minutes, the whole procession of drummers and crowd (about 30 to 40 people) went outside into the street (all the while drumming and maintaining the rhythm). Some passers by joined the crowed, some dancing and one took a large carton box over his body and made strange animalistic dance moves. It was for us complete magic, being included in a strange type of social ritual, with no idea of what will happen next and with no understanding of the their language.

Outside we were entertained by a clown performance, with passing busses more than once slowing down and/or coming to a stop while the driver and inhabitants stare at the show.

Everyone went back inside the house/building and inside continued to make music and dance. There were no pretensions, no designer clothes, cars or any other methods for portraying exclusivity or arrogance and we as two foreigners felt honoured but also welcome.

Everyone danced with everyone else, and dance they could. Often the dancing couples appeared quite intimate, with hips moving seamlessly to the rhythm, only to rearrange into new couples for the next song. I spent a lot of time entranced, just watching the people move. Manuela made the observation that Brazillians seem to be so much more body orientated than Europeans, who “live more in their minds” and I completely agree. During the day they wear skimpy clothing, are very active playing, competing, gesturing and generally moving their bodies about, and afterwards they spend the night dancing and embracing. Touches of the body are common; hugs, brushes, pats and often for single women, grabs.

Saturday 26 June

We take the bus to Pipa, a popular small coastal town about 40km south of Natal.

Monday 28 June, Pipa

Sanctuário Ecológico

This morning we left early to go to Sanctuário Ecológico. A private conservation area just outside Pipa. We met up with Cyro, two Argentinian friends and a local couple and hiked the trails in the park.

The park was beautifully preserved and maintained, with lots of signposts, markers and designated viewpoints with benches and rails. There was even a small labyrinth in next to one of the trails. Cyro informed us that the sanctuary is privately owned, by a Brit (named Dave AFAIK), that he sponsored local turtle conservation projects and also built an English school in the area. We were very impressed and inspired with this person’s effort and contribution in helping to preserve some of the local vegetation (instead of exploiting it or building a hotel complex).

Brazil and the South African struggle

Before the Brazil vs Chile match, they showed some of the public viewing areas in Brazil. Sao Paulo, Copacobana beach in Rio and also Salvador. I was interested to see a big African National Congress flag being waved in the Salvadorian crowd, although not completly surprised. I’ve noticed streets named after Nelson Mandela, grafiti of him on the walls and have heard a Portuguese Brazilian song (one of many apparantly) about him being sung the other night during the live performance in Natal.

Brazillians strongly identified with the plight of oppressed non-white South-Africans during Apartheid, and were enthusiastic supporters of Bafana Bafana during their stint in the world cup. It’s fascinating to witness how strongly the South African struggle still resonates with these people. On a different continent, but with still with a lot in common.

Later a thought occurred to me. Although there seems to be similar negative sentiment towards Israel’s occupation of the Palistinian territories, (in many places in the world, not notably strong in Brazil though), that for all the Palistinians’ tribulations, they lack a Nelson Mandela figure.

Tuesday 29 June

Praia dos Golfinhos

Initially the day did not seem very promising, with intermittent rain for most of the morning. Later, however, the weather cleared and we went to Praia dos Golfinhos (Dolphin’s beach), to go for a swim and relax in the sun.

Praia dos Golfinhos is a well-known hang-out for dolphins and we were hopeful to meet some of these fascinating creatures. The about 1 km stretch of beach was mostly deserted with only about 8 other people there, but in the sea motorboats carrying hopeful dolphin watching tourists were regularly making the rounds coming into the bay. No dolphins appeared though, and the tourists we’re distracted by the opportunity to swim near the boat and by the the tour operators who took some (no doubt facebook-profile) photos of them on the boat.

I felt a bit irritated by the boats, imagining the loud motors to be distracting or unnerving to the dolphins. Eventually the boats left, and we decided to take a dip in the sea. The waves are very calm, and the water warm, so much that we ventured deep enough to not be able to stand anymore.

For a second I thought I saw someone swimming quite a bit more ahead. Then, half a minute later, we saw two dolphins coming out of the water, just about 12 meters ahead of us. It was a completely magical event, and I felt a sense of joy and excitement unique to this type of experience. We continued to stay in the water for about 15 more minutes and saw the two dolphins appear about 8 more times. Later, when we walked home, they appeared again, with a boat in the background, heading back to the harbour, completely oblivious to their pr:esence.

Wednesday 30 June

Projeito dos Criancas (The childrens’ project)

The previous night we met a fellow traveler, who after seeing me practice WingRevolutionmartial arts (which I learned during my stay in Germany) told me about open Capoeira lessons being held the following morning. This was one of these very fortunate coincidences that seem to appear so often during our travels. I’ve been wanting to take some Capoeira lessons but was intimidated by the language barrier and the amount of time it would take to learn anything meaningful. This particular school was literally across our Pousada, and I thought that this was an opportunity not to miss.

So that morning, I got up early and went to the Capoeira school to see what’s up. The place was completely empty, but I decided to go in anyway to see if I could find someone. Inside I met Miguel, who was one of the instructors, and as luck would have it, he spoke English. There and then we had an hour of personal instruction. He showed me the main moves, showed me how they play the main musical instrument and told me some stories about Capoeira.

That night was another hour and half group lesson, with the local Capoeira master, some of the local kids, and a few really good fighters. For the second half of the lesson, everyone gathered in a circle, clapping hands on the beat of drums and instruments with the students (and trainers) taking turns to show of their moves and play-fight against one another. The better fighters were very popular oppononents with everyone wanting to duel against them.

After some time, I was called out to duel in the ring, with this being my first day of learning Capoeira. It was already a very humbling experience to first overcome my internal intertia, fears and doubts and join the school, but now I was also expected to show of the moves that I learned. The master was already in the ring, having dueled with a previous student, and we took hands locked hands (a courtesy before starting the duel) and then started to fight/dance. It was an intense, exciting and eventually euphoric experience and afterwards I gained a new appreciation for the benefits of opening yourself up to experiences which might seem intimidating at first.

The Capoeira (and on alternate days Hip-Hop) school, Projeto das Criancas, was started by a Norwegian woman named Helga as a means of community education and upliftment by training the local children free of charge.

Wrap up

That’s it for the first month of the trip. I’m typing this in a small internet shop in Canoa Quebrada, near Fortaleza in Natal, where we are currently staying. Wednesday the 7th we’re flying to Campo Grande and the Pantanal area. I’ll post another blog entry for July sometime in the beginning of August.

Bom dia!

Backpacking through South America

Wed, 26 May 2010 00:00:00 GMT

On Sunday 30th May I’ll be flying to Sao Paulo, Brazil for a 4 month adventure backpacking through South America.

This trip been more than 14 months in the making, involving lots of preparation, saving, buying gear, receiving inoculations, filling in forms, doing research, selling furniture, dealing with bureaucracy and other non-sexy admin that people don’t generally associate with backpacking through exotic lands. The amount of effort involved has been compounded by the fact that we are afterwards moving back to South Africa.

That being said, it’s all finally coming to a close and my girlfriend and I are slowly realising that we are nearing the start of a great adventure.

Needless to say, I will during this time be taking a hiatus from working on Plone and the packages that I maintain. I’m taking a netbook along to check emails, so I won’t be completely unreachable, but I cannot guarantee timely replies and/or updates.

Most of the packages I authored were done for Syslab, (usually under the ‘slc’ namespace), and the guys there would be able to help with merging patches and the like.

To Alex Pilz and the gang at Syslab, I would like to say, thanks for a wonderful year and a half in Germany! I couldn’t have asked for nicer and more competent people to work with and I am happy to have become good friends with all of you.

On Sunday begins a great new adventure. Hopefully I’ll be able to have a beer with some of the Plonistas, Pythonistas, Zopatistas and other -istas in South America!

Cheers!

Babble: Instant messaging for Plone

Thu, 29 Apr 2010 00:00:00 GMT

babble.client is new a chat client for Plone, similar in style to Google Chat and Facebook Chat.

I’ve been having a lot of fun at Syslab recently working on an Instant Messaging client and server combo for Plone. The client app is a Plone add-on (for both versions 3 and 4), which makes use of JQuery, polling and Ajax techniques, to try and create a Facebook/Google like chatting experience. Thanks to the excellent cross-browser support of JQuery, I didn’t have to spend a lot of time getting it to work on Internet Explorer.

The messaging server is a pure Zope 2 application and was designed to be a standalone and independently usable service, unaware of any client side solution. Its design is based on an earlier version that I had worked on at Upfront Systems, and for which Roché Compaan deserves a lot of of the credit.

Great, I would like to try out a DEMO!

For anyone curious to check it out, I’ve created a demo that automatically sets up a working Babble environment.

Please take note that this demo is not intended to be included in an existing Plone environment. It’s a separate buildout with a Zeo server and two clients (running under Supervisord) and a policy egg that installs and configures the message server and client.

If you include the babble.demo egg in another buildout, the Zeo setup and instance ports will most likely be different, requiring you to set up the portal_chat tool by yourself, thereby defeating the purpose of having a self-configuring demo.

Cool, I love to read docs, where are they?

Good! I’ve created Sphinx documentation that you may peruse for your reading pleasure.

What’s in a name?

For lack (and want) of an awesome name, I googled for Synonym + Chat and relatively quickly settled on Babble, only later realising that I chose a name that specifically refers to meaningless babby chatter and more generally functions as a less-than-complimentary description of someone’s incoherent ramblings.

Babble:

utter meaningless sounds, like a baby, or utter in an incoherent way; “The old man is only babbling—don’t pay attention”
to talk foolishly; “The two women babbled and crooned at the baby”
ripple: flow in an irregular current with a bubbling noise; “babbling brooks”
spill the beans: divulge confidential information or secrets; “Be careful—his secretary talks”
gibberish resembling the sounds of a baby

Babblebabblebabblebbable

That’s it. If you have encountered any problems or bugs while setting up and using *Babble*, feel free to contact me. Suggestions are also welcome.

Thanks to Nikel Weiss, who helped me with some testing and provided helpful suggestions and feedback. Thanks also to Alex Pilz for not deciding to use a Freenode solution ;)

Examples:

By itself, Babble provides a special Who’s online? portlet, to show currently active users. When you click on a user, a chat window will pop up at the bottom.

Babble is similar in looks to Facebook’s chat client. If you liked the bottom panel that Facebook sported (until recently), then make sure to also install actionbar.babble. This egg provides Babble integration for the ActionBar panel.

See the screenshot below for an example:

Using extract_messages in your Pylons/Python egg with a ./src directory layout

Wed, 24 Mar 2010 00:00:00 GMT

If you want to have your egg’s package contents inside a ‘src’ folder, you’ll need to tweak the configuration files a bit. Here’s what I’ve learned…

I’ve been spending a lot of time trying to figure out something quite simple, so I’ll write a blog post about it to hopefully save someone else some time with this.

I’m in the process of creating a Pylons egg. For illustration purposes, I’ll call it MyEgg.

The egg is created as follows:

paster create --template=pylons MyEgg

The default egg directory looks like this:

MyEgg 
  docs 
  myegg
    config
    controllers
    lib
    model
    public
    templates
    tests
MyEgg.egg-info

I prefer however to put the package contents myegg inside a ./src directory. That way, I can more easily use zc.buildout to manage MyEgg and its dependencies. I can then use buildout to manage and develop this egg’s dependencies by also putting them in ./src.

After almost 3 years of using buildout on a daily basis, this layout just fits my brain better.

The new folder structure would now look like this:

MyEgg 
  docs 
  src   
    myegg     
      config     
      controllers     
      lib     
      model     
      public     
      templates     
      tests 
MyEgg.egg-info

Updating your setup.py script:

To make Distutils aware of the new directory layout, you also need to add the following two lines to the setup method call in your setup.py script:

packages=find_packages('src', exclude=['ez_setup']),
package_dir={'': 'src'},

Extracting translatable strings from your python files:

To generate a *.pot file that contains all the translateable strings in your code, you run the following command:

python setup.py extract_messages

To have the new directory structure (with ./src) and still be able to use extract_messages to internationalize your egg takes a bit more tweaking.

Firstly, in the setup.cfg file, you need to append src to the paths in the different sections. For example, in theextract_messages section, it would look like this:

[extract_messages]
add_comments = TRANSLATORS:
output_file = src/myegg/i18n/myegg.pot
width = 80

NOTE: there are other commands (such as compile_catalog) that also need to be run later as part of enabling I18N support for your package. For the purpose of this blog post, I’m concentrating on extract_messages, but the above change in the setup.cfg file applies to the other sections as well. For more info on I18N in Pylons, refer to the Pylons Book.

And now for the problem I was struggling with ;)

The problem I encountered with this layout was that the extract_messages command (courtesy of Babel) that is used to extract translatable strings for I18N support, now didn’t find my python files anymore.

python setup.py extract_messagesrunning extract_messageswriting PO
template src/myegg/i18n/myegg.pot

If you look at a Pylon project’s setup.py files (if you created it with the paster’s pylons template), you will see the following lines commented out:

##message_extractors = {'myegg': [
## ('**.py', 'python', None),
## ('templates/**.mako', 'mako', None),
## ('public/**', 'ignore', None)]},

The solution to this problem was to uncomment the message_extractors parameter in setup.py, and to tell it to look in the src directory:

message_extractors = {'src': [
   ('**.py', 'python', None), ('templates/**.mako', 'mako',
    None), ('public/**', 'ignore', None)
]},

Now the extract_messages command works and creates a myegg.pot file that contains all the translatable strings in my app.

Here’s the output (shortened for brevity):

$ python setup.py extract_messages running extract_messages
extracting messages from src/__init__.py extracting messages from
src/myegg/__init__.py .... extracting messages from
src/myegg/tests/functional/__init__.py writing PO template file to
src/myegg/i18n/myegg.pot

Hope this helps another Pylons, Distutils or I18N newbie to save some time.

Why you should open source your code

Thu, 18 Mar 2010 00:00:00 GMT

This blog post is based upon my experience in developing add-ons for the Plone open source content management system. It’s therefore mostly aimed at the Plone community, although some of the insights mentioned might also apply to other communities and to web-development in general.

In the last 15 months, I have released 15 Plone add-on products as open-source python eggs, available for download on Plone.org and Pypi, the python package repository. In the preceding 4 years of Plone development, I released only a single product.

In this process, I have witnessed a personal metamorphosis. I can draw a relatively clear line between ‘Ye Olde Worlde’ web-developer who was set in his ways and narrow minded in his methods, and the new! improved! open source developer.

I’ve seen the transformative power of (the act of) open sourcing and although I’m sure that a lot of what I have learned is already old hat to many developers, I would like to share the insights I have gained from this experience. Many developers have been made aware of the inherent advantages of collaborating in an open-source community, but to take part in it yourself and to experience the inherent benefits first hand, is the only way of developing a true conviction for the appropriateness of open source collaboration.

The fact is, that you have a lot to gain by engaging with the community, through collaborating, sharing your ideas and making your code public.

Open source everything that might be useful to others

It’s relatively common practice in the Plone community to develop a so-called ‘policy package’ for a new project. The policy package should tie everything together, configure the Plone application to the client’s needs, and install all the other add-ons.

Additionally, a theme egg for implementing a custom theme is usually also required. These two packages, together with a collection of third-party and in-house developed packages (each responsible for a specific feature set), on top of Plone itself, constitutes the software product presented to the client.

Before I was introduced to the idea of a policy egg by Martin Aspeli and others who championed the idea, I would usually when given an assignment, develop only one Plone add-on product for it. This add-on then contained all the custom software code that I wrote for this specific client and/or product, regardless of whether certain parts of it would also be beneficial to other people or projects.

This is of course not the proper way to develop a sustainable, reusable and configurable solution for a client. Instead, every piece of functionality that constitutes a logical grouping of features and that could theoretically be reused somewhere else, should be packaged separately in its own python egg.

And here’s the catch: you should ideally open-source everything except the policy package. Even the theme egg should be considered for open sourcing.

The policy package serves as the puppetmaster, that ties these different pieces together to make the integrated unit (or puppet show). It is the proprietary configuration that contains the client specific implementation details, the client’s copyrighted logo and images, the site copy and everything else that makes the site unique to the client. It’s the secret sauce or the kernel’s 11 herbs and spices, in a matter of speaking.

With regards to the rest of the custom developed eggs in your project, release them as open source eggs on Pypi! You have a lot to gain by doing so.

Let me elaborate on why…

Advantages

Open sourcing your code forces you to be a better programmer

The implicit mental adjustment coming from the knowledge that you are writing source code which will be read and modified by others, is already enough to considerably enhance the quality of the code that you will write.

Why? Because of the added social pressure.

Glossophobia, the fear of public speaking, rumoured to be the most common fear in the minds of people, is a consequence of the fear of public humiliation.

Nobody wants to be humiliated publicly. Not for being a poor speaker and not for writing bad source code. The fact is, that you will make more of an effort to write clear and legible code that follows best practices when you are writing for an audience (the community of developers) than when you are writing for personal or ‘in-house’ consumption.

Before I started writing code with the intention of open-sourcing it, I would be much more inclined to include quick and dirty hacks to “save time” or to “simplify” things. Ironically, quick hacks such as hardcoding a value or overriding a template instead of putting it in a viewlet will in the long run, waste time and overcomplicate things.

A quick real world example: Your boss or client wants a facebook style action panel on the bottom of a site.

Do you:

Decide not to open source this functionality, and instead add the styles and templates to the theme egg while hardcoding the links on the panel because it’s quick and easy?
Decide from the getgo that you will open source this and therefore create a separate egg for this piece of functionality, that adheres to best practices to make it reusable, extendable and configurable?

The second option might take longer initially, but its benefits pay off in the long run, especially when the client requires technical support.

The community provides an additional level of quality control

I’m careful here to add the word ‘additional’ since you should not solely rely on the community to test your code for you. On the contrary, other people will be reluctant to try out your products if you don’t put in the initial effort or write tests and doing quality control yourself.

That being said, granted that you have released a quality package, you will receive an additional layer of community powered quality control by virtue of the fact that the developers and integrators using your code will let you know when they run into bugs, and often even fix them for you! “With many eyes, all bugs are shallow.”

The community provides new perspectives, ideas and enhancements.

This past year, various developers have contacted me with suggestions, ideas and improvements to some of the packages that I have released. Some have made a meaningful amount of effort to explain why a certain solution might be better, and in the process have provided me with a free education and broadened the scope of my Plone dev knowledge.

Overall, the quality of the packages that I have released, has been improved through the sustained contributions of other developers in the Plone community. This is after all one of the main advantages of open source, seen from a purely pragmatic point of view. To witness this process in action and with code that you yourself have contributed, is very satisfying.

Social currency (A social C.V or Résumé)

Open source contributions, being easily searchable and in the public domain, form an authorative record of the depth and quality of an individual or company’s output.

Compared to the sections traditionally contained in a Curriculum Vitae (Résumé), your open source contributions form your work experience section, while their contributors and users serve as references. Sustained open source contributions coupled with helpful and courteous interaction with the community, provides a more authoritive record of ability and character than any CV ever could.

Suggestions:

I’d like to round up this post by making some suggestions which I consider will better the chances of your product not only being used by the community, but also receiving code improvements and bugfixes.

Don’t host in-house:

Use the collective repository to host your Plone source code. We’re lucky to have it, and that’s what it’s there for. Alternatively, use Google’s project hosting or github. Although, in my opinion, it doesn’t make much sense to call something collective.* if it’s not hosted on the collective repo (I know others disagree with this).

You want the barrier to entry for participation in your code to be as low as possible. That means making it as easy as possible to quickly provide improvements and bugfixes.

If you host your open source products privately, you might gain users, but you will have far fewer contributors willing to fix and enhance your code, simply because they’re not willing to jump through hoops to be able to do so. Private or walled off hosting discourages contributions and promotes forking. Which would be a forking shame (sorry, I had to).

Start your project in the open:

Don’t develop your packages privately, waiting for them to be “good enough” to be added the to a public repository.

Start from the beginning on developing publicly on a repository such as the collective. This will ensure that you stay aware of the fact that you are writing code for others and not just for yourself. As mentioned before, when you consider the fact that others will have to read and digest your code, you will more carefully consider your method, class and variable names. Do they make sense? Are they consistent? Is the code explicit and readable? etc.

Release your Plone eggs on both Pypi and plone.org:

Some users prefer to search for Plone add-ons via plone.org, while others use Pypi. Cater to both to maximise your visibility. Plone.org goes a step further by among other things allowing you to upload a screenshot and your company’s logo.

Tools such as zest.releaser or jarn.mkrelease make releasing eggs a breeze. Use them! :)

Conclusion:

When it comes to open sourcing your code, the fear might arise, that by making your code available for others to use and adapt, you might in some way lower your chances of monetizing the code. For people living with a mindset of scarcity, in which ideas and creations must be guarded closely to safeguard their success and to prevent others from “stealing” or profiteering from them, it might be difficult to make the paradigm shift necessary, towards a mindset of abundance, opportunity, sharing and collaboration.

The fact is, that you have a lot to gain by engaging with the community, through collaborating, sharing your ideas and making your code public. ;)

Archetypes: Schema extending an object only inside a specific folder

Tue, 26 Jan 2010 00:00:00 GMT

It’s possible to register schema-extenders (and modifiers) for only certain subsections (folders) of your Plone site. It’s relatively simple, and all you need is to do is add a local site-manager to the folder and then register the adapters against it.

This blogpost assumes that you are already familiar with Plone’s Archetypes schema extensions and provides more advanced usage examples. If you’ve got no idea what I’m talking about but want to learn more, checkout the pypi page here first.

Introduction

By using simple adapters (known as schema-extenders), it’s possible to extend or modify the schemas of your Archetypes objects without actually getting your hands dirty. One advantage of using adapters to modify an Archetype’s schema is that it allows multiple Plone add-on products to modify the schema in an extendible and non-invasive way, without using any subclassing. You don’t need to touch the original code at all.

The Zope Component Architecture (ZCA) provides the functionality that makes all this happen. Adapters are registered (via ZCML or python) and are then kept track of by a ZCA component known as a site manager.

A normal Plone site provides a global site manager located at the root of the site. This site manager keeps track of all the registered adapters in the site. When you schema-extend an object, it’s the site manager that is queried for the possible adapters (schema-extenders and schema-modifiers) that apply to this object.

You don’t however have to have only one site-manager. You can easily create a local site-manager in a folder elsewhere in your site, and this allows you to have a folder specific registry of ZCA adapters and utilities. Concerning schema-extension, this means that you have schema-extenders that apply only in certain subsections (folders) of your site.

The nitty gritty

So, how do we do this? First we need to create the schemaextender that we will be applying only on a local folder. The following code describes this extender object. To keep it simple, it adds one extra field to the existing object’s schema, namely the ExtensionStringField.

import zope.interface from Products.Archetypes
import atapi from archetypes.schemaextender.field
import ExtensionField from archetypes.schemaextender.interfaces
import ISchemaExtenderclass

ExtensionStringField(ExtensionField, atapi.StringField):

class LocalExtender(object):

    zope.interface.implements(ISchemaExtender)
    _fields = [ExtensionStringField('MyLocalString', widget=atapi.StringWidget())]

    def __init__(self, context):
        self.context = context

    def getFields(self):
        return self._fields

Lets assume that we have now have a folder in which we want objects of a certain type to be extended by our extender above. To be able to do this, we first need to create a local site-manager inside this folder.

from five.localsitemanager import make_objectmanager_site

if not ISite.providedBy(folder):
    make_objectmanager_site(folder)

So, as mentioned, the code above will create a local site-manager inside your folder. How and when you call the above code is up to you. You can for example put it in an External Method and call it once, or you could create a subtype and have a reproducible way of creating local site-managers through the Plone UI.

Next, we have to register our schema-extender for a specific object this specific site-manager. Which kind of object it is, is determined by it’s interface(s). So, assuming we wanted to extend all the Documents in this folder, we would register our extender for the IATDocument interface.

from Products.ATContentTypes.interface import IATDocument
sm = folder.getSiteManager()
sm.registerAdapter(LocalExtender, (IATDocument,), ISchemaExtender, name=u'LocallyExtendedObject')

The first variable passed to registerAdapter, is our LocalExtender class, also known as the adapter factory. The factory is used to compute the adapter.
The second variable is a tuple containing a sequence of interfaces that the object must implement to make it a candidate for adaptation.
The third variable is the interface that the factory (schema-extender) must implement in order to make it a valid adpater in this context.
Lastly, the name is useful since it can be used to override adapters that applied to subtypes of an object.

For more info on local site-managers and registering ZCA components for them, see five.localsitemanager.localsitemaqnager.txt.

Using the Gmail SMTP server to send emails from Plone

Mon, 04 Jan 2010 00:00:00 GMT

Need to test emails on a local test instance, or just too lazy to set up your own mail server? Just use Google’s smtp server!

It is possible to use Gmail’s smtp server to send emails from your Plone site. It’s very simple and I’ll show you how.

You’ll of course need a Gmail account, get one here.

Setup

In your Plone site, log in as the administrator, then go to ‘Site Setup’ and then ‘Mail’.

Now in the form, give the following values:

SMTP server: smtp.gmail.com
SMTP port: 587
ESMTP username: Here you give your gmail email address, i.e john@gmail.com
ESMTP password: Give your gmail account password.
Site ‘From’ name: Use the default or whatever you prefer.
Site ‘From’ address: This is the reply-to address on the email. For users it would seem as if the email was sent from this address. It’s important that this address belongs to a user in the site, since it’s also the address to which emails will be sent when users contact the administrator via the ‘contact’ form.

Troubleshooting:

Make sure TLS is enabled. This dіdn’t work for me until I went to the ZMI (Site Setup > Zope Management interface), clicked on MailHost and uncheck the ’Disable TLS’ checkbox (the last option).
Also, make sure that your Python instance is compiled with SSL support.
Make sure that IMAP support is enabled for your Gmail account. When in Gmail, click on the ‘settings’ link at the top right, then the ‘Forwarding and POP/IMAP’ link and enable IMAP access.

Now, in your Plone site, click on the ‘contact’ link in your site and send yourself a congratulatory email!

Compiling Python with Readline support on Ubuntu Linux

Fri, 01 Jan 2010 00:00:00 GMT

Compiling Python on Ubuntu is often necessary for development purposes and for specific versions. Here I’ll show you in a few easy steps how to install your own Python instance into a folder in your home directory.

There are various advantages to compiling your own Python version, instead of using the system’s installed version.

Portability is one of them. By having a python instance in my home folder, I can easily install different versions of Ubuntu and just copy over the python files, thereby immediately and easily setting up my previous python environment.

Additionally, Zope versions prior to 2.12 require Python 2.4, which is not available on newer versions of Ubuntu.

Unfortunately, Readline support (i.e provides nifty things like the ability to see your command history via the up arrow) is not enabled by default when compiling your own custom python in Ubuntu. To be able to compile python and enable readline support, we

need to install some additional libraries.

jc@ubuntu:~$ sudo aptitude install build-essential libreadline5-dev

Now it’s time to download and extract the version of Python you would like to install.

<jc@ubuntu>:~$ mkdir -p ~/src/python <jc@ubuntu>:~$ cd ~/src/python/
<jc@ubuntu>:~/src/python$ wget http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tar.bz2
<jc@ubuntu>:~/src/python$ tar xjf Python-2.4.4.tar.bz2
<jc@ubuntu>:~/src/python$ cd Python-2.4.4/

We need to enable readline support by modifying the Setup.dist file. The following one-liner will do the trick:

jc@ubuntu:~/src/python/Python-2.4.4$ sed 's/#readline/readline/' Modules/Setup.dist > TMPFILE && mv TMPFILE Modules/Setup.dist

Next comes cmmi (configure, make, make install). I like to install Python in my home directory and would recommend it to anyone else. Hence the --prefix argument.

jc@ubuntu:~/src/python/Python-2.4.4$ ./configure --prefix=/home/jc/python/python-2.4.4/ && make && make install

Lastly, I make a sym-link to the new binary to make sure it’s in my execution PATH:

jc@ubuntu-12:59:~/$ ln -s /home/jc/python/python2.4/bin/python2.4 ~/bin/

You should now be able to call the python interpreter from the command line:

jc@ubuntu-13:01:~/bin$ python2.4
Python 2.4.6 (#2, Aug 19 2009, 17:13:56)
[GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

If you receive a command not found error, make sure that your ~/bin directory is in your PATH by editing your .bashrc file (note the leading .). For example, in my .bashrc file, I have the following PATH declaration:

PATH=$PATH:/home/jc/bin

That’s it! Best of luck with your new python install.