blog

Sprint Report: Merging Mockup and Patternslib

| categories: mockup, javascript, patternslib, austria, sprint, foss, plone

Alpine City Sprint, 21 to 26 January 2015

This is a report on what I did at the Plone Alpine City Sprint organized by Jens Klein and Christina Baumgartner in Innsbruck between 21 and 26 January 2015.

Firstly, I want to thank to them for organizing and hosting a great sprint and for being such friendly and generous hosts.

My goal for this sprint was to work on merging the Patternslib and Mockup Javascript frameworks and I'm happy to report that very good progress was made.

Long story short, the merge was successful and it's now possible to use patterns written for either project within a single framework.

Before I expand on how this was achieved during this sprint, I'll first provide some necessary background information to place things into context and to explain why this work was necessary.

What is Patternslib?

Patternslib brings webdesign and development together.

Patternslib's goal is to allow website designers to build rich interactive prototypes without having to write any Javascript. Instead the designer simply writes normal HTML and then adds dynamism and interactivity by adding special HTML classes and data-attributes to selected elements.

For example, here is a Pattern which injects a list of recent blog posts into a element in this page:

Click here to show recent blog posts

The declarative markup for this pattern looks as follows:

<section id="alpine-blog-injected">
    <a href="#portlet-recent-blogs"
       class="pat-inject"
       data-pat-inject="target: #alpine-blog-injected">

        Click here to show recent blog posts
    </a>
</section>

The dynamic behavior comes from the fact that I have the Patternslib Javascript library loaded in this page.

On pageload, Patternslib scans the DOM looking for elements which declare any of the already registered patterns. If it finds such an element, it invokes a Javacript module associated with that pattern.

In the above example, the <a> element declares that it wants the Inject pattern applied to it, by virtue of it having the pat-inject HTML class.

Patterns are configured with specific HTML5 data properties. In the example above, it is the data-pat-inject property which specifies that the target for injection is the #alpine-blog-injected element in the current page and the content being injected is specified by the href attribute of the <a> element. In this case, the href points to an anchor inside the current page, but it might just as well be a link to another page which will then get injected via Ajax.

More information on how to configure pat-inject can be found on the patternslib website.

Each pattern has a corresponding Javascript module which is often just a wrapper around an existing Javascript library (such as Parsley, Select2 or Pickadate) that properly defines the configuration options for the pattern and properly invokes or applies the 3rd party library.

So, in order to do all this, Patternslib can be broken down into the folowing components:

  • A registry which lists all the available patterns.
  • A scanner which scans the DOM to identify declared patterns.
  • A parser which parses matched DOM elements for configuration settings.
  • The individual Javascript modules which implement the different patterns.

What is Mockup?

Mockup, declarative interaction patterns for Plone..

Mockup was inspired by, and originally based upon, Patternslib and was meant to bring the power of declarative interaction patterns to Plone.

When Mockup started, Patternslib was undergoing significant refactoring and development and it was decided that Mockup should fork and go its own way.

What this means is that the 4 different components mentioned above, were all changed and due to these changes the Mockup project diverged from Patternslib and started developing in a similar but different direction.

So what's the problem?

While Mockup was being developed for the upcoming Plone 5 release, we at Syslab continued using and improving Patternslib in our projects.

Syslab built an intranet for the Star Alliance, which was based on a prototype design by Cornelis Kolbach, the conceptual creator of Patternslib. This design became the inspiration and blueprint for the Plone Intranet Consortium (PIC), which consists of 12 companies working together in a consortium to build an intranet solution on top of Plone.

So, the PIC are building a product using Patternslib, while Plone 5 itself is being built with Mockup, an incompatible fork of Patternslib.

This was clearly a bad situation because we now had:

  • Two incompatible Javascript frameworks being used with Plone.

    Not only were the two frameworks incompatible in the sense that patterns written for the one don't work on the other, but they could also not be used on the same page since the two implementations would compete with one another in invoking Javascript to act upon the same DOM elements.

  • Duplication of effort

    The same or similar patterns were being developed for both frameworks, and when one framework had a pattern which the other wanted, it could only be used after being modified such that it couldn't be used in the original framework anymore.

  • A splitting of the available workforce.

    Developers were either working on Mockup or Patternslib, but almost never on both, which meant that the expertise and experience of developers wasn't being shared between the two projects.

How could this be fixed?

To solve the 3 main problems mentioned above, we needed to merge the common elements of Mockup (specifically the registry, scanner and parser) back into Patternslib.

This will allow developers from both projects to work on the same codebase and enable us to use patterns from both projects together.

At the Alpine City Sprint in Innsbruck, I worked on achieving these goals.

Changes brought in by Mockup

After the fork, Mockup introduced various changes and features which set it apart from Patternslib.

In order to merge Mockup back into Patternslib, I studied these changes and with the help of others came up with strategies on what needed to be done.

Here are some differences and what was done about them:

Mockup allows patterns to also be configured via JSON, whereas Patternslib used a keyword:argument; format

A week before the sprint I added JSON parsing ability to the Patternslib parser, thereby resolving this difference.

Leaves first parsing versus root first parsing

Mockup parses the DOM from the outside in ("root first"), while Patternslib parses the DOM from the inside out ("leaves first").

According to Rok Garbas, the creator of Mockup, the outside-in parsing was done because it reduced complexity in the scanner and the individual patterns.

Wichert Akkerman who originally wrote the Patternslib scanner however provided IMO a very good reason why he chose "leaves first" DOM scanning:

If I remember correctly there are several patterns that rely on any changes in child nodes to have already been made.This is true for several reasons: 1) a pattern may want to attach event handlers to child nodes, which will break of those child nodes are later replaced, and 2) child nodes changing size might impact any measurements made earlier.

Indeed, while refactoring the Mockup code during the merge, I ran into just such a case where a pattern couldn't yet be initialized because patterns inside it weren't yet initialized. By turning around the order of DOM scanning, this problem was resolved and the code in that pattern could be simplified.

So, now after the merge, scanning is "leaves-first" for Mockup patterns as well.

Mockup patterns are extended from a Base object, very similar to how Backbone does it

Patternslib patterns on the other hand are simple Javascript objects without constructors.

The patternslib patterns are conceptually very simple and more explicit.

However, what I like about the Mockup approach is that you have a separate instance with its own private closure for each DOM element for which it is invoked.

After merging, we now effectively have two different methods for writing patterns for Patternslib. The original "vanilla" way, and the Mockup way.

The Mockup parser was completely rewritten

The Mockup parser looks nothing like the Patternslib one and also supports one less configuration syntax (the so-called "shorthand" notation).

This was one piece of code which could not be merged within the time available at the Alpine City Sprint.

So currently we still have two different argument parsers. The Patternslib parser needs to be configured much more expicitly, while the Mockup one is more implicit and makes more assumptions.

Merging these two parsers will probably have to be done at some future sprint.

There are some other more minor differences, such as that every Mockup pattern is automatically registered as a jQuery plugin, but merging these was comparatively easier and I'll won't go into further detail on them.

What I did during the Alpine Sprint

So, in summary:

I refactored the Patternslib registry, scanner and some core utils to let them handle Mockup patterns as well. Luckily the eventual patch for this was quite small and readable.

I changed the Mockup Base pattern so that patterns derived from it now rely on the registry and scanner from Patternslib.

I fixed lots of tests and also wrote some new tests.

This whole task would have been much more difficult and error prone if either Patternslib or Mockup had fewer tests. The Mockup team deserves praise for taking testing very seriously and this allowed me to refactor and merge with much more confidence.

What else is there to still be done?

Being able to use patterns from both projects and merging most of the forked code was a big win, but there are still various things that need to be done to make the merge complete and viable.

I've ranked them from what I think is most important to least important.

1. Update the documentation

Currently documentation is scattered and silo'd in various places (the Patternslib website, the Plone Intranet developer docs and the Mockup developer docs).

The Mockup docs are now out of date ater this merge and need to be brought up to date on these recent changes.

The Patternslib docs are less of a problem because they don't have to deal with Mockup (which can now be seen as an enhancement suite for it), but they can definitely still be improved upon, specifically with an eye on Mockup developers who will start relying on them.

The Plone Intranet consortium also has a useful walkthrough explaining how to create a Patternslib pattern from scratch..

2. Devise a way to also use the Patternslib parser for Mockup patterns

As mentioned, the Mockup patterns still use their own argument parser.

Letting them use the Patternslib's parser will either require extending the Mockup Base pattern to configure the Patternslib parser on a pattern's behalf or instead doing it explicitly in each of the individual patterns.

3. Decide on coding and configuration standards

Unfortunately the coding standards between the two projects differ significantly.

  • The Mockup developers use camelCase as declarative argument names while Patternslib uses dash-separated names.
  • The Mockup source code uses 2 spaces for indentation, Patternslib uses 4 spaces.

4. Remove unnecessary, duplicated patterns

Both projects have patterns for modals, injection, select2 and file upload. These should be merged to get rid of duplication.

5. Move generic patterns (e.g. pat-pickadate, pat-moment etc.) out of Mockup

Mockup has some generic patterns which might also be useful as Patternslib core patterns, in which case they should ideally be moved there.

Conclusion

The sprint was a big success and I'm very happy that all the work has already been merged into the master branches of the matternslib, mockup and mockup-core repositories.

Core code from Mockup is now succesfully merged back into Patternslib and we can use patterns from both projects in the same site.

I had a lot of fun working with talented and motivated software developers and had a lot of opportunities to improve my German.

Thanks again Jens and Christine for organising a great sprint and I look forward to doing such a sprint again!

Innsbruck, the alpine city

The Solution to Jaron Lanier's "Siren Servers"

| categories: economics, foss, open-source, surveillance

In his book Jaron Lanier claims that the information on the web is undervalued and underpriced.

According to him this is hollowing out the middle class because people aren't being sufficiently valued for their information-producing work, while a new rich elite is making vast sums of money by collecting and analysing this cheap information.

Who owns the future? by Jaron Lanier

In many cases the information being collected and analysed is given freely to companies by the users themselves in return for email, calendaring and social networking features.

This results in information assymmetry (e.g. who knows what about whom) and therefore a large disparity in power and wealth. Google knows everything about you and yet you know nothing of value about Google.

Siren Servers

These online services are hosted on servers, which Lanier has termed Siren Servers [1], after the Sirens of Greek Mythology. The Sirens are beautiful but dangerous creatures who lure nearby sailors with their enchanting music and voices, thereby letting them shipwreck on the rocky coast of their island.

These Siren Servers lure us into opening up our lives, so that they might freely gather data from us, on a truly massive scale, which they then analyse and profit from.

The results of this analysis is "kept secret and used to manipulate the rest of the world to advantage."

So much value is extracted from this data, that these Siren Servers are valued in the billions of dollars.

The Siren, Edward Armitage, 1888

The Siren, Edward Armitage, 1888

Lanier claims that the users of these servers are shortchanged, in that they don't get their data's worth in return. Not to mention the broader societal damage wrought by the surveillance-as-business-model engendered by this approach as well as the decimation of the middle class as software takes over many industries.

Siren Servers reduce risk for themselves, but in the process increase the risk for all the other smaller players. Uber's business model for example is all about shifting risks onto its drivers. They would not be profitable if they had to carry the costs of commercial insurance, licensing, vehicle maintenance and oversight of drivers.

"The total amount of risk in the market as a whole stays the same, perhaps, but it's not distributed evenly. Instead the smaller players take on more risk while the player with the biggest computer takes on less."

Lanier's solution to the emergence of Siren Servers is to "properly account" for the wealth created by people online. In other words, people need to be paid for what they do online, no matter how small and seemingly insignificant.

"If information age accounting were complete and honest, as much information as possible would be valued in economic terms. If, however "raw" information, or information that hasn't yet been routed by those who run the most central computers, isn't valued, then a massive disenfranchisement will take place."

The decimation of the middle class

Lanier says this process started in the creative industry, with musicians and photographers impacted quite early on, and that it's now spreading to other occupations, such as travel agents, estate agents and journalists.

Steve Albini playing guitar

Steve Albini playing guitar (2007)

Lanier writes:

"Making information free is survivable so long as only limited numbers of people are disenfranchised. As much as it pains me to say so, we can survive if we only destroy the middle classes of musicians, journalists and photographers. What is not survivable is the additional destruction of the middle classes in transportation, manufacturing, energy, office work, education and health care. And all that destruction will come surely if the dominant idea of an information economy isn't improved."

In other words, until we reconfigure the economic system to value raw data by compensating the people that generate it, we will have massive inequality.

Not everybody is of course as pessimistic or draws these same conclusions. Steve Albini, producer of Nirvana and Pixies albums and himself a touring musician, says in this keynote that the Internet has solved many problems for musicians and audiences alike and that things are actually better now than before.

Finance got networked in the wrong way

As an example of the destruction being wrought by the Siren Servers, Lanier asserts that the Great Recession was in part caused by their existence in the financial sector.

"Consider the expansion of the financial sector prior to the Great Recession. It's not as if that sector was accomplishing any more than it ever had. If it's product is to manage risk, it clearly did a terrible job. It expanded purely because of its top positions on networks. Moral hazard has never met a more efficient amplifier than a digital network. The more influential digital networks become, the more potential moral hazard we'll see, unless we change the architecture."

It's clear that Lanier believes that this problem can be solved by reconfiguring and improving the architecture of our digital networks.

Siren Servers and the rise of surveillance as a business model

As has become increasingly clear in the past few years, the dominant business model in Silicon Valley is one based upon surveillance.

Surveillance is the mechanism by which the Siren Servers gather much of the data they analyse and profit from.

Surveillance cameras

According to Lanier, many Silicon Valley techno-libertarians handwave any concerns about corporate surveillance.

"Surveillance by the technical few on the less technical many can be tolerated for now because of hopes for an endgame on which everything will become transparent to everyone. Network entrepeneurs and cyber-activists alike seem to imagine that today's elite network servers in positions of information supremacy will eventually become eternally benign or just dissolve."

However, what good reason is there for owners of these immensely powerful and valuable Siren Servers to one day willingly become transparent and open up in order for this power imbalance to be resolved?

On the supposed emergence of some kind of sharing/socialist utopia arising out of the aggressive right-wing libertarian practices of Silicon Valley, Lanier mocks:

"Free Google tools and free Twitter are leading to a world where everything is free because people share, but isn't it great that we can corner billions of dollars by gathering data no on else has?" If everything will be free, why are we trying to corner anything? Are our fortunes only temporary? Will they become moot when we're done?"

These "network servers in positions of information supremacy" hide their code and algorithms behind patents, IP-laws and copyright.

A whole legal facade has been erected precisely to prevent the hopeful scenario of "elite network servers" dissolving into an utopian abundance.

Lanier's proposed solutions

Cyber-keynesianism

Lanier describes his proposed solution as a sort of Cyber-Keynesianism. Based upon the idea from the economist J.M. Keynes that "stimulus" might be able to kick an economy out of a rut.

Lanier provides a qualitative explanation of the stimulus theory which I haven't heard before.

On a multi-dimensional mathematical landscape (as could be plotted based on the parameters of an economy), there are peaks and valleys. Let's assume that peaks are "optimal" economical configurations, where the greater good is best served, and valleys are horror scenarios (depressions, recession, collapse etc.).

Not all peaks are of equal hight, and when you are on a peak, you are usually surrounded by valleys. There might be a higher peak (i.e. better configuration) available, but it's not clear how to get there, since moving there might mean moving through valleys.

The idea behind stimulus is to provide the impetus, a kick if you will, to make this transition to a higher peak.

Technical Solution

Laniers solution on a technical level is to use a network with bidirectional links, instead of unidirectional links as we currently have with the web. The architecture of this network is inspired by Project Xanadu already founded by Ted Nelson in 1960.

Whenever someone publishes something on the web they would be informed if someone else links to their work, due to the bidirectionality of the links. This would supposedly put a stop to copyright infringement and also allow content creators to correctly identify and bill the consumers of their media.

Of course, this would only work if there were no anonymity on the web.

Lanier further proposes a single, public, digitally networked marketplace where every participant is tied to his personal real-life identity, and where anything anyone creates online has to be paid for.


My Criticism

I thoroughly enjoyed reading Lanier's analyses, and his descriptions of the mechanisms behind Siren Servers in particular. This was for me the best part of the book.

However, he often hand-waves obvious counter arguments or advances his own arguments in a non-rigorous way, often relying on personal anecdotes.

Digital networks' supposed role in hollowing out of the middle class

Considering his central thesis, that digital networking is hollowing out the middle class in developed societies, I'm not sure how much blame for that can be assigned to digital networking. Other plausible factors include demographic change, globalisation and the money printing policies of Central Banks which are fueling asset price bubbles while real wages remain stagnant, thereby making wage-earners poorer.

My expectation is that no particular factor can be singled out, and that the causes are multiple, varied and intertwined in complex and difficult to understand ways, precisely because the economy is a complex, chaotic system (in a mathematical sense).

Lanier fails to meaningfully address Free and Open Source software

Lanier only very superficially addresses the role of free and open source software (FOSS) and doesn't even mention (let alone extensively cover or deconstruct) the ideology of Free Software at all.

As a quick recap to people who don't know the difference between "Free Software" and "Open Source software":

Both groups concern themselves with creating software where the source code is visible, as opposed to software where you cannot inspect the source code, and are therefore unable to modify or know what it is really doing.

Gnu and Linux

The GNU and Linux mascots. The GNU represents Free Software

"Free Software" (as in freedom, not price) however stresses a moral and ethical dimension of software development and usage. It basically boils down to the fact that non-free software creates a dangerous and immoral power imbalance, slanted in favor of the creator of the non-free software and against the user of that software. [2]

Open Source software on the other hand ignores the moral dimension completely and takes a much more expedient approach. It simply asserts that developing software in the "open", in other words, with the source exposed, is a superior form of software development which will result in better code.

In the index of this book, there is no entry for "Free Software" or "Open Source Software". There is one entry "open source applications".

Lanier doesn't even attempt to address the viewpoint of the Free Software movement, which has been warning about these issues (structural power imbalance, Siren Servers, ubiquitous online surveillance) for decades. This is in my opinion a glaring omission from the book.

Lanier might have coined the catchy phrase Siren Servers, but his analysis of their role and negative effects on users and society might have well come from someone from the Free Software movement. See for example the article Who does that server really serve?.

As the saying goes, "There is no cloud, only other people's computers" [3]

Lanier of course, comes to a completely different conclusion on what needs to be done to resolve the situation, and appears to hold proponents of Free and Open Source Software (dimissively calling them "racals", "openness crusaders" and Pirate Party types), not only in contempt, but partly responsible for the emergence of Siren Servers and rise of surveillance as business model.

It is clear that if Lanier were to meaningfully address all ideological opponents of his vision for the future, then he would have had to address the ideas behind Free Software in a methodical and rigorous manner and not with dismissive hand-waving.

The fact of the matter is that Free Software does provide an alternative solution to the current mess of online surveillance, increasing risk and fragilization, extreme power imbalances and centralization into Siren Servers.

The solution is to ensure that the software you use is free. If software being used by a service is free and issued under a license that ensures its freeness, then that server will not have the ability to eventually become a Siren Server.

The reason for this is that users will know exactly what the software does with their data and will have the ability to switch to a different provider (while taking all their data with them), or to host it themselves, at a moment's notice.

This means that the power balance is no longer slanted in favor of the service provider, and users are much more in control of their data and content.

Lanier's hypocrisy

Throughout the book, Lanier criticises in particular Google and Facebook for their creepy surveillance business models and their usage of Siren Servers.

Lanier himself works for Microsoft, a company that is feverishly competing with Google and Facebook in surveilling users and in establishing their own Siren Servers.

However, Lanier of course does not at all address this conflict of interest or that fact that his employer is just as complicit as the companies he calls out. This apparent hypocrisy severely detracts from the integrity of his book.

Lanier's superficial and misconstrued criticisms of "openness"

Lanier's refusal to meaningfully engage with the different ideas and philosophies behind what he calls "openness" is for me one of the most disappointing aspects of the book.

Wikipedia

He deceptively equates Wikipedia and Facebook, saying that both are a culmination of the "information should be free" meme and that both contribute to the devaluation of user created content. In a previous book, he even calls Wikipedia "Digital Maoism".

Wikipedia is concerned with creating a digital Commons. By voluntarily contributing to the commons (as Wikipedia contributors do), you are not devaluing your work, you are making it priceless, in both the literal and figurative senses of the word.

Conversely, the information on Facebook is not free and is not part of a commons.

I find it good to know that there are some things in this world on which one cannot put a price tag, despite the efforts of people like Lanier.

As the saying goes. "A cynic is someone who knows the price of everything and the value of nothing."

Wikileaks

Lanier incorrectly claims that Wikileaks is concerned with abolishing all privacy. Anyone who has made even a half-arsed effort to understand what Wikileaks is about would know that they stand for transparency of the powerful and their institutions, in order to keep them democratically accountable, and that they don't advocate the elimination of all personal privacy.

Of course, this is a common mischaracterization of this organization, often done to intentionally discredit them. Either Lanier is also trying to discredit them or he couldn't be bothered to properly research their stated goals.

Linux

Lanier claims: "A linux always begets a Google" as if it's is somehow responsible for the emergence of Siren Servers. This doesn't explain Siren Servers built upon proprietary software, such as StackOverflow which uses Microsoft.

Inasmuch as one could argue that Siren Servers are the result of open source software (note the distinction) it is exactly due to open source's gutting of the moral aspect out of free software.

Lanier wants to remove anonymity on the web

Lanier criticises tech companies for their creepy surveillance business models, and yet his proposal for "fixing" the internet would eradicate all privacy online and have all human interactions facilitated via a single, public market place. It would be a surveillance and privacy nightmare.

Concerning anonymity on the web, Lanier says this is the result of the "pot-smoking liberals" and "paranoid conservatives" who originally built the system and who thought anonymity was "cool". Such a perfect example of his dismissive, non-rigorous and arbitrary reductionism which completely fails to meaningfully confront the potential benefits or dangers of anonymity on the web.

Lanier completely fails to address the fact that anonymity is often the only defence people may have against corrupt and malevolent state power. Whether this applies to activists during revolutions or to conscientous whistleblowers, doing away with privacy would be throwing away one of the few ways of ensuring free speech online.

Tahrir Square on November 27, 2012

Tahrir Square protests in 2012.

Monetizing all human interactions debases them

By wanting to turn the web into a giant marketplace where everyone operates with an exposed identity and every digital creation, no matter how trivial, needs to be paid for, Lanier wants to further the already ongoing monetization of the public commons. As Charles Eisenstein writes sacred economics, monetizing all human interactions actually debases them.

Lanier calls his approach "humanitarian"; it is anything but. Instead, by monetizing and thereby commodifying every online human interaction, whether it's a joke, a compliment or a statement of support, these interactions are stripped of their inherent sacred quality of humanity.

If you know that someone who sends you a message of support or love will receive a micropayment for that message, could the original intent of the message not be called into doubt?

My proposed solutions

It's about freedom, not cost

Lanier's proposed solution ultimately has to do with costs and trying to find a way to get people paid for anything they write and then publish online, no matter how trivial.

However, in so doing he comes up with a proposed solution that does away with all privacy and anonymity.

His proposal of using a network with bidirectional links might threaten the Siren Server status of services such as Facebook and Google. However, it's not clear how such a network will solve the problem of Uber or AirBNB becoming Siren Servers, thereby pushing out the middle class jobs of taxi drivers and guest house operators.

It's not only about the network architecture, as Lanier suggests. What's more important is Software and Computing Freedom.

If you don't have the freedom to study, use and modify the software that you use, then you will be systematically exploited and preyed upon, no matter what the network architecture looks like.

Free software specifically, is defined by these four freedoms:

  • The freedom to run the program as you wish, for any purpose
  • The freedom to read the source code and to modify it as you wish.
  • The freedom to redistribute original copies so you can help your neighbor.
  • The freedom to distribute your own modified copies to others.

Being able to read the source code is essential for software to be free. An organisation is however only compelled to share their source code if they intend to distribute compiled copies as well. If the software is used privately, i.e. in-house, then they are under no such obligation.

Open Source on the other hand, also gives the right to read the source code, but many open source licenses do not ensure the last two freedoms, and therefore offer weaker protection.

There is however a large overlap between the two groups. Most open source software is also free software, and vice versa.

Free software coupled with free and open protocols means that users are not beholden to specific service providers. It's exactly this aspect of it which is why the dominant tech companies avoid free software like the plague and instead embrace the watered down notion of "open source". Their goal is to maintain strict vendor lock-in, making it difficult for users to switch to other service providers by holding their data and online personas ransom. Something which they could not do if their software respected the above four freedoms.

Who pays for fee software?

When discussing free software, the question often arises how one should make money with software if you cannot enforce artificial scarcity, vendor lock-in, holding user's data ransom or spying on them.

This is perhaps a topic for another blog post. In short, it definitely is possible to make money writing freedom respecting software and the more people insist on using only free software, the more economic opportunities will arise.

Software is better seen as a living, changing and evolving ecosystem, than as a static object. It needs constant care and attention, to keep it relevant, accurate, applicable and functioning. This care and attention is a service which can and should be paid for.

Who pays for this service? The users do. Whether they are governments, universities, corporations or private individuals.

How do they pay for it? Organisations sign service and support contracts. They also commission the writing of new features or entirely new applications.

Private individuals can pay for free software through crowdfunding, bounties, and donations. [4]

Public institutions and foundations can issue grants or fund the development of free software for usage in government and publically owned enterprises.

If all governments decided to only fund and use free software, it would create a massive amount of investment in the sector. Additionally, money spent on developing free software can be invested in a country's own programmer workforce, instead of it being sent overseas to oversized foreign corporations.

However, those companies that currently spy on us, hold our data ransom and restrict our digital freedoms, won't change by themselves. It's up to the users to take matters into their own hands, to demand freedom and to support those companies, organizations and people who develop free software.

Decentralization

The problems Lanier identify are largely caused by data-assymetry (who has data on whom) and the resulting siren servers.

An architectural approach to solving this problem, which complements free software, is decentralization.

Decentralization refers to a network structure (called topology) where each node in the network can connect directly with any other node, without the intervention of a middle man (such as Facebook or Google).

In the network topologies pictured here, the fully connected topology is the most decentralized, while the star network is the most centralized.

Network Topologies

Network Topologies

Creating large decentralized networks is more difficult than creating centralized ones, which is why the web has come to rely on centralized Siren Servers to such a large degree.

However, much exciting work is being done on decentralizing the web and putting people back in control.

Consider Finance, and Lanier's assertion that Siren Servers contributed to the financial crisis preceding the Great Recession.

It's more difficult to have a financial Siren Server when financial data is decentralized and therefore effectively available to everyone.

This is already the case with decentralized cryptocurrencies such as Bitcoin. The bitcoin protocol is an open and free protocol, the original bitcoin wallet is free software and the transaction information contained in the blockchain is available to all.

An improvement upon Bitcoin would be to allow truly anonymous transactions, and this appears to be on the horizon.

Bitcoin and cryptocurrencies are however only one application which can be built on top of the revolutionary distributed consensus blockchain technology upon which it rests.

There's already work on distributed cloud hosting, distributed microblogging and a myriad of other services running in a decentralized (non-Siren Server) manner.

Federation

An older idea than blockchain-based decentralization is federation. Think of the way in which you can send an email from a Gmail account to someone with a Yahoo account, or to someone who hosts their own mail server.

Why can we send email to people with different service providers, but we cannot send chat messages from a Google account to an iMessage account, or between Whatsapp and Yahoo messenger?

This ability of email is the result of something called federation. The ability of servers from different providers and with different implementations, to communicate and relay messages between one another.

Email is federated but facebook messages are not, because you cannot send a facebook message to someone outside of facebook.

There is no technical reason why instant messaging cannot be federated. In fact, XMPP, the protocol on which many of the messaging services are originally based, allows federation.

The reason we don't have federation in messaging is because it's not in the interest of Siren Servers. It would actively undermine their dominance.

If any of the big email providers, Google, Yahoo, Microsoft or Apple, could get away with only allowing emails within their own system, they would have done it by now. The proof is in the fact that they actively prevent federation in their messaging apps. The reason they cannot do it with email is because federated email achieved critical mass before any single company could kill it off.

Unfortunately, this was not the case with instant messaging. Therefore, if you actively want to work against Siren Servers, you should only use instant messaging servers which support federation, such as XMPP.

You can sign up for a free XMPP chat account here. And if you'd like to chat with me, add me as a contact: jc@opkode.com

Conclusion

As you can see, I believe in a completely different remedy than the one which Lanier suggests. One that's based on a methodology, legal framework and software which is already being used instead of something which Wired magazine in 1995 called the longest-running vaporware project in the history of computing.

I thoroughly enjoyed Lanier's analyses and his unique perspective on things which I think made the book worth reading.

However, his non-rigorous approach, his dismissiveness or ignorance of alternative narratives and approaches to resolving the issues he mentions, as well as his hypocritical omission of his own employer's complicity, left me disappointed and uninspired.

Footnotes:

[1]Bruce Sterling calls them "The Stacks", vertically integrated social media. See Bruce Sterling At SXSW 2013: The Best Quotes.
[2]If you're interested in more detail about Free Software, watch this presentation by Richard Stallman:

Introduction to Free Software and the Liberation of Cyberspace

[3]The earliest reference to this that I could find, is this article
[4]See Kickstarter, IndieGogo, Startjoin, Patreon, Bountysource, Gratipay and Snowdrift.

Antifragile Software Ecosystems

| categories: open-source, bitcoin, cryptocurrencies, antifragile, foss, plone

Antifragile, by  Nassim Nicholas Taleb

I recently finished reading the book Antifragile, by Nassim N. Taleb, in which he discusses the idea of "antifragility", the property that allows something to ultimately improve and grow stronger when subjected to volatility and stressors.

I highly recommend Taleb's books, they have a tendency to throw conventional wisdom on its head and make you think differently of our world, in which randomness, volatility, opacity and non-linearity takes precedence.

As a software developer, I kept wondering how the insights gained from this book apply to the world of software. Especially given the fact that software is very fragile. The answer lies in the communities or ecosystems that form around software. Many of them are fragile, but some are antifragile.

If you're also a software developer, you'll want to be in the second category.

What is antifragile?

Taleb splits everything into three categories, fragile, robust and antifragile.

The initial chapters explain in detail why robust is not the opposite of fragile, and why the neologism "antifragile" is needed to describe the opposite of fragility.

Medusa is antifragile

Medusa, courtesy of Wikimedia.org

If fragile systems are sensitive to stress and volatility, and need to be kept "safe" from harm, lest they suffer or break down, then antifragile systems are ones which ultimately benefit from randomness and stress.

In Greek mythological terms, one can think of Medusa, which grows another two snakes on her head for each one that's cut off. That's antifragile.

So fragile and antifragile can be considered opposites, e.g. positive and negative, while robust is simply neutral. It suffers little harm from volatility but also gets little to no benefit.

Evolution is antifragile

Ostriches have evolved from dinosaurs

Ostriches have evolved from dinosaurs

Some good examples of antifragile systems are found in nature and ecology. For evolution to work (i.e. for a species to adapt to its randomly changing environments), random mutations are required between individuals in that species.

A random mutation in a member of a species holds the possibility of making that member better adapted to its environment. By virtue of it being better adapted, that member will on average live longer and produce more offspring, which will be likely to have inherited the same advantageous mutation. The process continues, propagating the beneficial mutation throughout the species, in so doing, the species evolves and becomes more fit and adapted to its environment.

Nature is able to discard less well-adapted specimens and keep the better-adapted ones. An ability which Taleb refers to as optionality. It has the option to keep the "good" specimens and discard the "bad" ones.

The opposite of an option is an obligation. Nature is under no obligation to keep the poorly adapted members of a species, and if it were, it wouldn't be antifragile anymore.

So, although individual members within a collection (like a species) are fragile to certain stressors, the overall effect on the collective level is antifragility.

Software is fragile

/img/fragile.jpg

Image courtesy of Wikimedia.org

Unmaintained and unsupported software degrades when exposed to the stressors of time and use.

The more software you install on your computer, the higher the chance that your machine will eventually become slow and prone to crashes and bugs.

The word "bitrot" is sometimes used to refer to the gradual degradation of a software program over time and is related to the idea of entropy.

Large software systems that are composed of multiple interacting subsystems, are especially vulnerable to bitrot, due to the fact that a given subsystem cannot anticipate all the bugs or changes that might be introduced to other subsystems over time.

The more a piece of software is used, the more variance there is in how it's being used. For example in the order, type and amount of inputs it receives as well as in the hardware its being installed on.

Lets assume that a very specific interaction with the software, which the programmer never foresaw, will cause the software program to crash and become permanently degraded.

Given enough time, someone will eventually interact with the software in this specific way, thereby permanently degrading the functionality of the software program (for example due to data corruption).

Are software businesses also fragile?

Businesses are generally fragile. They require a certain level of stability to operate and are sensitive to volatility (like regulatory changes, lawsuites or changing consumer tastes).

An antifragile software business however would be one that does well whenever some metric related to the software experiences increased volatility.

Remember:

  • something that's antifragile wants more volatility, because the benefit from increased volatility outweighs the harm.
  • Similarly, you are fragile to a specific source of volatility when the potential downsides outweigh any upsides.

So, let's take "number of security flaws" as the metric.

The fortunes of a company that does security audits would improve every time there is a publicised security flaw in the software itself, or the software is compromized by an attacker.

They would therefore be antifragile to the "bad news" of exposed security flaws.

Perhaps with an upper bound. If there are too many security flaws, the reputation of the software itself might suffer irrevocably, and with it that of the company.

This illustrates the point that antifragile systems can handle volatility and stressors up to a point. Given enough volatility and stress, the entire planet can die out.

Conversely, a company whose business model depends on the software being stable, and on attacks not happening, would be fragile to the software having security flaws.

A community of fragile members can itself be antifragile

So, as mentioned earlier, an ecosystem can be antifragile, even if the individual units that make up that system are fragile.

It is however important that the units themselves are relatively redundant and that the fragility is not systemic inside the entire ecosystem.

To put it in more technical terms, units inside the system need to be "hot-swappable" and their interactions with one another should be "loosely coupled". This is necessary to prevent complex and hard dependencies within the system, which in the case of breakdown of one unit would cause cascading failures and damage rippling through the entire system.

Antifragile software ecosystems

So, we now get to the crux of this article.

Software itself might be fragile, as well as businesses built around that software, but the larger ecosystem that encompasses the software and businesses, can be antifragile.

A software ecosystem that is distributed over many small and relatively self-contained businesses, each providing services around a common shared resource of Free and Open Source software (FOSS), coupled with a low barrier to entry for new businesses, is antifragile.

Conversely, a software ecosystem that is centralized around one large provider (often playing the role of gatekeeper or rent-seeker), artificially restricting access to the software, attempting to stifle any competition with its own services, is fragile and sensitive to large negative Black Swans (difficult to predict but extreme events).

I'll elaborate on why this is by identifying the requirements for an antifragile software ecosystem.

Requirements for an antifragile software ecosystem

  • Distributization

    Centralized systems provide a large single point of failure that can cause the collapse of the entire ecosystem. Distributed systems can take shocks much better and are able to route around failures and breakdowns.

  • Redundancy

    When one unit breaks down, others need to be able to pick up the slack. For that to happen, they need to collectively be able to perform the functions of the defunct one. For example, if a software community relied solely on one unit in that community for supplying hosting services, then the entire community would break down if that one unit collapsed.

  • Optionality

    The individual members in an antifragile ecosystem need to be able to modify and adapt.

    For a software ecosystem, this means the ability to modify and adapt the software itself. Only through being able to modify the software in a distributed, bottom-up way, can the ecosystem withstand stressors in an antifragile way, by making use of optionality.

    Optionality refers to having the option to take (or do) something or not. As mentioned earlier, it's how evolution is able to "design" extremely sophisticated organisms, by keeping the "good" ones and discarding the bad ones, and it applies to software ecosystems as well.

    The Plone logo

    For example, in the Plone CMS (content management sytem) community, add-on modules are developed in a redundant and distributed way. There are multiple blog add-ons, commenting add-ons, newsletter add-ons and so on. Through a process of optionality and selection, the good ones rise to the top while the weaker ones are eventually discarded.

Why the requirement for Free and Open Source software?

FOSS facilitates all three of the above requirements. Communities and ecosystems that develop around FOSS are by nature distributed, redundant and provide optionality.

Distributization and redundancy come from the lack of gatekeepers, "rights-holders" and rent-seekers. Anyone can start using FOSS and selling services around it from anywhere in the world. No license, special qualification or permission required.

Optionality comes from the fact that anyone can do trial-and-error tinkering with the software and keep their improvements. They can also choose the best or most appropriate changes and modifications made by other people.

Without optionality a software ecosystem will not be antifragile, and without FOSS, a software ecosystem will not have optionality.

Additionally, software that is Free (as in Libre) displays qualities much more akin to information than non-Free software, which due to it's source opacity functions more as a black-box application. As Taleb explains in his book, information is antifragile. You can't easily kill it, and by trying to suppress it you have to acknowledge its existence and refer to it.

Lastly, the lower barrier to entry fosters competition.

In order to differentiate themselves, individual developers or businesses need to compete on quality and/or price. It's survival of the fittest, as opposed to the rent-seeking, vendor lock-in and monopoly so often seen in proprietary software ecosystems.

Note

In the book, Taleb alludes to Silicon Valley's "fail fast, be foolish" model as being antifragile as opposed to a more centralized environment with inherent systemic risks, such as the New York banking system (his example).

Silicon Valley might provide an antifragile method for producing businesses, but the software produced by these businesses don't necessarily result in antifragile ecosystems.

Look at the software businesses that have arisen out of Sillicon Valley. Facebook and Google are both large and centralized. Increasing centralization and size results in increasing fragility of the web.

If Google's servers were to go down (or be compromized or blocked by a government), a very large part of the web would cease to function. The chance of Google's servers going down are of course very small, it's a negative Black Swan event. Something which society nevertheless needs to shield itself against.

Antifragile against what?

Having mapped out the requirements for an antifragile software ecosystem, it makes sense to consider the kinds of stressors such an ecosystem will be antifragile against.

This section is a bit more speculative, mostly because we cannot know the exact nature and extend of the stressors a system will be exposed to. Luckily we don't need to know the stressors to be able to make use of antifragility.

It's also important to know that stressors are information. In fact, stressors often appear through the disclosure of "negative" information. When negative information is revealed, it can cause shocks to software ecosystems. The fragile ones will suffer and might even break down, while the antifragile ones will ultimately benefit.

Here are some examples of stressors a software ecosystem can be antifragile against:

  • Malicious Hacking (Cracking)

    FOSS is more resistant to cracking than closed source software. Additionally, distributed systems are much harder to target and infect.

    Increased awareness and fear of cracking, makes FOSS, its ecosystems and communities more attractive to software users (individuals, companies, governments etc.) and therefore antifragile to the "bad" news.

Surveillance cameras

Image courtesy of Wikimedia.org

  • Surveillance

    A big part of surveillance is actually also cracking, so the same rules apply here.

    It's much more difficult for eavesdroppers to embed backdoors in software with freely available and visible source code.

    Encryption software is especially vulnerable and can't be trusted if its closed source.

    News of security flaws spread quickly throughout the community and fixes are generally pushed out almost immediately. You won't easily have the problem of a FOSS community member privately disclosing security flaws to an intelligence agency (so they might use if for surveillance) while hiding it from the users.

    Again as surveillance and privacy fears increase, FOSS becomes more attractive, making the communities and ecosystems around it more antifragile.

  • Negative economic circumstances

    Many governments are experiencing budgetary constraints and employing austerity. Governments are able to make significant savings when they switch to FOSS ecosytems. Not mentioning the increased autonomy and control of their software and resulting digital formats.

There are of course many more stressors that might ultimately benefit distributed FOSS ecosystems, for example censorship and economic boicot, but for the sake of brevity I won't go into detail for each one.

Conclusion

I've mapped out the requirements for a software ecosystem (and community) to be antifragile, and have alluded to Plone as an example of such an ecosystem.

The Bitcoin logo

The Plone software, like other Free software (Drupal, Wordpress, Apache, Firefox, Libre Office, GNU/Linux etc.), is a technological commons. It belongs to humanity as a whole.

Everything I wrote also applies to the ecosystems around cryptocurrencies, like Bitcoin, Litecoin and Anoncoin.

There are usually vibrant communities of developers, packagers, integrators, evangelists and users around these pieces of Free software, and they usually conform to the requirements mentioned above. Members are distributed all over the world, there is redundancy in development skills and support offerings and the source code is Free (as in beer and in speech).

Thanks for reading and please leave a comment if you have anything to add, I'd be happy to discuss the finer details perhaps missed in this article.


Converse.js 0.7 released

| categories: javascript, cryptography, xmpp, converse, chat, otr, foss

Now supports Off-the-record encryption

I'm happy to announce that the 0.7 release of Converse.js now supports Off-the-record (OTR) encrypted messaging.

The OTR protocol not only encrypts your messages, it provides ways to verify the identity of the person you are talking to, plausible deniability and perfect forward secrecy by generating new encryption keys for each conversation.

How this works is actually fascinating. The encryption keys are generated via a Diffie-Helman exchange. On youtube there is a video with a very simple and intuitive explanation on how the Diffie-Helman key exchange works. For more details, you can also read the OTR version 3 protocol spec.

I was able to add OTR support for converse.js by using Arlo Breault's excellent otr.js library, which is also used in Cryptocat and Diaspora.

Many thanks go out to him for this.

Note

Note:

Arlo cautions that the otr.js library has not yet been properly vetted by security researchers and shouldn't be used in life or death situations.

This word of caution applies doubly so to its integration into converse.js!

I'm not a professional cryptographer and this should be seen as an experimental feature.

Caveat emptor!

Screencast

Note: For detailed fullscreen, make sure that video quality is set to HD, by clicking the gear icon on the video player.

On Javascript Cryptography

Implementing Cryptography in Javascript is relatively controversial and in 2011 Matasano security wrote a scathing criticism called Javascript Cryptography Considered Harmful.

I read that article before I started working on OTR support for converse.js and I'd like to go through some of its criticisms step by step to explain how they might apply to converse.js and what I've done to mitigate them.

If you're going to read on, it would make sense to go read that article first.

Secure delivery of Javascript to browsers being a chicken-egg problem

Their criticism appears to apply to the case where developers use Javascript cryptography to implement secure communications with a server instead of relying on a standard protocol such as SSL/TLS.

The OTR crypto is not trying to compete with or replace SSL/TLS.

Sending data in the clear with normal HTTP is not secure and should be avoided. Therefore, if you are going to use OTR with converse.js, always make sure that the site users HTTPS (i.e. SSL/TLS).

Ideally you would also want a mechanism to sign the Javascript and then verify it's authenticity once it's been downloaded.

This can be done with code that runs in browser extensions (like Cryptocat does), but (AFAIK) unfortunately not with browser-agnostic javascript (like converse.js).

This looks to me like the achilles heel for browser javascript crypto.

Besides the webserver serving the Javascript, it's also important that the BOSH service you use employs SSL/TLS, as well as the XMPP server the BOSH service connects to.

The XMPP community is on a drive to secure all the major public XMPP servers. See the Manifesto created by Peter Saint-Andre and signed by many in the XMPP community.

On http://xmpp.net they let you test the security of servers on the Jabber/XMPP network.

Besides using javascript served via TLS, you can also download the converse.js tarball to your computer, unpack the files and access them directly. The index.html page will open up a page identical to the one on http://conversejs.org.

Browser Javascript being hostile to cryptography

The prevalence of content-controlled code.

If the website fetches javascript from thirdparty websites (for example to serve advertisements), then you not only have to trust the server to serve non-malicious code, but also all those 3rd party servers.

The solution is to not use such a site for secure communications. Clear your cache regularly, Use hosts you trust, or as I've mentioned above, bypass the webserver entirely by using converse.js from your own filesystem.

The malleability of the Javascript runtime.

Javascript is indeed very malleable. Methods and attributes on objects can be replaced during run-time (see Duck Typing and Monkey Patching),

This makes it easy to inject malicious code into Javascript.

However, in order to do this, the attacker's code needs access to those Javascript objects, and fortunately Javascript provides a way of hiding sensitive objects inside closures, thereby preventing an attacker from accessing and manipulating them.

Simply put, a closure defines a limited scope for objects, variables and functions which when used correctly, cannot be accessed from any other Javascript code.

Any variables, objects or functions defined locally inside another (parent) function (with the var) keyword, are not accessible outside of that parent function.

If the parent functions is anonoymous (is not referenced anywhere) or the child function's runtime outlives the parent's, you have a private, enclosed scope with data that cannot be accessed from outside that scope.

All the data in converse.js is encapsulated inside such closure.

The lack of systems programming primitives needed to implement crypto.

Matasano claims that Javascript lacks a secure random number generator RNG.

Without having authoritative knowledge on whether their claim is valid, I'll refer you to this blog post by Nadim Kobeissi (author of Cryptocat), where he claims:

This is in fact not the case. window.crypto.getRandomValues() is a secure random number generator available in JavaScript, and supported by major web browsers.

His blog post is well worth reading, and he also addresses some other criticisms for which I didn't have anything to add.

Conclusion

OTR encryption in converse.js provides increased security and privacy for your chats. Messages are encrypted and not logged or cached

Any Jabber/XMPP services that snoop on your communications by logging your messages (like for example Facebook does) will only have access to the encrypted ciphertext, making it useless for surveillance or exploitation.

Features coming future releases

The current implementation of OTR doesn't have any encryption policy support. With that, I mean the ability to set a policy such as Always encrypt messages.

Currently you need to always manually enable encryption.

Distributed services and self-hosting

There are more and more Free and Open Source (FOSS) self-hosted web-applications becoming available as alternatives to so-called centralized "cloud" solutions.

See for example roundcube, kolab, ownCloud, MailPile, friendica and buddycloud.

Converse.js can be integrated into any of these applications, and since you should ideally host them yourself, you'll have more control on what Javascript is actually being served.

Feedback, patches, donations

All of the work I did on OTR integration was in my free time and free of charge. A gift and my personal contribution towards the movement for providing free communication solutions that are not founded upon surveillance and commercial exploitation of personal data.

All the code is open source and contributions are always welcome, be it in audits, features, bugfixes, documentation, words of encouragement or tips.

If you're so inclined, I'd appreciate any tips at the following bitcoin address: 16FsPqE9DhFTryxrUenpsGX4LJ1TPu8GqS


« Previous Page