Thoughts on Beautiful Architecture ch. 12 - When the Bazaar Sets Out to Build Cathedrals

In this chpater Till Adam and Mirko Boehm set out to describe the KDE project, focusing on Akonadi and ThreadWeaver. Along the way, they gave out insights on large-scale open source projects as a whole.

The authors made some keen statements on the strengths of open source projects. The mantra of "those who do the work decide" not only empowers the foot soldiers also install a sense of ownership and responsibilities in them. The ideas that people are the most scarce and valuable resource and the only criteria to include code in the software is its quality and if it is being maintained install honor in the authors. These are the same ideas being kicked around in for-profit software companies but are never fully practiced due to various non-technical reasons such as time to market, inter-team politics, leaders's desire to strongarm an decision, etc. Sometimes it is rather surprising what the scope and the quality an open source project can achieve when motivation by money and fame has failed.

Back to KDE and Akonadi in specific, it is driven by needs of the masses. It is the more Bazaar-style project of the two the authors talk about. Its background provides a good example about how an architecture based on some underlying assumptions deteriorated over time due to requirements changes. Since changes do not come instantly but rather gradually, there is always a constant struggle to either retrofit or rebuild. In the space of Akondi framework which deals with personal information management, there has been huge changes in its scope and scale as more applications converge to share and access personal data (a recurring theme in this class - data is king), the community finally come to a consensus to rebuild. To tackle this complex problem, the Layer architecture pattern is used to separate the concerns. Adding a platform API layer between applications and the storage access infrastructure so the applications using the Genome API and those using the KDE API can both share the infrastructure makes it development-platform-independent which greatly increases its usability.

In contrast, the ThreadWeaver project follows a more Cathedral-style path: it stemmed from an individual's vision. The individual has a vision to provide a conveniently available job scheduler for concurrency to make concurrent programming easy (another recurring theme - concurrent framework to shield the burden of concurrency from applications). The ThreadWeaver scheduler takes care of job scheduling and dependency enforcement at the same time allows its user enough control by specifying job dependency, priority, and queue policy. The specific example on the use of inverse ordering and priority to gives more weights on data that is already partially processed over the new data which prevents all data being processed at locked steps is a nice pattern that can be used by most of the Pipeline architecture that needs the similar behavior.

The authors made another key statement at the end: to take advantage of concurrency, it is not enough to provide a tool to move stuff to threads, the difference is scheduling. This is especially true if the move to concurrency is to achieve better performance.