Open Source Maturity Model

1. Open Source Maturity Model(s)

Navica's maturity model is not the only one that an open source software evaluator might turn to. Rather confusingly, the commercial consultancy firm CapGemini developed a different assessment method, also called the Open Source Maturity Model. Navica's Open Source Maturity Model grew from a book written by its CEO, Bernard Golden, entitled Succeeding with Open Source, which was published in 2004. Bernard Golden is also credited as one of the inspirations behind the rival Business Readiness Rating (BRR) evaluation method. Another similar model is the Method for Qualification and Selection of Open Source software (QSOS), developed by a French team working for the commercial outsourcing company Atos Origin.

Each of these assessment models shares certain common features:

The assessment models differ in regard to the complexity of the process and the balance of the tests to be conducted.

2. Navica's OSMM

The first phase of the Navica OSMM is to select the software that you are going to evaluate. As with equivalent systems, the OSMM does not provide much guidance on compiling the short list.

The second phase involves assigning weighting factors. The OSMM breaks down the key aspects of the software to evaluate into six categories. The weightings assigned to each of these categories must be decided according to what the software is to be used for, and who will be using it. The default weighting in each category is as follows:

Each of these categories comes with its own ‘Template’, which suggests what aspects within the category should be evaluated, and the maximum score that should be awarded to each aspect (again totalling 10). Assigning this score constitutes the third phase of the OSMM.

Taking the Documentation template as an example, up to two points should be awarded based on the documentation provided by the software creators themselves, up to three points based on the availability and extensiveness of Web postings, and up to five points based on the amount of commercial published documentation. So a piece of software with excellent built-in help files, a reasonably extensive online forum, but which has never had a book written about it, might score 4 out of 10: 2 points for creator documentation; 2 points for community documentation; but no points for commercial support.

If the quality of documentation is likely to be a very important factor in getting the most out of the software, one might want to change the weighting to reflect this greater significance. An organisation that does not have the resources to implement the software itself might, on the other hand, look to a professional services firm to help out, in which case they would give greater weighting to the Professional Services category.

The fourth phase consists of multiplying the score for each category by its weighting, to produce a final score between zero and one hundred. This is then measured upon a ‘maturity’ table.

The maturity table takes into account whether the users are ‘Early adopters’ or ‘Pragmatists’, and whether the application is intended for experimental or pilot study purposes or whether it is to be used in a production environment. The recommended minimum score for early adopters using software in a production environment is 60; 70 for more risk-averse pragmatists.

3. How the OSMM compares against the BRR

The OSMM and BRR are in many respects very similar but there are two key differences:

Both models will take time if they are to be fully applied across a range of open source products, although the OSMM's relative lack of detail leaves more of the assessment and scoring up to the user. The BRR tests, on the other hand, specify exactly how to score each item under consideration.

Whichever method is preferred, both make it clear that they are there to assist judgement rather than replace it with a simple score. The act of reviewing software using the frameworks is in itself likely to greatly improve one's understanding of it, even leaving aside the scoring element.

4. The Open Source Maturity Model today

The Open Source Maturity Model has not seen the level of adoption that was hoped. It could be argued that the lack of interest in this, and related, approaches to open source project evaluation indicates a flaw in the approach. Certainly, the fact that these evaluation methods cannot be applied equally to closed source products makes the process less attractive to those wishing to evaluate open source alongside closed source products. Despite this, OSS Watch considers the Open Source Maturity Model and similar tools such as the Business Readiness Rating to be useful for those new to open source project evaluation.

As the Open Source Maturity Model wanes, the Qualification and Selection of Open Source Software (QSOS) project slowly grows in popularity, although even QSOS cannot yet be considered a mature project. Another alternative model is the Reuse Readiness Levels developed by the NASA Earth Science Data Systems Software Reuse Working Group. Whilst the OSMM considers non-technical criteria, QSoS considers both technical and non-technical criteria and the Reuse Readiness Rating considers technical criteria.

5. Further reading

Links: