Cost, Benefits and Quality of Software Development Documentation: a Systematic Mapping

Abstract

Context: Software documentation is an integral part of any software development process. Researchers and practitioners have expressed concerns about costs, benefits and quality of software documentation in practice. On the one hand, there is a lack of a comprehensive model to evaluate the quality of documentation. On the other hand, researchers and practitioners need to assess whether documentation cost outweighs its benefit. Objectives: In this study, we aim to summarize the existing literature and provide an overview of the field of software documentation cost, benefit and quality. Method: We use the systematic-mapping methodology to map the existing body of knowledge related to software documentation cost, benefit and quality. To achieve our objectives, 11 Research Questions (RQ) are raised. The primary papers are carefully selected. After applying the inclusion and exclusion criteria, our study pool included a set of 69 papers from 1971 to 2011. A systematic map is developed and refined iteratively. Results: We present the results of a systematic mapping covering different research aspects related to software documentation cost, benefit and quality (RQ1-11). Key findings include: (1) validation research papers are dominating (27 papers), followed by solution proposals (21 papers). (2) Most papers (61 out of 69) do not mention the development life-cycle model explicitly. Agile development is only mentioned in 6 papers. (3) Most papers include only one "System under Study" (SUS) which is mostly academic prototype. The average number of participants in survey-based papers is 106, the highest one having approximately 1000 participants. (4) In terms of focus of papers, 50 papers focused on documentation quality, followed by 37 papers on benefit, and 12 papers on documentation cost. (5) The quality attributes of documentation that appear in most papers are, in order: completeness, consistency and accessibility. Additionally, improved meta-models for documentation cost, benefit and quality are also presented. Furthermore, we have created an online paper repository of the primary papers analyzed and mapped during this study. Conclusion: Our study results show that this research area is emerging but far from mature. Firstly, documentation cost aspect seems to have been neglected in the existing literature and there are no systematic methods or models to measure cost. Also, despite a substantial number of solutions proposed during the last 40 years, more and stronger empirical evidences are still needed to enhance our understanding of this area. In particular, what we expect includes (1) more validation or evaluation studies; (2) studies involving large-scale development projects, or from large number of study participants of various organizations; (3) more industry-academia collaborations; (4) more estimation models or methods to assess documentation quality, benefit and, especially, cost. (C) 2014 Elsevier Inc. All rights reserved.