Context. Although research in software engineering largely seeks to improve the practices and products of software development, many practices are based upon codification of expert knowledge, often with little or no underpinning from objective empirical evidence. Software design patterns seek to codify expert knowledge to share experience about successful design structures. Objectives. To investigate how extensively the use of software design patterns has been subjected to empirical study and what evidence is available about how and when their use can provide an effective mechanism for knowledge transfer about design. Method. We conducted a systematic literature review in the form of a mapping study, searching the literature up to the end of 2009 to identify relevant primary studies about the use of the 23 patterns catalogued in the widely referenced book by the “Gang of Four.” These studies were then categorized according to the forms of study employed, the patterns that were studied, as well as the context within which the study took place. Results. Our searches identified 611 candidate papers. Applying our inclusion/exclusion criteria resulted in a final set of 10 papers that described 11 instances of “formal” experimental studies of object-oriented design patterns. We augmented our analysis by including seven “experience” reports that described application of patterns using less rigorous observational forms. We report and review the profiles of the empirical evidence for those patterns for which multiple studies exist. Conclusions. We could not identify firm support for any of the claims made for patterns in general, although there was some support for the usefulness of patterns in providing a framework for maintenance, and some qualitative indication that they do not help novices learn about design. For future studies we recommend that researchers use case studies that focus upon some key patterns, and seek to identify the impact that their use can have upon maintenance.

Yann-Gaël Guéhéneuc, 2013/09/13

The paper starts by justifying the need of a systematic literature survey on design patterns and on design methods in general. It rightly states that lots of results are “strongly advocated in the absence of evidence”. It also interestingly highlights the difference between studies in clinical medicine and in empirical software engineering: “the interventions used in primary studies in clinical medicine are performed upon the participants, whereas for software engineering, they are performed by the participants” (emphases mine) and “that the skill levels of the participants [is] a significant confounding factors”. These statements are interesting and highlight the importance of understanding and categorising developers. The paper also states that teaching design patterns, i.e., teaching design practices, is difficult and that multiple examples of patterns are needed to assist with the learning process.

The results are based on papers published between 1994 and 2009. It would be interesting to reproduce this study with more recent papers and with more authors. They suffer from the limits of the raters (the two authors). A third author would have been needed to counterbalance votes and avoid discussions, which may have been biased through raters learning effect.

The results report that most studies involve modifications, i.e., maintenance tasks, and elements of coding, which is not surprising as the phases of design and implementation are often mixed even though they should not. It would be interesting to understand how much developers do actually distinguish architecture from design from implementation and whether (if ever) distinct developers perform design and implementation.

The results include interesting discussions and three recommendations. In the discussions, the paper states that several studies involve more than one design pattern. This observation is not surprising because (1) design patterns are supposed to work together, they form a “language”, and (2) some studies use automatically occurrences of design patterns that are not instances, i.e., they are false positive occurrences. As recommendations, the paper states that:

• “The use of experiments, particularly using student participants and short-term tasks, should be used with care for studies of design. Case studies may be more appropriate for exploring the complex cognitive issues involved” — That is all the difficulties in teaching and studying design patterns!
• “Studies of design patterns should use research questions that are related to specific patterns and their roles” — That is what we have been doing although it is often difficult to distinguish roles in implementation, which calls for design studies.
• “Observational studies need to be reported rigorously and ensure that the links between any conclusions and the reported experiences are explicit” — That is something sometimes limiting because reviewers always want more!