Evaluation of the ACAP analysis method for process-based validation of textual and graphical design methods

The targeted improvement of design methods requires validationstudies to record and evaluate difficulties in the application of theinvestigated methods. Current analysis methods for the validationof design methods are limited to the collection and evaluationof the design outcome and do not consider the applicability ofthe design method. In this paper, the authors evaluate an analysismethod which, in addition to the benefits of a design method, alsodetects difficulties in its applicability. The Attention-Comprehension-Application-Performance analysis (ACAP analysis) method has beennewly developed for this purpose. It investigates the applicabilityand captures metrics attention, comprehension, and correctness ofapplication as well as the benefit of the design method. To eval-uate the ACAP analysis method, a method for sheet metal designwas examined in a laboratory study with 25 university students withmechanical engineering as their major subject. The results of theevaluation showed that the ACAP analysis method (1) identifies dif-ficulties in the applicability of the design method; (2) determines thebenefitofthedesignmethod;and(3)capturestheimpactoftheiden-tified difficulties on the benefit. ... mehrThus, the ACAP analysis method canevaluate causes of existing difficulties using objective metrics.