Evaluation of the Applicability and Effectiveness of a Model-Based Activity for Supporting the Synthesis of Validation Environments

The synthesis of validation environments for cyber-physical systems is challenging and often implicit, experience-driven, and difficult to reuse across product generations. This limits transparency, comparability, and reuse of validation environments. Model-based systems engineering (MBSE) offers the potential to address these challenges by structuring system knowledge in machine-readable models and enabling systematic and traceable evaluation of alternative solution concepts for validation environments. Building on this idea, this paper specially evaluates one derived model-based activity that supports criteria-based comparison and selection of alternative design solutions during the synthesis of validation environments. The activity was applied and evaluated in a controlled empirical study within a realistic development environment using a part of an automotive brake system validation environment as an example case. Participants solved engineering tasks both without and with the provided support in form of a model-based activity, and the results were evaluated using quantitative Likert-scale assessments and qualitative feedback. The results indicate a positive evaluation of the investigated model-based activity regarding applicability and effectiveness. ... mehrIn addition, the support provided led to more consistent and traceable solution selections across different groups. The findings therefore provide first empirical indications that this specific model-based activity can improve transparency and consistency for decisions regarding the synthesis of validation environments.