Compact Functional Test Pattern Generation for DNNs Using Evolution Strategies

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical systems, making it essential to ensure their reliability in the presence of hardware faults. Such faults can degrade model accuracy or lead to silent and undetected errors, highlighting the need for effective testing strategies, particularly model-specific functional testing. We propose a functional black-box testing framework that detects hardware faults exclusively through their observable effects on network outputs. Functional test patterns are generated offline through fault modeling, fault injection, and optimization. Since maximizing fault coverage while minimizing test-set size yields a non-differentiable optimization problem, we employ an evolutionary optimization approach. Specifically, we use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to generate compact test sets that achieve high fault coverage, where compaction refers to selecting a minimal subset of test patterns without compressing or altering them. We evaluate our approach across multiple DNN architectures and datasets, compare it with a gradient-based baseline, and demonstrate that it achieves high fault coverage (up to 100%) using up to 16× fewer test patterns across the evaluated DNN architectures.