Evaluating the Overlap of User Preferences with Search Engine Performances using UCISE

Background: The assessment of information systems such as Search Engines (SEs) from the users’ perspective needs to assure that it does not introduce a bias towards the systems’ technical implementation. Instead, the socio-technical nature of SEs and a users’ contextual evaluation based on previous experiences should be at the center of attention.
Objective: Therefore, we introduce a methodological approach for a User Centered Information System Evaluation (UCISE) and apply it to a SE evaluation. This approach allows for comparing selected SEs based on a derived set of user-based evaluation criteria.
Method: Our methodology consists of four steps, including a criteria synthesis, a criteria importance assessment by a large-scale survey and an expert assessment of selected SE performances regarding the user-based criteria. In the final step, we visualize the fit between the users’ criteria preferences and the assessed SE performances.
Results: The first step of our methodological approach enabled the synthesis of a set of ten criteria for the SE differentiation from user-perspective. Of these ten criteria, only the three criteria ‘search result quality’, ‘ease of use’ and ‘privacy protection’ emerged as being perceived as ‘extremely important’ or ‘important’ by the user. ... mehrThe assessment of selected SEs with regard to all ten criteria shows significant differences between the providers, which lead to a very differentiated picture in the visualization of the fit of providers to user preferences.
Conclusion: Our research indicates that seven out of ten criteria are not decisive for a user’s SE selection. The ‘search result quality’ emerges as the only criterion being perceived as ‘extremely important’ for a user’s SE evaluation, followed by ‘ease of use’ and ‘privacy protection’. When assessing the fits of three selected SE providers to the derived criteria, Google largely outperforms its competitors. This appears to be realized by overcompensating its major weakness in ‘privacy protection’ by superior results in ‘search result quality’ and ‘ease of use’ where the other assessed providers noticeably lack behind.