Human-Centered Crowd Feedback for Information Systems Development

User involvement in information system (IS) development is crucial for project success and user satisfaction. Traditional evaluation methods like usability tests and focus groups are often limited by scale, cost, and time constraints. Crowdsourcing offers a scalable, cost-effective, and timely alternative for gathering user feedback during IS development. Despite its potential, the feedback provider’s perspective is often overlooked, raising concerns about meeting their needs and optimizing feedback quality. This dissertation explores crowd-feedback systems through a human-centered design process to balance the needs of feedback providers with the requester’s goal of high-quality feedback collection. The dissertation tackles two main challenges: the need to better understand design feature impacts in crowd-feedback systems and the failure of existing crowd-feedback systems to collect user feedback in a real context. The research includes five studies delivering four innovative design solutions, evaluating their impacts, and understanding different stakeholder perceptions. The first study maps out the crowd feedback landscape, identifying key inputs, design features, crowdsourcing configurations, and effects. ... mehrBased on this, it proposes multiple future research directions. Building upon these foundations, the second study designs ’Feeasy’, a human-centered crowd-feedback system, and evaluates the impact of its features on feedback providers and feedback outcomes. The third study develops a configuration system enabling the customization of feedback requests. Addressing the second challenge, the fourth study tests integrating user and crowd feedback through the ’CrowdSurfer,’ a browser extension, noting that feedback tasks embedded in regular internet use are seen as less effort but may reduce feedback quality. The fifth study introduces preference-based personalization in microtasking, assessing its influence on crowdworker performance and offering design guidelines for personalized microtasks. This research contributes to the fields of human-computer interaction and information systems by detailing crowd-feedback system design and investigating personalized crowdsourcing systems in real contexts. The findings provide actionable design knowledge for practitioners to improve IS evaluations’ scalability and human-centricity. The four developed artifacts apply this knowledge to real-world crowd feedback scenarios, aiming to improve IS evaluations by focusing on stakeholders’ needs.