Usually students consume learning material and write an exam at the end of the lecture. Such a process follows a summative learning pattern, which can be considered a standard approach at universities. Studies in educational theory indicate, however, that active involvement–instead of passive consumption–should be fostered in learning since active learning proved to be superior to passive learning. To benefit from active learning arrangements, we implemented an active involvement of students into the exam preparation for an introduction to Information Systems course at the University of Cologne. Students were asked to design exercises and provide solutions to selected topics. Subsequently, they received feedback to their submissions, which supports the self-assessment on the subject. An empirical evaluation shows general agreement for such active involvement of students and also indicates that students participating in the task creation are more likely to pass an exam than students denying the participation. This paper presents our crowdsourcing-based learning approach and discusses challenges for its implementation.