With digitalization and virtual entertainment being the megatrends of the 21st century, there is reasonable concern about the role of physical activity (PA) in the daily life of children and adolescents. To identify risk-groups with insufficient PA and to guide interventions, continuous and representative tracking of PA is crucial. In this paper, representative PA data of children and adolescents from the Motorik-Modul (MoMo) baseline study (2003–2006, N = 4,528) is compared to those of Wave 2 (2014–2017, N = 3,708). Participants aged 4–17 were drawn out of 167 sample points in Germany and the data was weighted to ensure representativeness for Germany. Organized (sports clubs and schools) and unorganized (unorganized sports and playing outside) PA was measured by questionnaire and stratified by sex, age, and socioeconomic status. Contrary to common expectation, overall PA remained stable among youths in the past ten years, however, there is an ongoing trend towards organized forms of PA at the expense of unorganized sports and playing outside. Besides different trends in settings, there is inequality in PA distribution among socioeconomic status and gender, unequally pronounced in different settings.