Objective. The aim of this study is to examine the relationship between different types of daily life physical activity (PA) and physical fitness (PF) and health throughout adulthood. Methods. A total of 723 men and women, aged 28–76 years, participated 1681 times during four measurement points from 1992 to 2010 in this study. We assessed self-reported PA, anthropometrics, physical health status (HS), and PF in each study year. Hierarchical linear modeling (HLM) was used to analyze the measures. Results. PF and HS worsened with increasing age while sports activity (SA) declined. The modeling showed that sex, age, and SES play important roles concerning PA, PF, and HS. Athletes show higher HS and HF than nonathletes. Habitual activity (HA) also showed a positive relationship with PF and HS, but effects were lower than for SA. Work related activity (WRA) showed no meaningful relationship with PF or HS. Conclusions. Comparable amounts of PA can lead to different effects on PF or HS. Our findings underline the importance of contexts, content, and purposes of PA when health or fitness benefits are addressed. Simply moving your body is not enough.