In practical applications, state estimation requires the consideration of stochastic and systematic errors. If both error types are present, an exact probabilistic description of the state estimate is not possible, so that common Bayesian estimators have to be questioned. This paper introduces a theoretical concept, which allows for incorporating unknown but bounded errors into a Bayesian inference scheme by utilizing sets of densities. In order to derive a tractable estimator, the Kalman filter is applied to ellipsoidal sets of means, which are used to bound additive systematic errors. Also, an extension to nonlinear system and observation models with ellipsoidal error bounds is presented. The derived estimator is motivated by means of two example applications.