Closed-Form Information-Theoretic Roughness Measures for Mixture Densities

In estimation, control, and machine learning under uncertainties, latent variables are usually described by a probability density function (pdf). The optimal reconstruction of a continuous pdf from given samples or moments is an important and ubiquitous task. Unfortunately, it typically results in an underdetermined optimization problem, as the pdf is not fully constrained by the given samples or moments. For regularization, we use Fisher Information (FI) that acts as a roughness measure, i.e., selects the smoothest pdf fulfilling the constraints, in an information-theoretic sense. For the important class of mixture densities, FI can only be computed numerically. In this paper, we derive a closed-form solution for FI for mixtures by transforming the problem to the space R of root mixtures (RMs). This results in a tandem processing scheme simultaneously working in the original mixture space M and the corresponding RM space: The density parameters are optimized in root mixture space based on the closed-form FI. The desired constraints are evaluated in the original mixture space M.