Training Quantized Neural Networks with ADMM Approach

Deep learning models have achieved remarkable success in various learning tasks, but their high computational costs pose challenges for deployment in resource-limited scenarios. In this paper, we focus on addressing this issue by quantizing deep learning models, where network weights are represented by a smaller number of bits. We formulate this problem as a discrete optimization problem and draw inspiration from the Alternating Direction Method of Multipliers to optimize the parameters in a neural network. We introduce two approaches to quantize neural networks using the Alternating Direction Method of Multipliers algorithm. The first approach is a gradient-free optimization method for training the quantized neural network. It avoids many problems of gradient descent, such as saturation effects and saddle points. In contrast, the second approach is a gradient-based method to quantize the neural network layerwisely using Alternating Direction Method of Multipliers based on a pre-trained neural network. After each layer is quantized, the parameters of non-quantized layers are updated to compensate for the loss of accuracy.