Feed-Forward-Only Training of Neural Networks

While artificial neural networks have reached immense advances over the last decade, the underlying approach to training neural networks, that is, solving the credit assignment problem by computing gradients with back-propagation, has remained largely the same. Nonetheless, back-propagation has long been criticized for being biologically implausible as it relies on concepts that are not viable in the brain. With delayed error forward projection (DEFP), I introduce a feed-forward-only training algorithm that solves two core issues for biological plausibility: the weight transport and the update locking problem. It is based on the similarly plausible direct random target projection algorithm but improves the approximated gradients by using delayed error information as a sample-wise scaling factor in place of the targets. By evaluating delayed error forward projection on image classification with fully-connected and convolutional neural networks, I find that it can achieve higher accuracy than direct random target projection, especially for fully- connected networks. Interestingly, scaling the updates with the error yields significantly better results than scaling with the gradient of the loss for all networks and datasets. ... mehrIn total, delayed error forward projection demonstrates the applicability of feed-forward-only training algorithms. This offers exciting new possibilities for both in-the-loop training on neuromorphic devices and pipelined parallelization.