KIT | KIT-Bibliothek | Impressum | Datenschutz

Feed-Forward-Only Training of Neural Networks

Flügel, Katharina 1; Streit, Achim ORCID iD icon 1
1 Scientific Computing Center (SCC), Karlsruher Institut für Technologie (KIT)

Abstract (englisch):

While artificial neural networks have reached immense advances over the last decade, the underlying approach to training neural networks, that is, solving the credit assignment problem by computing gradients with back-propagation, has remained largely the same. Nonetheless, back-propagation has long been criticized for being biologically implausible as it relies on concepts that are not viable in the brain. With delayed error forward projection (DEFP), I introduce a feed-forward-only training algorithm that solves two core issues for biological plausibility: the weight transport and the update locking problem. It is based on the similarly plausible direct random target projection algorithm but improves the approximated gradients by using delayed error information as a sample-wise scaling factor in place of the targets. By evaluating delayed error forward projection on image classification with fully-connected and convolutional neural networks, I find that it can achieve higher accuracy than direct random target projection, especially for fully- connected networks. Interestingly, scaling the updates with the error yields significantly better results than scaling with the gradient of the loss for all networks and datasets. ... mehr

Volltext (Version 2) §
DOI: 10.5445/IR/1000153057/v2
Veröffentlicht am 05.10.2023
Cover der Publikation
Zugehörige Institution(en) am KIT Scientific Computing Center (SCC)
Publikationstyp Hochschulschrift
Publikationsdatum 01.12.2021
Sprache Englisch
Identifikator KITopen-ID: 1000153057
HGF-Programm 46.21.04 (POF IV, LK 01) HAICU
Verlag Karlsruher Institut für Technologie (KIT)
Umfang XI, 71 S.
Art der Arbeit Abschlussarbeit - Master
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page