Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Mammadov M., Tas E., Omay R. E.

JOURNAL OF STATISTICAL COMPUTATION AND SIMULATION, vol.78, no.11, pp.1055-1064, 2008 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 78 Issue: 11
  • Publication Date: 2008
  • Doi Number: 10.1080/00949650701496172
  • Journal Name: JOURNAL OF STATISTICAL COMPUTATION AND SIMULATION
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus
  • Page Numbers: pp.1055-1064
  • Keywords: Multilayer neural networks, Faster learning, Learning rate, Momentum, Local quadratic approximation, QUADRATIC-FUNCTIONS, STEEPEST DESCENT, MOMENTUM TERM
  • Anadolu University Affiliated: Yes

Abstract

An acceleration of backpropagation algorithm with momentum (BPM) is introduced. At every stage of the learning process, local quadratic approximation of the error function is performed and the Hessian matrix of the quadratic function is approximated. Effective learning rate and momentum factor are determined by means of maximum and minimum eigenvalues of the approximated Hessian matrix at each step. BPM algorithm is modified so as to work automatically with these effective parameters. Performance of this new approach is demonstrated in comparison with well-known training algorithms on conventional problems by an experimental evaluation.