Release Time:2019-03-09  Hits:

Indexed by: Journal Article

Date of Publication: 2014-05-05

Journal: NEUROCOMPUTING

Included Journals: Scopus、EI、SCIE

Volume: 131

Page Number: 208-216

ISSN: 0925-2312

Key Words: Feedforward neural networks; Online gradient method; Smoothing L-1/2 regularization; Boundedness; Convergence

Abstract: Minimization of the training regularization term has been recognized as an important objective for sparse modeling and generalization in feedforward neural networks. Most of the studies so far have been focused on the popular L-2 regularization penalty. In this paper, we consider the convergence of online gradient method with smoothing L-1/2 regularization term. For normal L-1/2 regularization, the objective function is the sum of a non-convex, non-smooth, and non-Lipschitz function, which causes oscillation of the error function and the norm of gradient. However, using the smoothing approximation techniques, the deficiency of the normal L-1/2 regularization term can be addressed. This paper shows the strong convergence results for the smoothing L-1/2 regularization. Furthermore, we prove the boundedness of the weights during the network training. The assumption that weights are bounded is no longer needed for the proof of convergence. Simulation results support the theoretical findings and demonstrate that our algorithm has better performance than two other algorithms with L-2 and normal L-1/2 regularizations respectively. (C) 2013 Elsevier B.V. All rights reserved.