Release Time:2019-03-09  Hits:

Indexed by: Journal Article

Date of Publication: 2014-02-01

Journal: NEURAL NETWORKS

Included Journals: Scopus、PubMed、EI、SCIE

Volume: 50

Page Number: 72-78

ISSN: 0893-6080

Key Words: Feedforward neural networks; Batch gradient method; Smoothing L-1/2 regularization; Convergence

Abstract: The aim of this paper is to develop a novel method to prune feedforward neural networks by introducing an L-1/2 regularization term into the error function. This procedure forces weights to become smaller during the training and can eventually removed after the training. The usual L-1/2 regularization term involves absolute values and is not differentiable at the origin, which typically causes oscillation of the gradient of the error function during the training. A key point of this paper is to modify the usual L-1/2 regularization term by smoothing it at the origin. This approach offers the following three advantages: First, it removes the oscillation of the gradient value. Secondly, it gives better pruning, namely the final weights to be removed are smaller than those produced through the usual L-1/2 regularization. Thirdly, it makes it possible to prove the convergence of the training. Supporting numerical examples are also provided. (C) 2013 Elsevier Ltd. All rights reserved.