Directed-Weighting Group Lasso For Eltwise Blocked CNN Pruning

Ke Zhan (Beijing University Of Technology), Shimiao Jiang (Alibaba, Inc.), Yu Bai (JD.com, Inc.), Yi Li (JD.com, Inc)

Abstract
Eltwise layer is a commonly used structure in the multi-branch deep learning network. In a filter-wise pruning procedure, due to the specific operation of the eltwise layer, all its previous convolutional layers should vote for which filters by index to be pruned. Since only an intersection of the voted filters was pruned, the compression rate is limited. The work proposes a method called Directed-Weighting Group Lasso (DWGL), which enforces an index-wise incremental (directed) coefficient on the filter-level group lasso items, so that the low index filters getting high activation tend to be kept while the high index ones tend to be pruned. When using DWGL, less filter is retained during the voting process and the compression rate can be boosted. The paper test the proposed method on ResNet series networks. On CIFAR-10, it achieved a 75.34% compression rate on ResNet-56 with a 0.94% error increment, and a 52.06% compression rate on ResNet-20 with a 0.72% error increment. On ImageNet, it achieved a 53% compression rate with ResNet-50 with a 0.6% error increment, which speed up the network by 2.23 times, it further achieved a 75% compression rate on ResNet-50 with a 1.2% error increment, which speed up the network by 4 times.

DOI
10.5244/C.33.51
https://dx.doi.org/10.5244/C.33.51

Files
Paper (PDF)

BibTeX