Abstract—Model pruning is an important technique in real-world machine learning problems, especially in deep learning. This technique has provided some methods for compressing a large model to a smaller model while retaining the most accuracy. However, a majority of these approaches require a full original training set. This might not always be possible in practice if the model is trained in a large-scale dataset or on a dataset whose release poses privacy. Although we cannot access the original training set in some cases, pre-trained models are available more often. This paper aims to solve the model pruning problem without the initial training set by finding the sub-networks in the initial pre-trained model. We propose an approach of using genetic algorithms (GA) to find the sub-networks systematically and automatically. Experimental results show that our algorithm can find good sub-networks efficiently. Theoretically, if we had unlimited time and hardware power, we could find the optimized sub-networks of any pre-trained model and achieve the best results in the future. Our code and pre-trained models are available at: https://github.com/sun-asterisk-research/ga_pruning_research.
Index Terms—Genetic algorithm - GA, model compression, data-free learning..
Toan Pham Van, Thanh Nguyen Tung, Linh Bao Doan are with Sun-Asterisk Inc., Vietnam (e-mail: pham.van.toan@sun-asterisk.com, nguyen.tung.thanh@sun-asterisk.com, doan.bao.linh@sun-asterisk.com).
Ta Minh Thanh is with Le Quy Don Technical University, Ha Noi, Vietnam. He is now with the Faculty of Information Technology, Le Quy Don Technical University (e-mail: thanhtm@lqdtu.edu.vn).
[PDF]
Cite: Toan Pham Van, Thanh Nguyen Tung, Linh Bao Doan, and Thanh Ta Minh, "An Evolution Approach for Pre-trained Neural Network Pruning without Original Training Dataset," International Journal of Knowledge Engineering vol. 8, no. 1, pp. 1-6, 2022.
Copyright © 2022 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
