Caffe for Deep Compression
This is a simple caffe implementation of Deep Compression(https://arxiv.org/abs/1510.00149), including weight prunning and quantization.
According to the paper, the compression are implemented only on convolution and fully-connected layers.
Thus we add a CmpConvolution and a CmpInnerProduct layer.
The params that controlls the sparsity including:
For a better understanding, please see the examples/mnist and run the demo script, which automatically compresses a pretrained MNIST LeNet caffemodel.
# clone repository and make $ git clone https://github.com/may0324/DeepCompression-caffe.git $ cd DeepCompression-caffe $ make -j 32 # run demo script, this will finetune a pretrained model $ python examples/mnist/train_compress_lenet.py
the sparse parameters of lenet are set based on the paper as follows:
|layer name||sparse ratio||quantization num|
In practice, the layers are much more sensitive to weight prunning than weight quantization.
So we suggest to do weight prunning layer-wisely and do weight quantization finally since it almost does no harm to accuary.
In the script demo, we set the sparse ratio (the ratio of pruned weights) layer-wisely and do each finetuning iteration. After all layers are properly pruned, weight quantization are done on all layers simultaneously.
The final accuracy of finetuned model is about 99.06%, you can check if the weights are most pruned and weight-shared for sure.
The size of finetuned model is still the same as the original one since it is stored in 'caffemodel' format. Although most of the weights are pruned and shared, the weights are still stored in float32. You can only store the non-zero weight and cluster center to reduce the redundacy of finetuned model, please refer to the paper.
Please refer to http://blog.csdn.net/may0324/article/details/52935869 for more.