Single Image Joint Motion Deblurring and Super-Resolution Using the Multi-Scale Channel Attention Modules
DOI:
https://doi.org/10.51408/1963-0076Keywords:
Motion deblurring, super-resolution, channel attentionAbstract
During the last decade, deep convolutional neural networks have significantly advanced the single image super-resolution techniques reconstructing realistic textural and spatial details. In classical image super-resolution problems, it is assumed that the low-resolution image has a certain downsampling degradation. However, complicated image degradations are inevitable in real-world scenarios, and motion blur is a common type of image degradation due to camera or scene motion during the image capturing process. This work proposes a fully convolutional neural network to reconstruct high-resolution sharp images from the given motion blurry low-resolution images. The deblurring subnetwork is based on multi-stage progressive architecture, while the super-resolution subnetwork is designed using the multi-scale channel attention modules. A simple and effective training strategy is employed where a pre-trained frozen deblurring module is used to train the super-resolution module. The deblurring module is unfrozen in the last training phase. Experiments show that, unlike the other methods, the proposed method reconstructs relatively small structures and textural details while successfully removing the complex motion blur. The implementation code and the pre-trained model are publicly available at https://github.com/misakshoyan/joint-motion-deblur-and-sr.
References
A. Vaswani et al. “Attention Is All You Need”, arXiv preprint arXiv:1706.03762, 2017.
X. Tao et al., “Scale-recurrent network for deep image deblurring”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, USA, pp. 8174–8182, 2018.
S. W. Zamir et al., “Multi-Stage Progressive Image Restoration”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, pp. 14816-14826, 2021.
S. W. Zamir et al. “Restormer: Efficient Transformer for High-Resolution Image Restoration”, arXiv preprint arXiv:2111.09881, 2021.
X. Mao et al. “Deep Residual Fourier Transformation for Single Image Deblurring”, arXiv preprint arXiv:2111.11745, 2021.
S. Nah, T. H. Kim, and K. M. Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, pp. 257-265, 2017.
X. Chu et al. “Revisiting Global Statistics Aggregation for Improving Image Restoration”, arXiv preprint arXiv:2112.04491, 2021.
Y. Zhang et al., “Image super-resolution using very deep residual channel attention networks”, Proceedings of European Conference on Computer Vision (ECCV), Munich, Germany, pp. 294-310, 2018.
T. Dai et al., “Second-Order Attention Network for Single Image Super-Resolution”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, USA, pp. 11057-11066, 2019.
B. Niu et al., “Single image super-resolution via a holistic attention network”, Proceedings of European Conference on Computer Vision (ECCV), Glasgow, UK, pp. 191–207, 2020.
J. Liang et al., “SwinIR: Image Restoration Using Swin Transformer”, Proceedings of IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, Canada, pp. 1833-1844, 2021.
B. Lim et al., “Enhanced deep residual networks for single image super-resolution”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, USA, pp. 1132-1140, 2017.
M. Shoyan et al., “Single Image Joint Motion Deblurring and Super-Resolution”, Proceedings of 13th International Conference on Computer Science and Information Technologies (CSIT), Yerevan, Armenia, pp. 182-186, 2021.
X. Zhang et al. “Gated fusion network for degraded image super resolution”, International Journal of Computer Vision, vol. 128, no. 6, pp. 1699-1721, 2020.
S. Nah et al., “NTIRE 2021 Challenge on Image Deblurring”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, USA, pp. 149-165, 2021.
H. Bai et al, “Learning A Cascaded Non-Local Residual Network for Super-resolving Blurry Images”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, USA, pp. 223-232, 2021.
Wikipedia, (2014) The peak signal-to-noise ratio, [Online]. Available: https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio
Z. Wang et al., “Image quality assessment: from error visibility to structural similarity”, IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.
R. Zhang et al., “The unreasonable effectiveness of deep features as a perceptual metric”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, USA, pp. 586-595, 2018.
S. Xi, J. Wei and W. Zhang, “Pixel-Guided Dual-Branch Attention Network for Joint Image Deblurring and Super-Resolution”, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, USA, pp. 532-540, 2021.
R. Xu et al., “EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, USA, pp. 414-423, 2021.
Y. Dai et al., “Attentional Feature Fusion”, Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, pp. 3559-3568, 2021.
T. H. Kim, B. Ahn and K. M. Lee, “Dynamic Scene Deblurring”, Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 3160-3167, 2013.
J. Pan et al., “Blind image deblurring using dark channel prior”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, pp. 1628-1636, 2016.
R. Gonzalez and R. Woods, Digital Image Processing, 4th ed., Pearson, New York, 2018.
M. Bevilacqua et al. “Low-complexity single-image super-resolution based on nonnegative neighbor embedding”, Proceedings of British Machine Vision Conference (BMVC), Guildford, UK, paper 135, pp. 1-10, 2012.
R. Timofte, V. D. Smet, and L. V. Gool. “A+: Adjusted anchored neighborhood regression for fast super-resolution”, Proceedings of Asian Conference on Computer Vision (ACCV). Singapore, Singapore, pp. 111-126, 2014.
J. Hu, L. Shen and G. Sun, “Squeeze-and-Excitation Networks”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132-7141, 2018.
W. Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, pp. 1874-1883, 2016.
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: convolutional networks for biomedical image segmentation”, Proceedings of Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, pp. 234-241, 2015.
P. Charbonnier et al, “Two deterministic half-quadratic regularization algorithms for computed imaging”, Proceedings of 1st International Conference on Image Processing, Austin, USA, pp. 168-172, 1994.
S. Nah et al., “NTIRE 2019 challenges on video deblurring and super-resolution: Dataset and study”, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, USA, pp. 1996–2005, 2019.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization”, arXiv preprint arXiv:1412.6980, 2014.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
