Mamba-based Thermal Image Dehazing
DOI:
https://doi.org/10.51408/1963-0126Keywords:
Thermal Image, Image Dehazing, Thermal Image Enhancement, MambaAbstract
Atmospheric phenomena such as rain, snow, urban, forest fires, and artificial disasters can degrade image quality across various applications, including transportation, driver assistance systems, surveillance, military, and remote sensing. Image dehazing techniques aim to reduce the effects of haze, dust, fog, and other atmospheric distortions, enhancing image quality for better performance in computer vision tasks. Haze not only obscures details but also reduces contrast and color fidelity, significantly impacting the accuracy of computer vision (CV) models used in object detection, image classification, and segmentation. While thermal infrared (TIR) imaging is often favored for long-range surveillance and remote sensing due to its resilience to haze, atmospheric conditions can still degrade TIR image quality, especially in extreme environments.
This paper introduces MTIE-Net, a novel Mamba-based network for enhancing thermal images degraded by atmospheric phenomena like haze and smoke. MTIE-Net leverages the Enhancement and Denoising State Space Model (EDSSM), which combines convolutional neural networks with state-space modeling for effective denoising and enhancement. We generate synthetic hazy images and employ domain-specific transformations tailored to thermal image characteristics to improve training in low-visibility conditions. Our key contributions include using the Mamba architecture with 2D Selective Scanning for thermal image enhancement, developing a specialized Enhancement and Denoising module, and creating a labeled thermal dataset simulating heavy haze. Evaluated on the M3DF dataset of long-range thermal images, MTIE-Net surpasses state-of-the-art methods in both quantitative metrics (PSNR, SSIM) and qualitative assessments of visual clarity and edge preservation. This advancement significantly improves the reliability and accuracy of critical systems used in remote sensing, surveillance, and autonomous operations by enhancing image quality in challenging environments.
References
M. Pavethra and M. Umadevi, “Deep Learning approaches for Image Dehazing”, 6th IEEE International Conference on Recent Advances and Innovations in Engineering (ICRAIE), Kedah, Malaysia, 2021.
A. Parihar, et al., “A comparative study of image dehazing algorithms” In IEEE 5th International Conference on Communication and Electronics Systems (ICCES), pp. 766–771, 2020.
J. Baek, S. Hong, J. Kim and E. Kim, “Efficient Pedestrian detection at nighttime using a thermal camera” Sensors, vol. 17, no. 8, p. 1850, 2017.
A. Goldberg, T. Fischer and Z. Derzko, “Application of dual-band infrared focal plane arrays to tactical and strategic military problems” in Proc. SPIE, vol. 4820, pp. 500–514, 2003.
W. Wong, H. Lim, C. Loo and W. Lim, “Home alone faint detection surveillance system using thermal camera” in Proc. 2nd Int. Conf. Comput. Res. Develop., pp. 747–751, 2010.
J. Berni, et al., “Remote sensing of vegetation from UAV platforms using lightweight multispectral and thermal imaging sensors”, Int. Arch. Photogramm. Remote Sens. Spat. Inform. Sci. 38:6, 2008.
J. Wang, et al., “Range-restricted pixel difference global histogram equalization for infrared image contrast enhancement”, Opt Rev 28, 145–158, 2021.
C. Liu et al., “Adaptive Contrast Enhancement for Infrared Images Based on the Neighborhood Conditional Histogram” Remote Sensing, 2019.
T. Mudavath and V. Niranjan, “Thermal image enhancement for adverse weather scenarios: a wavelet transform and histogram clipping approach”, SIViP 18, pp. 6547–6558, 2024.
R. Soundrapandiyan, et al. “A comprehensive survey on image enhancement techniques with special emphasis on infrared images”, Multimed Tools Appl 81, 9045–9077, 2022.
A. Grigoryan and S. Agaian, “Asymmetric and symmetric gradient operators with application in face recognition in Renaissance portrait art”, Proc. of SPIE, Defense + Commercial Sensing, Mobile Multimedia/Image Processing, Security, and Applications, vol. 10993, p. 12, Baltimore, Maryland, April 2019.
S. Hovhannisyan et al., “AED-Net: A Single Image Dehazing”, in IEEE Access, vol. 10, pp. 12465-12474, 2022.
S. Hovhannisyan, H. Gasparyan and S. Agaian, “EOD-Net: enhancing object detection in challenging weather conditions using an innovative end-to-end dehazing network”, Twelfth International Conference on Image Processing Theory, Tools and Applications (IPTA), Paris, France, pp. 1-6, 2023.
D. Berman, T. Treibitz and S. Avidan, “Single image dehazing using haze-lines”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 3, pp. 720–734, 2020.
Z. Chen, Z. He and Z. -M. Lu, “DEA-Net: single image dehazing based on detail-enhanced convolution and content-guided attention”, IEEE Transactions on Image Processing, vol. 33, pp. 1002-1015, 2024.
A. Parihar, Y. Gupta, Y. Singodia, V. Singh and K. Singh, “A comparative study of image dehazing algorithms”, 5th International Conference on Communication and Electronics Systems (ICCES), IEEE, pp 766–771, 2020.
B. Li et al., “Benchmarking single image dehazing and beyond”, IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 492–505, 2019.
K. Lee et al., “Brightness-based convolutional neural network for thermal image enhancement”, IEEE Access, vol. 5, pp. 26867-26879, 2017.
K. Xiaodong et al., “Single infrared image enhancement using a deep convolutional neural network” Neurocomputing, vol. 332, pp. 119-128, 2019.
A. Gu and T. Dao, “Mamba: linear-time sequence modeling with selective state spaces”, arXiv preprint arXiv:2312.00752, 2023.
E. Nguyen et al., “Modeling images and videos as ´ multidimensional signals with state spaces”, Advances in neural information processing systems, vol. 35, pp. 2846–2861, 2022.
B. Patro and V. Agneeswaran, “Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges”, arXiv preprint arXiv:2404.16112, 2024.
J. Smith, A. Warrington and S. Linderman, “Simplified state space layers for sequence modeling, " arXiv preprint arXiv:2208.04933, 2023.
L. Zhu et al., “Vision mamba: Efficient visual representation learning with bidirectional state space model”, arXiv preprint arXiv:2401.09417, 2024.
Z. Wang, A. Bovik, H. Sheikh and E. Simoncelli, “Image quality assessment: From error visibility to structural similarity”, IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
S. Agaian et al., “A new measure of image enhancement”, IASTED Int. Conf. on Signal Proc. & Comm., pp.19-22, Sept. 2000.
W. Yang et al., “Blind image quality assessment with a multi-task CNN for enhanced measurement”, Sig. Pr. Im. Com., 105, 116672, 2022.
X. Xu, R. Wang, C. Fu andJ. Jia, “Snr-aware low-light image enhancement”, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17714–17724, 2022.
H. Zhou, D. Greenwood and S. Taylor, “Self-supervised monocular depth estimation with internal feature fusion”, British Machine Vision Conference (BMVC), 2021.
F. Erlenbusch et al., “Thermal infrared single image dehazing and blind image quality assessment”, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, BC, Canada, 2023, pp. 459-469, 2023.
A. Reza, “Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement”, J. VLSI Sig. Proc. Syst. Signal Image Vid. Tech.., vol. 38, no. 1, pp. 35-44, 2004.
A. Buades, B. Coll and J. Morel, “Non-local means denoising," Image Processing On Line, 1, pp. 208–212, 2011.
B. Gupta and M. Tiwari, “Minimum mean brightness error contrast enhancement of color images using adaptive gamma correction with color preserving framework”, Optik 127, no. 4, pp. 1671-1676, 2016.
Q. Wang and and R. Ward, “Fast image/video contrast enhancement based on weighted thresholded histogram equalization”, IEEE Trans. On Consumer Electronics 53, no. 2, 757-764, 2007.
K. Lee et al., “Brightness-based convolutional neural network for thermal image enhancement”, IEEE Access, vol. 5, pp. 26867-26879, 2017.
K. Xiaodong et al., “Single infrared image enhancement using a deep convolutional neural network”, Neurocomputing, vol. 332, pp. 119-128, 2019.
J. Liu et al., “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection”, IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
C. Yuan, D. Zhao and S. Agaian, “MUCM-Net: a mamba powered ucm-net for skin lesion segmentation”, arXiv preprint arXiv:2405.15925.
A. Oulefki, T. Trongtirakul, S. Agaian et al., “multi-view vr imaging for enhanced analysis of dust accumulation on solar panels”, Solar Energy, vol. 279, pp. 112708, 2024.
A. Oulefki, Y. Himeur, T. Trongtirakul et al., “Detection and analysis of deteriorated areas in solar pv modules using unsupervised sensing algorithms and 3D augmented reality”, Heliyon, vol. 9, pp. e27973, 2024.
H. Ayunts, A. Grigoryan and S. Agaian, “Novel entropy for enhanced thermal imaging and uncertainty quantification”, Entropy, vol. 26, 374, 2024.
A. Wilson et al., “Recent advances in thermal imaging and its applications using machine learning: A review”, IEEE Sens. J., vol. 23, pp. 3395–3407, 2023
T. Trongtirakul and S. Agaian "New retinex model-based infrared image enhancement", Proc. SPIE 12526, Multimodal Image Exploitation and Learning, 1252606, 2023.
R. Kalman, “A new approach to linear filtering and prediction problems”, 1960.
S. Hochreiter and J. Schmidhuber, “Long Short-Term memory”, Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
H. Zhang et. al., “A survey on visual mamba”, arXiv preprint, arXiv:2404.15956v2, 2024.
L. Lu, et. al., Comparative study of histogram equalization algorithms for image enhancement. Mob. Multimed. Image Process. Secur. Appl. 2010, 7708, 337–347
T. Trongtirakul and S. Agaian, “New Retinex model-based infrared image enhancement”, Proc. SPIE 12526, Multimodal Image Exploitation and Learning, 1252606, 2023.
T. Trongtirakul, and S. Agaian, “Transmission map optimization for single image dehazing”, Proc. SPIE 12100, Multimodal Image Expl. and Learning, 121000C, May, 2022.
A. Horé and D. Ziou, “Image quality metrics: PSNR vs. SSIM”, 20th International Conference on Pattern Recognition, pp. 2366-2369, Turkey, 2010.
S. Agaian et al., “A new measure of image enhancement”, IASTED Int. Conf. on Signal Proc. & Comm., Sept. 2000.
W. Yang et al., “Blind image quality assessment with a multi-task CNN for enhanced measurement”, Sig. Pr.: Im. Com., 105, 116672, 2022.
S. Agaian, M. Roopaei and D. Akopian, “Thermal-image quality measurements”, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.
T. Dao and A. Gu, “Transformers are ssms: Generalized models and efficient algorithms through structured state space duality”, arXiv preprin arXiv:2405.21060, 2024.
M. Ahamed and Q. Cheng, “TSCMamba: Mamba meets multi-view learning for time series classification”, arXiv preprint arXiv:2406.04419, 2024.
Y. Cao and W. Zhang, “Mamba4KT: An Efficient and Effective Mamba-based Knowledge Tracing Model”, arXiv preprint arXiv:2405.16542, 2024.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Sargis A. Hovhannisyan
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
