Open-Pit Mining Area Extraction Using Multispectral Remote Sensing Images: A Deep Learning Extraction Method Based on Transformer
Abstract
:1. Introduction
- (1)
- In order to resolve the segmentation errors caused by the disappearance of local area features such as edges and textures in the mining area, this article proposes a multi-scale local spatial feature complementary module. The module learns multi-scale local spatial features and supplements them into the global features of Transformer blocks so as to enhance the network’s ability to obtain local spatial detail information and improve the problem of disappearing small-scale object features and insufficient information expression.
- (2)
- The decoder for pixel-by-pixel classification ignores the importance of contextual learning when assigning labels to each pixel, and the upsampling of features is prone to cause blurring of the edges of the mining area. In order to fully utilize the learned contextual semantic features to solve the problem of sticky edges and boundary blurring in mining areas, this article proposes the attention mask decoder. It is able to retain the edge details of the mined area better and improve the accuracy degradation that may result from downsampling and then upsampling the feature map.
- (3)
- This article demonstrates the promising application of Transformer for the intelligent extraction of open-pit mining areas in complex environments. Considering Transformer’s good parallel computing and global feature acquisition capabilities, as well as SegMine’s good performance, it would be a promising model for the classification and extraction of open-pit mining areas.
2. Related Works
2.1. Land Cover Extraction by Deep Learning
2.2. Open-Pit Mining Area Extraction by Deep Learning
3. Methodology
3.1. Model Architecture
3.2. Encoder
3.3. Decoder
4. Experiments
4.1. Datasets and Settings
4.2. Quantitative Experimental Results
4.3. Qualitative Experimental Results
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Bauer, M.E. Remote sensing of environment: History, philosophy, approach and contributions, 1969–2019. Remote Sens. Environ. 2020, 237, 111522. [Google Scholar] [CrossRef]
- Tong, X.Y.; Xia, G.S.; Lu, Q.; Shen, H.; Li, S.; You, S.; Zhang, L. Land-cover classification with high-resolution remote sensing images using transferable deep models. arXiv 2022, arXiv:1807.05713. [Google Scholar] [CrossRef]
- Shendryk, Y.; Rist, Y.; Ticehurst, C.; Thorburn, P. Deep learning for multi-modal classification of cloud, shadow and land cover scenes in PlanetScope and Sentinel-2 imagery. ISPRS J. Photogramm. Remote Sens. 2019, 157, 124–136. [Google Scholar] [CrossRef]
- Zhou, W.; Ming, D.; Lv, X.; Zhou, K.; Bao, H.; Hong, Z. SO–CNN based urban functional zone fine division with VHR remote sensing image. Remote Sens. Environ. 2020, 236, 111458. [Google Scholar] [CrossRef]
- Maxwell, A.E.; Warner, T.A.; Fang, F. Implementation of machine-learning classification in remote sensing: An applied review. Int. J. Remote Sens. 2018, 39, 2784–2817. [Google Scholar] [CrossRef]
- Xia, G.S.; Yang, W.; Delon, J.; Yann, G.; Hong, S.; Henri, M. Structural high-resolution satellite image indexing. ISPRS TC VII Symp. 2009, 38, 298–303. Available online: https://hal.science/hal-00458685v1 (accessed on 1 January 2024).
- Xia, G.-S.; Liu, G.; Bai, X.; Zhang, L. Texture characterization using shape co-occurrence patterns. IEEE Trans. Image Process. A Publ. IEEE Signal Process. Soc. 2017, 26, 5005–5018. [Google Scholar] [CrossRef]
- Anwer, R.M.; Khan, F.S.; van de Weijer, J.; Monlinier, M.; Laaksonen, J. Binary patterns encoded convolutional neural networks for texture recognition and remote sensing scene classification. ISPRS J. Photogramm. Remote Sens. 2018, 138, 74–85. [Google Scholar] [CrossRef]
- Li, Q.; Mou, L.; Liu, Q.; Wang, Y.; Zhu, X.X. HSF-Net: Multiscale deep feature embedding for ship detection in optical remote sensing imagery. IEEE Trans. Geosci. Remote Sens. 2018, 56, 7147–7161. [Google Scholar] [CrossRef]
- Zhu, X.X.; Tuia, D.; Mou, L.; Xia, G.-S.; Zhang, L.; Xu, F.; Fraundorfer, F. Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geosci. Remote Sens. Mag. 2017, 5, 8–36. [Google Scholar] [CrossRef]
- Reichstein, M.; Camps-Valls, G.; Stevens, B.; Jung, M.; Denzler, J.; Carvalhais, N.; Prabhat. Deep learning and process understanding for data-driven Earth system science. Nature 2019, 566, 195–204. [Google Scholar] [CrossRef]
- Hong, D.; Gao, L.; Yokoya, N.; Yao, J.; Chanussot, J.; Du, Q.; Zhang, B. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE Trans. Geosci. Remote Sens. 2020, 59, 4340–4354. [Google Scholar] [CrossRef]
- Liu, Y.; Chen, D.; Ma, A.; Zhong, Y.; Fang, F.; Xu, K. Multiscale U-shaped CNN building instance extraction framework with edge constraint for high-spatial-resolution remote sensing imagery. IEEE Trans. Geosci. Remote Sens. 2020, 59, 6106–6120. [Google Scholar] [CrossRef]
- Ji, S.; Wei, S.; Lu, M. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set. IEEE Trans. Geosci. Remote Sens. 2018, 57, 574–586. [Google Scholar] [CrossRef]
- Belli, D.; Kipf, T. Image-conditioned graph generation for road network extraction. arXiv 2019, arXiv:1910.14388. [Google Scholar] [CrossRef]
- Lian, R.; Huang, L. DeepWindow: Sliding window based on deep learning for road extraction from remote sensing images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 13, 1905–1916. [Google Scholar] [CrossRef]
- Vargas-Munoz, J.E.; Srivastava, S.; Tuia, D.; Falcao, A.X. OpenStreetMap: Challenges and opportunities in machine learning and remote sensing. IEEE Geosci. Remote Sens. Mag. 2020, 9, 184–199. [Google Scholar] [CrossRef]
- Zhong, Y.; Han, X.; Zhang, L. Multi-class geospatial object detection based on a position-sensitive balancing framework for high spatial resolution remote sensing imagery. ISPRS J. Photogramm. Remote Sens. 2018, 138, 281–294. [Google Scholar] [CrossRef]
- Gao, Y.; Li, J.; Yang, T.; Deng, W.; Wang, D.; Cheng, H.; Ma, K. Stability analysis of a deep and large open-pit based on fine geological modeling and large-scale parallel computing: A case study of Fushun West Open-pit Mine, Geomatics. Nat. Hazards Risk 2023, 14, 2266663. [Google Scholar] [CrossRef]
- Zhang, C.; Harrison, P.A.; Pan, X.; Li, H.; Sargent, I.; Atkinson, P.M. Scale Sequence Joint Deep Learning (SS-JDL) for land use and land cover classification. Remote Sens. Environ. 2020, 237, 111593. [Google Scholar] [CrossRef]
- Zhou, Y.; Li, J.; Feng, L.; Zhang, X.; Hu, X. Adaptive scale selection for multiscale segmentation of satellite images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2017, 10, 3641–3651. [Google Scholar] [CrossRef]
- Cao, R.; Tu, W.; Yang, C.; Li, Q.; Liu, J.; Zhu, J.; Zhang, Q.; Li, Q.; Qiu, G. Deep learning-based remote and social sensing data fusion for urban region function recognition. ISPRS J. Photogramm. Remote Sens. 2020, 163, 82–97. [Google Scholar] [CrossRef]
- Kemker, R.; Salvaggio, C.; Kanan, C. Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning. ISPRS J. Photogramm. Remote Sens. 2018, 145, 60–77. [Google Scholar] [CrossRef]
- Ma, L.; Liu, Y.; Zhang, X.; Ye, Y.; Yin, G.; Johnson, B.A. Deep learning in remote sensing applications: A meta-analysis and review. ISPRS J. Photogramm. Remote Sens. 2019, 152, 166–177. [Google Scholar] [CrossRef]
- Ding, L.; Zhang, J.; Bruzzone, L. Semantic segmentation of large-size VHR remote sensing images using a two-stage multiscale training architecture. IEEE Trans. Geosci. Remote Sens. 2020, 58, 5367–5376. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef]
- Sharma, A.; Liu, X.; Yang, X.; Shi, D. A patch-based convolutional neural network for remote sensing image classification. Neural Netw. 2017, 95, 19–28. [Google Scholar] [CrossRef]
- Waldner, F.; Diakogiannis, F.I. Deep learning on edge: Extracting field boundaries from satellite images with a convolutional neural network. Remote Sens. Environ. 2020, 245, 111741. [Google Scholar] [CrossRef]
- Zhang, X.; Zhou, Y.; Luo, J. Deep learning for processing and analysis of remote sensing big data: A technical review. Big Earth Data 2022, 6, 527–560. [Google Scholar] [CrossRef]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16 × 16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2010.11929. [Google Scholar] [CrossRef]
- Bazi, Y.; Bashmal, L.; Al Rahhal, M.M.; Al Dayil, R.; Al Ajlan, N. Vision transformers for remote sensing image classification. Remote Sens. 2021, 13, 516. [Google Scholar] [CrossRef]
- Hao, S.; Wu, B.; Zhao, K.; Ye, Y.; Wang, W. Two-stream swin transformer with differentiable sobel operator for remote sensing image classification. Remote Sens. 2022, 14, 1507. [Google Scholar] [CrossRef]
- Ma, J.; Li, M.; Tang, X.; Zhang, X.; Liu, F.; Jiao, L. Homo–heterogenous transformer learning framework for RS scene classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2022, 15, 2223–2239. [Google Scholar] [CrossRef]
- Xiao, W.; Zhang, W.; Lyu, X.; Wang, X. Spatio-temporal patterns of ecological capital under different mining intensities in an ecologically fragile mining area in Western China: A case study of Shenfu mining area. J. Nat. Res 2020, 35, 68–81. [Google Scholar]
- Xiang, Y.; Zhao, Y.; Dong, J. Change Detection of Mining Areas in Remote Sensing Imagery Based on Improved UNet Twin Networks. China Coal Soc. 2019, 44, 3773–3780. [Google Scholar]
- Wang, C.; Chang, L.; Zhao, L.; Niu, R. Automatic identification and dynamic monitoring of open-pit mines based on improved mask R-CNN and transfer learning. Remote Sens. 2020, 12, 3474. [Google Scholar] [CrossRef]
- Zhang, F.; Wu, Y.; Yao, X.; Liang, Z. Opencast mining area intelligent extraction method for multi-source remote sensing image based on improved densenet. Remote Sens. Technol. Appl. 2020, 35, 673–684. [Google Scholar]
- Wang, J.; Gou, C.; Wu, Q.; Feng, H.; Han, J.; Ding, E.; Wang, J. Rtformer: Efficient design for real-time semantic segmentation with transformer. arXiv 2022, arXiv:2210.07124. [Google Scholar] [CrossRef]
- Strudel, R.; Garcia, R.; Laptev, I.; Schmid, C. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 7262–7272. [Google Scholar] [CrossRef]
- Xie, E.; Wang, W.; Yu, Z.; Anandkumar, A.; Alvarez, J.M.; Luo, P. SegFormer: Simple and efficient design for semantic segmentation with transformers. Adv. Neural Inf. Process. Syst. 2021, 34, 12077–12090. [Google Scholar] [CrossRef]
- Zhang, W.; Huang, Z.; Luo, G.; Chen, T.; Wang, X.; Liu, W.; Yu, G.; Shen, C. Topformer: Token pyramid transformer for mobile semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; pp. 12083–12093. [Google Scholar] [CrossRef]
- Cheng, B.; Misra, I.; Schwing, A.G.; Kirillov, A.; Girdhar, R. Masked-attention mask transformer for universal image segmentation. arXiv 2021, arXiv:2112.01527. [Google Scholar] [CrossRef]
- Chen, Z.; Duan, Y.; Wang, W.; He, J.; Lu, T.; Dai, J.; Qiao, Y. Vision transformer adapter for dense predictions. arXiv 2022, arXiv:2205.08534. [Google Scholar] [CrossRef]
Model | mIoU (%) | Precision (%) | Recall (%) | Dice (%) |
---|---|---|---|---|
Segmenter [38] | 85.68 | 85.29 | 84.11 | 90.32 |
RtFormer [39] | 84.23 | 88.85 | 81.54 | 90.59 |
SegFormer [40] | 85.56 | 85.95 | 85.56 | 90.60 |
TopFormer [41] | 85.53 | 88.77 | 84.94 | 91.55 |
Mask2Former [42] | 86.12 | 84.61 | 82.68 | 90.57 |
ViT-Adapter [43] | 85.78 | 85.63 | 83.07 | 90.69 |
SegMine | 86.91 | 89.90 | 85.33 | 92.27 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Qiao, Q.; Li, Y.; Lv, H. Open-Pit Mining Area Extraction Using Multispectral Remote Sensing Images: A Deep Learning Extraction Method Based on Transformer. Appl. Sci. 2024, 14, 6384. https://doi.org/10.3390/app14146384
Qiao Q, Li Y, Lv H. Open-Pit Mining Area Extraction Using Multispectral Remote Sensing Images: A Deep Learning Extraction Method Based on Transformer. Applied Sciences. 2024; 14(14):6384. https://doi.org/10.3390/app14146384
Chicago/Turabian StyleQiao, Qinghua, Yanyue Li, and Huaquan Lv. 2024. "Open-Pit Mining Area Extraction Using Multispectral Remote Sensing Images: A Deep Learning Extraction Method Based on Transformer" Applied Sciences 14, no. 14: 6384. https://doi.org/10.3390/app14146384