104 / 2023-05-30 09:48:06

Multi-level hyperspectral image (HSI) and multispectral image (MSI) fusion

hyperspectral image (HSI) and multispectral image (MSI) fusion

Hyperspectral imaging is a new means of observations and it captures multiple narrow-band spectral images from the same scene. With the increase in the spectral bands, the energy captured by each spectral band decreases, resulting in a reduction in the spatial resolution of the images from these spectral bands. Low spatial resolution of hyperspectral images (HSIs) makes it difficult to meet the needs of applications, including the accuracy of classification, detection of changes and detection of objects. To obtain a high-spatial-resolution hyperspectral image (HR-HSI), hyperspectral image (HSI) and multispectral image (MSI) fusion fuses both a low-spatial-resolution hyperspectral image (LR-HSI) and a high-spatial-resolution multispectral image (HR-MSI).

The key to HSI and MSI fusion is to take advantage of the properties of spectral information of HSIs and spatial details of MSIs. At present, hyperspectral remote sensing images contain rich radiation, spatial, and spectral information, and are a comprehensive carrier of multiple types of information. Considering the complex spatial and spectral information of remote sensing images, Multi-level HSI and MSI fusion was proposed. The details of the method are as follows.

First, in order to extract multi-level spatial and spectral features from MSIs with rich spatial information and HSIs with rich spectral information, spatial and spectral feature extraction models were designed to learn multi-level spectral and spatial features through spectral and spatial multi-head attention, respectively. The spectral and spatial multi-head attention showed advantages in capturing long-range dependencies and self-similarity prior.

Second, multi-hierarchical encoders with spatial and spectral guidance extracted multi-hierarchical fused features from LR-HSI and HR-MSI under the guidance of the multi-level spatial and spectral features, respectively. Significantly, the multi-level spectral and spatial features were used to supplement spectral and spatial information and guide encoding in the multi-hierarchical encoders.

Finally, a spatial-spectral decoder was designed to construction a required HR-HSI by multi-hierarchical fused features, and a spatial-spectral loss function was designed to maintain the consistency of spatial-spectral structure. The spatial and spectral details were fully preserved through the spatial-spectral loss function.

Experiments were conducted to demonstrate the performance of the proposed multi-level HSI and MSI fusion. First, the ablation test was conducted. Second, the fused images obtained using the different methods for synthetic datasets were evaluated using both visual and quantitative measures. Finally, the fused images obtained using the different methods for the real datasets were evaluated. Experiment results on synthetic and real datasets showed that the proposed method outperformed common existing methods by means of both visual and quantitative evaluations. It is concluded that the proposed multi-level HSI and MSI fusion can lead to performance improvement of the image fusion.