1. Introduction
Remote sensing data from different sensors vary in temporal and spatial resolutions. These sensors images have been widely used in many fields, such as Sentinel-2, which plays an important role in many applications [
1], including monitoring vegetation [
2,
3], urban extraction [
4], and the construction of hydrological models [
5], while GaoFen-2 focuses on fine land cover classification and feature extraction [
6]. The sensor design must balance capturing spatial details and addressing repeated coverage concerns [
7]. It is particularly challenging to obtain high-quality images in areas with cloudy or rainy conditions. The integration of two sensors offers vital advantages in constructing dense image time series with a high spatial resolution, essential for the long-term and detailed monitoring of surface processes. Therefore, the development of super-resolution reconstruction (SR) methods to improve the resolution of Sentinel-2 images to generate synthetic high-resolution time series datasets has attracted extensive attention.
The traditional image SR algorithm is mainly based on interpolation or traditional machine learning [
8,
9]. Due to the reliance of Bicubic methods on local neighborhood information, unsatisfactory results have been achieved [
8,
10]. The emergence of deep learning has shown promising results in SR. Convolution neural networks (CNNs) have made significant progress in image SR [
11,
12,
13,
14,
15,
16,
17]. SRCNN produces clearer reconstructed images and significantly improves the reconstruction speed [
18]. ESPCN accelerates feature learning and reduces network complexity by directly passing feature information in the LR space [
19]. However, with the increasing demand for reconstruction results, the depth of the network is also increasing, leading to the problem of vanishing gradients. The residual structure has gradually played a role in SR tasks. It allows CNNs to model more deeply without suffering from vanishing/exploding gradients [
11]. Subsequently, some heavyweight models have been proposed [
13,
16,
20,
21,
22]. However, due to excessive focus on minimizing the mean square reconstruction error, the reconstruction results are “overly smooth” [
23,
24,
25].
Generative adversarial networks (GANs) have been introduced to image SR by leveraging adversarial learning between the generator and discriminator to generate high-quality results. For example, SRGAN demonstrates the ability of GANs in inverse imaging problems, which combines multiple loss functions to ensure that the generated results have both good perceptual and objective quality [
26]. ESRGAN improves reconstruction by addressing noise susceptibility and enhancing residual blocks with residual dense blocks [
27]. In the field of remote sensing, inspired by the natural image reconstruction method, research based on the characteristics of remote sensing images has also been carried out. Meng et al. had addressed the challenges of large-scale remote sensing images by combining dense sampling and residual learning methods to enhance the effectiveness of feature extraction [
28]. Despite the benefits of convolutions in modeling image dependencies, their ability to learn long-term dependencies is limited. To address this problem, the self-attention mechanism is introduced to carefully coordinate the details of each position of the image with the details of the distant part of the image [
29]. Combining the self-attention mechanism with GANs allows for the extraction of global features without introducing excessive parameters [
30].
In recent years, due to its advantage of a self-attention mechanism, the transformer model has become an essential choice in the field of natural language processing. However, the complexity of the transformer model has posed a significant challenge to its performance. To address this issue, an Efficient Super-Resolution Transformer (ESRT) is proposed. The ESRT aims to enhance the ability of SR networks to capture long-range contextual dependencies while significantly reducing the GPU memory cost [
31]. However, due to the rich land coverage and complex land use scenes contained in remote sensing images within a limited pixel size, the loss of high-frequency detail information is more severe. It is difficult to effectively reconstruct the lost high-frequency information by focusing only on spatial-domain information. Restoring high-frequency information through power spectrum features in the frequency domain appears simpler and more effective. Wavelet transform combines temporal and frequency resolutions, adapting window sizes and facilitating detail feature extraction. It can process high-frequency information such as edges and details while reducing computational memory [
32,
33,
34]. In DWSR, residual networks and wavelet transforms exploit the residual sparsity of wavelet coefficients. Different wavelet subbands, providing complementary structural information in different directions, predict the HR image [
35]. WRAN [
36] uses a multi-kernel convolutional to adaptively aggregate features of the receptive fields from different-sized receptive fields of wavelet coefficients. Nevertheless, the multi-band image information of different sensors is complex and degraded and there are few studies on the frequency-domain multi-scale reconstruction of image details. Combining wavelet transform and the residual structure of GANs ensures network gradients and extracts deeper features [
37]. In addition, loss constraint in the wavelet domain retains the advantages of wavelet transform in high-frequency information reconstruction and avoids excessive smoothing of reconstructed images [
38].
However, most of the existing algorithms extract the spatial- and frequency-domain features using a single network. Deepening or widening the network improves algorithm performance, it also increases computational complexity, and at the same time, it is difficult for these methods to effectively couple spatial and frequency information. Therefore, there is a need to develop a new network model that can maintain the advantages of extracting spatial information while enhancing the ability to extract frequency-domain information.
So far, although many SR methods based on remote sensing have been developed, it has been challenging to acquire paired high-resolution and low-resolution images for training, leading to a lack of corresponding “high-resolution and low-resolution” real remote sensing image pairs. Additionally, most models are trained on natural image datasets, like DIV2K, UC_Merced [
39], and WHU-RS19 [
40], which results in significant performance degradation when applied to real remote sensing image SR tasks [
41]. To better address the degradation information between HR and LR remote sensing images, this study focuses on developing a benchmark paired remote sensing image dataset considering the following four critical features: (1) different sensors, (2) diversity of regions, (3) consistency of time, and (4) challenging scenarios. Based on the four aforementioned aspects, we propose a new benchmark dataset GF_Sen, formed by images obtained from GaoFen-2 and Sentinel-2.
The main objectives of this study are as follows: (1) To propose a cascaded spatial frequency-domain generative adversarial network CSWGAN, specifically for cross-sensor remote sensing images reconstruction. We first proposed the SGAN and WGAN, which integrate the advantages of a self-attention mechanism and wavelet transform, enriching the global feature information and high-frequency details in the reconstructed image. The CSWGAN combines the advantages of two networks through a cascaded structure, making its reconstruction results closer to HR images; (2) to propose a new benchmark paired remote sensing dataset GF_Sen for SR tasks, to evaluate the performance of several widely used deep learning models (SRCNN, ESPCN, SRGAN, ESRGAN, etc.). This dataset aims to provide a fair and comprehensive comparison between different models.
2. Materials and Methods
In this section, we will first give a brief introduction to the two main methods used in the CSWGAN. Then, we will introduce the proposed method including the design of the network architectures and loss functions.
2.1. Self-Attention Mechanism
The self-attention module through 1 × 1 convolutions calculates the correlation matrix with the C channel feature map, which represents the spatial correlation between any two positions in the input characteristic graph, as shown in
Figure 1. Each location is calculated and updated by the weighted sum of all other locations. Among them, the weight value is determined by the learning dependency between the two positions. The formula for calculating the extent of different locations in each area of the reconstructed image is as follows:
where
indicates the extent to which the model attends to the ith location when synthesizing the jth region. N is the number of feature locations of features. The output of the attention layer o is defined as
In the above formulation,
and
are the learned weight matrices, which are implemented as 1 × 1 convolutions. Among them,
. In addition, we further multiply the output of the attention layer by a scale parameter and add back the input feature map. Therefore, the final output is given by
where γ is a learnable scalar and it is initialized as 0, which allows the network to first rely on the information in the local neighborhood, and then gradually learn to assign more weight to the non-local evidence.
2.2. Wavelet Prediction and Reconstruction
In this paper, the wavelet transform method is used to decompose the characteristic image obtained by the residual module into a sequence of wavelet coefficients of the same size. The wavelet prediction net can be split into Nw = 4n parallel independent subnets, where n is the decomposition level. The corresponding wavelet coefficients are obtained from the input feature extraction block output of each subnet. It is assumed that the characteristic size of wavelet coefficients is the same as that of the LR input.
The reconstruction net is used to transform the wavelet images of the total size N
w × 3 × h × w into the original image space of the size 3 × (r × h) × (r × w). It is represented by r × r size filter, composed of a deconvolution layer with a step size of r, and Ne is the channel size of the last layer of the feature extraction block. Let
and
denote the ground-truth and inferred wavelet coefficients, respectively. The overall process can be defined as
where
2.3. A Cascaded Spatial Frequency-Domain Generative Adversarial Network for SR: CSWGAN
This paper proposed the CSWGAN, which was a cascade network of a self-attention-based GAN (SGAN) and wavelet-based GAN (WGAN), as shown in
Figure 2.
The main idea is to design a cascade framework to implement the application of spatial and frequency-domain features to obtain high-quality SR images.
The main architecture is based on SRGAN [
26], with residual blocks serving as an important component in the network construction, enabling the recovery of photo-realistic textures on public benchmarks. The activation functions used in the generator and discriminator are different, with the generator employing PReLU to allow negative values to pass through the activation function. This helps to address the vanishing gradient problem and enhances the overall performance of the network by enabling the model to learn more complex data relationships. In the discriminator, LeakyReLU activation is used to prevent the need for max-pooling throughout the network.
In order to enhance the learning of the spectral and geometric position information in remote sensing images, a self-attention branch is added outside the residual block of the backbone network. By incorporating self-attention mechanisms into feature extraction methods, dense global contextual information can be obtained, thereby establishing interdependencies between pixels. This enhances the network’s utilization of global information and deep features. Then, the SR task is transformed into a wavelet coefficient prediction task, where the wavelet prediction module is tightly integrated with the wavelet reconstruction module within the GAN. By accurately predicting the corresponding wavelet coefficients, the LR image is decomposed into different scales and orientations, resulting in a high-quality HR image with rich texture details and global feature information.
The main task of the SGAN network is to learn and obtain the geometric structure information of the image, as well as low-frequency information such as the overall appearance and contours of the image. On the other hand, the WGAN is designed according to the characteristics of the frequency domain. Its main task is to learn high-frequency information such as image texture and edge. The SGAN learns spatial low-frequency features and their correlations through a self-attention mechanism. Based on the output characteristics of the SGAN, the WGAN can better distinguish noise and detail information, further optimize low-frequency and enhance high-frequency learning, thus improving the quality and fidelity. Therefore, by using the spatial-domain super-resolution (SDSR) output of the SGAN as the input to the WGAN and generating the final reconstruction output, the CSWGAN network can fully leverage the advantages of these two different networks. Moreover, training this cascade structure using multiple generators and discriminators is very time-consuming, so each adversarial system is trained separately first and then jointly in an end-to-end manner.
2.4. Loss Function
Since the CSWGAN is logically composed of two independent networks, there is no overall loss function. The loss functions of the SGAN and WGAN work in their respective networks.
2.4.1. Loss Function of SGAN
In this paper,
and
are used as the LR image and HR image, respectively. We constructe a robust loss function to force generator G to generate an HR image similar to
,
where
and
represent the coefficients of L
adv (adversarial loss) and L
tv (regularization loss), with values of 1 × 10
−3 and 2 × 10
−8. The network requires the reconstruction effect at the pixel level through
, trains the discriminator D to minimize the adversarial loss, and encourages G to generate a reconstructed image close to the
. It is well known that adversarial learning strategies can maintain the visual authenticity of the generated images; however, it is also vulnerable to noise pollution. In order to restrain the noise and keep the image smooth, TV regularization loss is introduced in this paper.
where N denotes the number of images,
denotes a set of model parameters in G,
denotes a set of model parameters in D,
denotes the reconstructed image,
denotes the probability that the
is a real HR image, and
represents the gradient of the output of G.
2.4.2. Loss Function of WGAN
Different to the spatial-domain-based network that directly learns the mapping relationship between HR and LR pixel values, the WGAN aims to learn the relationship between the wavelet coefficients of frequency-domain characteristics between HR and LR images. The WGAN constrains the network through two losses in the spatial and frequency domain of the image.
where
is the balance parameter, which is 0.1. The WGAN takes the MSE loss function
as the loss constraint of the full image to make the reconstruction result achieve a balance in smoothness and texture details; it can hardly obtain high-frequency texture details. Therefore, in order to pay attention to local texture information and prevent high-frequency wavelet coefficients from converging to 0, the WGAN adopts
and
avoids the above problems, respectively.
where
is the balance parameter,
and
are the ground-truth wavelet coefficients and the inferred wavelet coefficient, respectively. k indicates the start index of the wavelet coefficients to be penalized for taking small values,
represents the balance weight,
and
are slack values. More attention can be paid to the local texture with large weight assigned to the high-frequency coefficient. At the same time, it can capture the global topology information.
2.5. Quantitative Evaluation Indices
The evaluation of super-resolution results mainly includes subjective evaluation and objective evaluation. We used various image quality evaluation indicators, including peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) [
42], standard deviation (SD), information entropy (IE) [
43], root mean square error (RMSE), and spectral angle (SAM) [
42], as shown in
Table 1. These indicators not only measure the fidelity and differences between real HR images and SR images on the ground, but they can also better explain the amount of information contained in the image and evaluate many subtle differences from the perspective of spectrum. SD and IE are non-reference image quality evaluators. Additionally, three experts with three years of experience in this field compared the naturalness of SR images.
6. Conclusions
In this paper, we introduced a new real high- and low-resolution dataset (GF_Sen) and proposed a cascade structure network CSWGAN for super-resolution.
GF_Sen is a benchmark paired dataset of real remote sensing images, which is composed of Sentinel-2 and GaoFen-2 images. The experiments demonstrate that networks trained on this dataset can have a better performance than those trained on a simulated dataset.
In addition, a GAN-based remote sensing image super-resolution method CSWGAN is proposed in this paper, which adopts a cascaded generative adversarial network and is composed of single-structure networks proposed by the SGAN and WGAN. The CSWGAN combines the advantages of the SGAN and WGAN and has strengthened its capability to model long-range dependencies between different frequency domains and regions, as well as acquiring global topological information and local texture details. This solves the challenge of using a single network model to simultaneously target real low-resolution images and obtain better reconstruction results. Finally, comparing with different networks and different scene types and scales of scenes, it has been proven that the CSWGAN has a more obvious advantage in reconstructing scenes with more detailed textures, and achieves the best quantitative and qualitative results on relevant datasets.