We propose, to the best of our knowledge, a novel deep learning-enabled four-dimensional spectral imaging system composed of a reflective coded aperture snapshot spectral imaging system and a panchromatic camera. The system simultaneously captures a compressively coded hyperspectral measurement and a panchromatic measurement. The hyperspectral data cube is recovered by the U-net-3D network. The depth information of the scene is then acquired by estimating a disparity map between the hyperspectral data cube and the panchromatic measurement through stereo matching. This disparity map is used to align the hyperspectral data cube and the panchromatic measurement. A designed fusion network is used to improve the spatial reconstruction of the hyperspectral data cube by fusing aligned panchromatic measurements. The hardware prototype of the proposed system demonstrates high-speed four-dimensional spectral imaging that allows for simultaneously acquiring depth and spectral images with an 8 nm spectral resolution between 450 and 700 nm, 2.5 mm depth accuracy, and a 1.83 s reconstruction time.