We propose a multi-layer space-time block coded orthogonal frequency division multiplexing (Multi-Layer STBC OFDM) scheme. It combines the spatial diversity and spatial multiplexing in order to exploit the benefits of both techniques. The transmit antennas are divided into several layers and each layer is encoded by STBC. The antennas transmit OFDM signals in order to deal with frequency-selective fading channels. In addition to Exhaustive Detection Scheme which applies the same detection at each subcarrier independently, we exploit the subcarrier correlation to develop two subcarrier-grouping based detection schemes. The subcarriers bound into the same group share the same detection parameters, e.g., layer detection order or nulling matrices, which are obtained at the center subcarrier of that group. The proposed schemes can significantly reduce the complexity with very small performance loss. The simulation results show that compared with Exhaustive Detection Scheme, the propose detection scheme is capable of reducing 51.79% complexity with only 0.5dB performance loss for 4×4 system, and reducing 69.96% complexity with only 1.0dB performance loss for 6×6 system.