Time-frequency analysis is an effective method to extract features from vibration signals by acquiring time-frequency spectrum from time series. However, gearbox is usually operating in harsh working environment especially in variable rotor speed condition, thus fault component is buried in strong background noise and harmonic interference. Therefore, a time-frequency domain feature enhanced sparse matrix and singular value vector optimization method is proposed to detect and extract gearbox fault features more accurately. A novel time-frequency transform method is implemented to concentrate the energy of gearbox fault character. The minimax concave penalized sparse optimization is implemented to emphasize the sparsity of time frequency domain and the model is derived by proximal operator. Then, the sparse matrix and singular value vector optimization model is built to extract the feature of gearbox fault. The simulated signal and experimental signal both validate the effectiveness of the proposed method.