In radiation therapy, inter-and intra-fractional tumor motions significantly limit the efficiency of radiation delivery, and bring potential risk to healthy organs and tissues adjacent to the tumor during treatment. To deliver a high-dose radiation adapting to the intra-fractional tumor motion, real-time imaging and tracking the tumor motion during the treatment became critical tasks in radiation therapy research. In this paper, we present our recent studies on tracking the respiration-induced lung tumor motion based on kilo-voltage (kV) and mega-voltage (MV) X-ray fluoroscopy systems. Especially, our focus is on developing robust tracking algorithms to track the deformable tumor motion in real-time. For the kV fluoroscopy, a kernel-based method, which is robust against the tumor deformation and low-contrast image quality, is proposed to track the tumor position. In addition, for the MV fluoroscopy, a region-scalable level set method (LSM) is proposed to tracking the tumor boundary. Experimental results conducted on phantom and clinical data demonstrated the effectiveness of our proposed methods.