In radiation therapy, respiration-induced tumor motion significantly limits the efficiency of the radiation delivery, and brings potential risk to healthy organs and tissues. In order to deliver a sufficient high-dose radiation in adaptive with the tumor motion, a kilo-voltage (kV) X-ray fluoroscopy imaging system has been used to monitor the tumor motion in real-time during the treatment. In this paper, we present a fast and robust tracking algorithm to track deformable lung tumor motion in the kV fluoroscopic image sequence. Given a kV fluoroscopy, the tumor motion is represented by a nonlinear homographic transformation of a pre-defined tumor template. The homographic transformation is then estimated by minimizing a sum-of-squared-difference (SSD) between the template image and the observed image. To improve the computational efficiency, an efficient second-order minimization method is employed to solve the problem of SSD minimization. The experimental results conducted on clinical kV fluoroscopies demonstrated that the proposed method is capable of tracking the tumor motion in real-time and its performance is superior to conventional tracking methods in terms of tracking accuracy and computational cost.