We develop a markerless tumor motion tracking technique for accurate and safer image-guided radiation therapy (IGRT). The technique is implemented based on a new image model of the moving tumor and the background structure in an x-ray fluorocopic image sequence. By using the technique, the moving tumor image can be extracted from the sequential fluoroscopic images. The extraction from the fluoroscopy is obviously ill- posed, but we have suggested that it can be regularized into a well-posed problem by temporally accumulating constraints that must be satisfied by the extracted tumor image and the background. In this paper, the effect of the tumor extraction and motion of both the tumor and background in the image model is extensively studied on the tracking accuracy. The tracking accuracy of the proposed method with extraction of both the moving tumor and background was within $0.2$ mm of the spatial resolution for a phantom dataset. The accuracy within 1 mm can be clinically sufficient and is superior to the results by the previous method with extraction model of only the moving tumor and by a conventional method without extraction. Thus, the results clearly demonstrate the efficiency and usefulness of the proposed extraction model for the IGRT.