We develop a new prediction method of respiratory motion for accurate dynamic radiotherapy, called tumor following radiotherapy. The method is based on a time-variant seasonal autoregressive (TVSAR) model and extended to further capture time-variant and complex nature of various respiratory patterns. The extended TVSAR can represent not only the conventional quasi-periodical nature, but also the residual components, which cannot be expressed by the quasi-periodical model. Then, the residuals are adaptively predicted by using another autoregressive model. The proposed method was tested on 105 clinical data sets of tumor motion. The average errors were 1.28 ± 0.87 mm and 1.75 ± 1.13 mm for 0.5 s and 1.0 s ahead prediction, respectively. The results demonstrate that the proposed method can outperform the state-of-the-art prediction methods.