This paper proposes model-based compliant motion control scheme for assembly tasks using vision and force information. Many problems, especially assembly tasks, could be solved by integrating visual servoing and impedance control with model-based compliant motion design, in which the coordinate frame of the impedance control is attached to the model of assembled object. To implement the model-based compliant motion control scheme, real-time model-based object pose estimation is used for the visual servoing implemented by a position-based approach. The coordinate frame used for the impedance control, which is attached to the object model, is determined based on the estimated object pose. The proposed control is effective for assembly of two parts like connector insertion task, in which the compliant motion should be designed according to the geometric structure of the assembled part, such as a shape of a socket. Finally, experimental results illustrate the effectiveness of the proposed control.