In this paper, we propose a vision based compliant motion control method for part assembly work. Some industrial parts are deformed during assembly of parts. If work progress continues, deformation of the part increases, humans check the deformation and adjust force corresponding to the progress state. The proposed method enables a robot manipulator to adjust force applied for assembly work like a human. In our proposed method, force applied for the work is generated by visual information from a camera reading the deformation of the parts. Processing the visual information quantifies the deformation and the data show the work progress. NCC is generally used for template matching, but in this paper we use it for quantifying deformation. Connectors are assembled by a robot manipulator using proposed method and impedance control in experiments. Experimental results are presented to verify the effectiveness of the proposed method.