We propose a system that allows multiple groups of users at remote locations to naturally share a 3D image of real objects. All users can interactively observe a 3D stereoscopic image without distortion from their own viewpoints. The system basically consists of a combination of subsystems: imaging and display. The imaging subsystem generates images of the real object observed from arbitrary viewpoints based on image-based rendering technique implemented on GPU. The display system generates a 3D virtual image of the real object to be interactively observed by multiple people around the tabletop display. People at one place just put the real object on their imaging system to capture a set of its images from sparse viewpoints around it; other groups of multiple persons at remote places connected by networks observe its virtual image from arbitrary viewpoints, as if there is a copy of the real object, on their display systems. This paper describes details of the system configurations and algorithms. Then discussions are made based on experimental results.