In this paper, a three-dimensional (3D) environment reconstruction framework based on graph optimization is proposed that uses acoustic images captured in an underwater environment. Underwater tasks such as unmanned construction using robots are becoming more and more important. In recent years, acoustic cameras which are forward-looking imaging sonars are being commonly used in underwater inspection. However, the loss of elevation angle information makes it difficult to get a better understanding of underwater environments. To cope with this, we apply 3D occupancy mapping method based on the acoustic camera rotating around the acoustic axis to generate 3D local maps. Next, from the local maps and a graph optimization scheme, we minimize the error of camera poses and build a global map. Experimental results demonstrate that our 3D mapping framework for the acoustic camera can reconstruct dense 3D models of underwater targets robustly and precisely.