This paper proposes fast map building algorithm for multiple modal exteroceptive(external) sensors based on trajectory optimization. To realize localization using multiple modal exteroceptive sensors needs consistent environment maps (landmark maps). Multiple sensor SLAM(Simultaneous Localization And Mapping) is one of solutions to build maps consistently. In implementation of SLAM, approximations of sensor model contribute to get speeded up the algorithm. However in several case, these approximations deteriorate quality of the map. This paper proposes fast and accurate map building algorithm using multiple modal sensors based on sparse linear equations solver. This algorithm needs less approximations-but computation speed is fast. In addition, application of measured data from multiple sensors to this algorithm is described.