The main difficulty of construction automation is the variety of materials. To cope with this problem, we propose a concept of automated assembly system for a modular structure. This system uses passive building blocks called "structure modules" and an assembler robot that is specialized to handle them. This "modular" concept drastically simplifies structural construction and allows automation. This paper describes the hardware design of the structure module and the assembler robots. Especially, die connection mechanism, the essential element of the system, is explained in detail We built a prototype model based on the design to evaluate its automatic construction capability. Some experimental results illustrate the feasibility of the proposed method.