In this paper, we introduce a passive mobile robot with casters developed based on the concept of passive robotics. This mobile robot consists of two casters with servo brakes, one passive rigid wheel and a controller. We can manipulate the robot by controlling external force/moment applied by a human based on the control of the servo brakes attached to the casters. We consider the characteristics of the servo brakes and control the brake torque of each caster based on the braking force and moment constraint so that several motion functions of the robot are realized based on the applied force. This allows the robot to track a path without using servo motors. The motion control based on the environment information is also realized, so that we could avoid the collision with obstacles. These functions are implemented to the robot experimentally and experimental results illustrate the validity of the robot system and its control method.