In order to realize the autonomous scooping of the crushed rocks by the wheel loader, first of all, the loader has to recognize the shape of the rock pile and to plan the scooping task autonomously. Generally, scooping task is carried out by penetrating the bucket into the rock pile and lifting up the bucket to scoop the rocks. Therefore, in order to plan the scooping task, the scooping point and penetration direction of the bucket have to be determined. In this study, the algorithm to determine the scooping point and penetration direction of the bucket was proposed based on the shape of the rock pile recognized by image processing, and scooping experiments were carried out by using the wheel loader model with a stereo vision system. First of all, the wheel loader recognized the rock pile and determined the scooping point and penetration direction of the bucket. Then, it moved to the scooping point and penetrated the bucket into the rock pile, and scooped the rocks. The working efficiency by using the algorithm proposed here was compared with the one by teaching playback. It was confirmed that the working efficiency by using the algorithm proposed in this study is much higher than that by the teaching playback.