We implement a binocular vision system based on a disparity-energy model that emulates the hierarchical multi-layered neural structure in the primary visual cortex. Layer 1 performs difference-of-Gaussian filtering that mimicks the center-surround receptive fields (RF) in the retina, layer 2 performs Gabor filtering mimicking the orientation selective filtering performed by simple cells and layer 3 has complex cells tuned to detecting 5 different disparities. A VLSI architecture is developed based on stochastic computing that is compact and adder-free. Even with a short stream length, the proposed architecture achieves better disparity detection than a floating-point version by using a modified disparity-energy model. A 1 × 100 pixel processing system is synthesized using TSMC 65nm CMOS technology and achieves up to 79% reduction in area-delay product compared to a fixed point implementation.