A low-sensitivity design and implementation of allpass structures based fractional delay (FD) filters is proposed in this paper. It is based on a phase-sensitivity minimization of each individual first- and second-order allpass section in the filter cascade realization depending on its transfer function (TF) poles position. It is shown that the poles of the FD TFs are taking positions not typical for the conventional filters and there is no second-order allpass sections providing low sensitivity for these positions. A new second order allpass section, suitable for such design, is proposed and compared with the other known second-order sections. The design procedure is verified experimentally. The overall sensitivity of the properly designed FD filter is considerably decreased, permitting thus a severe multiplier coefficient quantization and efficient multiplierless implementation.