This paper presents the reliability analysis of a manual ultrasonic time of flight diffraction test system for characterization of surface breaking inclined cracks. Probability of detection (POD) and probability of sizing (POS) curves for the detection and sizing of such cracks have been developed from experimental data. In the development of POD and POS curves, it has been assumed and subsequently verified that the signal responses (a values for an angle of inclination of the crack 0) have a normal distribution. The effects of probe angle and probe center spacing on both POD and POS have been investigated. The curves obtained in this work can be useful in risk based test planning for simple geometry components, such as pipelines and boiler headers, using a manual/automated time of flight diffraction system.