The aim of this paper is to experimentally characterize the liquid jet breakup unsteadiness in a coaxial air-blast atomizer. The current research focuses on the measurement of the fluctuations of the jet breakup length and the flapping instability of the liquid jet, which contribute to the downstream fluctuations of the spray characteristics. The optical connectivity technique was used to measure the instantaneous breakup length of the water jet. Also, time resolved shadowgraph images of the primary jet breakup process were captured by high-speed imaging to characterize the jet instabilities at different axial locations from the atomizer exit. Experiments were performed for a wide range of air-to-liquid momentum flux ratio (M) and aerodynamic Weber number (Weg) corresponding to membrane- and/or fiber breakup mode of the jet disintegration process. The mean jet breakup length was found to vary inversely with M through a power law relation in agreement with the literature, while the breakup length fluctuations were found to first decrease and then increase with M. In order to capture the unsteady dynamics of the jet breakup process, the proper orthogonal decomposition analysis of the optical connectivity images was performed. The jet flapping and the fluctuations of the jet breakup length were identified as the second and the third spatial proper orthogonal decomposition modes, respectively, for all operating conditions of the atomizer. The amplitude and the frequency of the instabilities were measured by temporal tracking of the liquid–air interface on the shadowgraph images. The disturbance close to the injector exit corresponds to the Kelvin–Helmholtz instability, while close to the jet breakup point the jet exhibits the flapping instability, which is characterized by lateral oscillation of the jet about the atomizer axis. The influence of the liquid jet Reynolds number and momentum flux ratio on the KH and the flapping instabilities are examined. © The Author(s) 2018.