Advanced LIGO's recent observations of gravitational waves (GWs) from merging binary black holes have opened up a unique laboratory to test general relativity (GR) in the highly relativistic regime. One of the tests used to establish the consistency of the first LIGO event with a binary black hole merger predicted by GR was the inspiral-merger-ringdown consistency test. This involves inferring the mass and spin of the remnant black hole from the inspiral (low-frequency) part of the observed signal and checking for the consistency of the inferred parameters with the same estimated from the post-inspiral (high-frequency) part of the signal. Based on the observed rate of binary black hole mergers, we expect the advanced GW observatories to observe hundreds of binary black hole mergers every year when operating at their design sensitivities, most of them with modest signal to noise ratios (SNRs). Anticipating such observations, this paper shows how constraints from a large number of events with modest SNRs can be combined to produce strong constraints on deviations from GR. Using kludge modified GR waveforms, we demonstrate how this test could identify certain types of deviations from GR if such deviations are present in the signal waveforms. We also study the robustness of this test against reasonable variations of a variety of different analysis parameters. © 2017 IOP Publishing Ltd.