Set in the context of the development of bioinspired robotics systems, this paper seeks to understand the influence of the choice of the flapping orientation of fins on the propulsive performance of small underwater vehicles. In particular, the thunniform mode of Body and/or Caudal Fin (BCF) propelled systems is studied. This research is motivated by the fact that not much literature is available on the influence of flapping orientation of marine organisms and a number of mechanisms are found in nature. Dorso-ventral flapping with a positive metacentric height is shown to yield better self-stabilizing effects and lesser energy consumption compared to sideways flapping. Moreover, with dorso-ventral flapping, the choice of metacentric height could lead to the possibility of adjusting the body's rotational oscillation amplitudes to positively affect the downstream fluid interactions for the caudal fin. This is not possible with sideways flapping where the designer would be forced to change the flapping kinematics or the body shape in the sagittal plane, to adjust the body oscillation amplitudes. While the main body of results are obtained using simulations for underwater vehicle dynamics with coefficients of the REMUS underwater vehicle, stability analysis for a generalised case is also presented.