We report the structural and magnetic properties of the 4d (M = Rh) based and 5d (M = Ir) based systems Ba3MxTi3-xO9 (nominally x=0.5, 1). The studied compositions were found to crystallize in a hexagonal structure with the centrosymmetric space group P63/mmc. The structures comprise of A2O9 polyhedra [with the A site (possibly) statistically occupied by M and Ti] in which pairs of transition metal ions are stacked along the crystallographic c axis. These pairs form triangular bilayers in the ab plane. The magnetic Rh and Ir ions occupy these bilayers, diluted by Ti ions even for x=1. These bilayers are separated by a triangular layer which is dominantly occupied by Ti ions. From magnetization measurements we infer strong antiferromagnetic couplings for all of the materials but the absence of any spin-freezing or spin-ordering down to 2 K. Further, specific heat measurements down to 0.35 K show no sign of a phase transition for any of the compounds. Based on these thermodynamic measurements we propose the emergence of a quantum spin liquid ground state for Ba3Rh0.5Ti2.5O9, and Ba3Ir0.5Ti2.5O9, in addition to the already reported Ba3IrTi2O9. ©2016 American Physical Society.