In this paper we investigate the nonlinear behavior of a coupled reactor separator system. The focus is on considering the effect of energy as well as the mass coupling between the reactor and the separator. The reactor holdup is controlled using the reactor effluent stream and the flash is operated under a constant heat duty. The fresh feed flow rate is flow controlled. We compare the behavior of the system when there is only mass coupling to the case when there is mass and energy coupling. It is shown that the presence of energy coupling can render a system unstable. We also investigate the effect of a nonideal vapor liquid equilibrium (VLE) in the form of a minimum boiling azeotrope and a maximum boiling azeotrope. It is shown that, when the VLE exhibits a minimum boiling azeotrope, the system is more stable when the liquid stream is recycled from the flash to the reactor. © 2006 Curtin University of Technology and John Wiley & Sons, Ltd.