The analysis of transport protocols, along with queue management policies, forms an important aspect of performance evaluation for the Internet. In this article, we study Compound TCP (C-TCP), the default TCP in the Windows operating system, along with the Exponential-RED (E-RED) queue policy and the widely used Drop-Tail queue policy. We consider queuing delay, link utilization and the stability of the queue size as the performance metrics. We first analyse the stability properties of a nonlinear model for C-TCP coupled with the E-RED queue policy. We observe that this system, in its current form, may be difficult to stabilize as the feedback delay gets large. Further, using an exogenous and non-dimensional parameter, we show that the system loses local stability via a Hopf bifurcation, which gives rise to limit cycles. Employing Poincaré normal forms and the center manifold theory, we outline an analytical framework to characterize the type of the Hopf bifurcation and to determine the orbital stability of the emerging limit cycles. Numerical examples, stability charts and bifurcation diagrams complement our analysis. We also conduct packet-level simulations, with E-RED and Drop-Tail, in small and large buffer-sizing regimes. With large buffers, E-RED can achieve small queue sizes compared with Drop-Tail. However, it is difficult to maintain the stability of the E-RED policy as the feedback delay gets large. On the other hand, with small buffers, E-RED offers no clear advantage over the simple Drop-Tail queue policy. Our work can offer insights for the design of queue policies that can ensure low latency and stability. © 2017 The authors. Published by Oxford University Press on behalf of the Institute of Mathematics and its Applications. All rights reserved.