Semi-active systems using magnetorheological fluids have been realized in many novel devices such as linear dampers, rotary dampers, brakes, and so on. Rotary vane-type magnetorheological damper is one such device that uses magnetorheological fluid as a hydraulic medium and a controllable magnetorheological valve to generate variable resistance. This device, due to its limited angle motion, lends itself to a natural application for prosthetic knee joint. In this article, a bypass rotary vane-type magnetorheological damper suitable for prosthetic knee device is designed. In the proposed design, the rotary vane chamber and the bypass magnetorheological valve are connected using hydraulic cables and ports. The design of rotary cylinder is implemented based on the largest possible dimensions within the envelope of a healthy human knee, while the magnetorheological valve is designed optimally using a multi-objective genetic algorithm optimization. Off-state braking torque, induced on-state braking torque and mass of the valve are selected as three objectives. The torque and angular velocity requirements of the normal human knee are used as design limits. The optimal solution is chosen from the obtained Pareto fronts by prioritizing the objective of weight reduction of magnetorheological valve. The optimal solution is capable of producing a damping torque of 73 Nm at a design speed of 8.4 rpm and current supply of 1.9 A. Potential benefits offered by this design when compared with multi-plate magnetorheological brake are flow mode operation, large clearance gap, and fewer design components, thus reducing the manufacturing complexity. © The Author(s) 2020.