Semi-active technology offers good advantages in terms of controllability and adaptability. Magneto-rheological (MR) fluid is a class of smart fluids which display significant changes in its rheological properties under the influence of a magnetic field. Previous studies carried out using MR brake for the transfemoral prosthetic device were of multi-plate models which are complex in design and also to manufacture. Therefore, in the present study, a multi-coil rotary inverted drum brake is optimized with braking torque as the objective function. One of the advantages of the multi-coil design from multi-plate is that the former has fewer components and leads to a simpler design. The outermost geometric constraints are decided based on the knee cross-sections in anterior-posterior and mediolateral directions. Four design geometric variables are selected which are: coil depth, coil height, casing axial thickness, and casing radial thickness. A design of experiments technique is used to obtain 27 combinations of design variables. Magnetostatic analysis at each design point is performed and average flux densities in the annular and the radial gaps are determined. Regression analysis is conducted on the design data to obtain braking torque as a function of four design variables. Later, genetic algorithm is used to obtain the optimum geometric dimensions. A total maximum braking torque of 13.4 Nm is obtained using the optimum dimensions for a design current of 2 A. © 2021, The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.