Constrained groove pressing is one of the severe plastic deformation techniques to produce ultra-fine grained material. The process defects such as unevenness and cracks on sheet surface limit its commercial applications by reducing the pass numbers thereby the degree of grain refinement. The objective of the present study is to optimize the die design to increase the pass numbers on the basis of die geometry, material properties and sheet thickness by avoiding the local failure. Two response variables namely, maximum equivalent strain and strain inhomogeneity factor, were used to define the objective function. The principal component analysis followed by the gray relational analysis was deployed to develop a single objective function in terms of gray relational grade. A hybrid approach of genetic algorithm followed by gradient-based optimization was adopted to identify the optimal parameters. AA5083 sheets could be successfully deformed with the improved die design while in case of low carbon steel, the number of passes increased from five to seven. The maximum equivalent strain imparted in each material also increased. It has been observed that the yield strength of AA5083 sheet improved by 85% after the first pass with the modified die. © 2020, © 2020 Taylor & Francis.