Chemical processes rely on several layers of protection to prevent accidents. One of the most important layers of protection is human operators. Human errors are a key contributor in a majority of accidents today. Estimation of human failure probabilities is a challenge due to the numerous drivers of human error, and still heavily dependent on expert judgment. In this paper, we propose a strategy to estimate the reliability of control room operators by measuring their control performance on a process simulator. The performance of the operator is translated to two metrics – margin-of-failure and available-time to respond to process events – which can be calculated using process operations data that can be generated from training simulator based studies. These metrics offer a qualitative estimate of operators’ reliability. We conducted a set of experiments involving 128 students of differing capabilities from two different institutions and tasked to control a simulated ethanol production plant. Our results demonstrate that differences in the performance of expert vs. novice student operators can be clearly distinguished using the metrics. © 2017 Elsevier Ltd