Benchmarking is not only about making comparisons but, through these, learning lessons to improve actual performance or knowledge. Comparing modelling paradigms based only on the conceptual model specifications is not enough; rather a well-defined benchmarking process and the execution of experiments are required. A benchmarking strategy is applied to three models of an oil refinery supply chain, using the agent-based or equation-based paradigm. Despite the different paradigms, the models share the same assumptions and model boundaries and an attempt is made to provide same initial conditions and stochastics. The benchmarking process shows that clear definitions of the modelling paradigms are needed to avoid confusion and to enable mining of specific and guiding conclusions from the benchmarking studies. Agentbased models and equation-based models rely on different modelling attributes: The first are mostly identified by the model elements (i.e. individuals) while the latter are mostly identified by the system description elements (i.e. equations). We present a way to visualize this and to add nuance to the choice of labels to allow for the conclusions of the benchmarking study to be generalised beyond the models that are compared, to learn about the advantages and shortcomings of modelling paradigms. Finally, some misconceptions regarding agent-based modelling are identified. The lessons learnt apply to supply chain domain but are extensible to other domains.