Numerical simulation of nonlinear waves to reproduce the laboratory measurements has been a topic of great interest in the recent past. The results reported in the literature are mainly focused on qualitative comparison or on the relative errors between the numerical simulation and measurements in laboratory and hence lack in revealing the existence of phase shift in nonlinear wave simulation. In this paper, the simulation of nonlinear waves in mixed Eulerian and Lagrangian framework using finite element method (FEM) is investigated by applying two different velocity calculation methods viz, cubic spline and least squares (LS). The simulated wave surface elevation has been compared with the experimental measurements. The coherence analysis has been carried out using the wavelet transformation, which gives a better understanding between the numerical and the experimental results with respect to the time-frequency space, compared with the conventional Fourier transformation. It is observed that the application of cubic spline approach leads to a higher phase difference for steeper waves. The present study has shown that the phase difference exists at the higher modes rather than at the primary period. For waves with steepness (wave height/wave length) higher than 0.04, LS approach is found to be effective in capturing the higher-order frequency components in the event of nonlinearity. In addition, the comparison of numerical simulations with that from PIV measurements for the tests with solitary waves is also reported. Copyright © 2009 John Wiley & Sons, Ltd.