The task of matching image patches is a fundamental problem in computer vision. When sufficiently textured patches are normalized up to similarity transformation, a simple Normalized Cross Correlation (NCC) of corresponding patches will give a high value. In practice, using it on patches per se may not perform well due to the noisy variations of pixel intensities. A more prudent approach will be to apply it to the abstract features extracted by a deep convolutional network. We study the applicability of an NCC based convolutional network for the task of Patch Matching. Further, there may be cases where the network may fail due to insufficient textures. In those cases, a simple pixel difference based method will be beneficial. To this end, we propose to improve the two basic architectures, Siamese networks and Central-Surround stream networks, using robust matching layers for learning the similarities of patches, assisted by a simple cross-entropy loss function. We empirically verify the performance of the proposed models on the challenging UBC Patches dataset and show that they are close to the state-of-the-art. Further, we evaluate their resilience to large illumination changes in two experimental scenarios: 1) by manually varying the patches of UBC Patches by an affine model 2) by using the publicly available Webcam dataset. We demonstrate that our models are indeed very resilient to illumination variations; they reduce the false positive rate to nearly 10%, and improve over the popular methods by nearly 5%. Further, we demonstrate the generalisability of the proposed NCC based matching layer by applying it to Face Recognition and show that it improves the performances of well known networks on a real-world, surveillance dataset. © 2018 IEEE.