Cardiac arrhythmias are presently diagnosed by manual interpretation of Electrocardiography (ECG) signals. Automated ECG interpretation is required to perform efficient screening of arrhythmia from long term ECG data. Existing automated ECG interpretation tools however require extensive preprocessing and knowledge to determine relevant features. Thus there is a need for a comprehensive feature extractor and classifier to analyze ECG signals. In this paper, we propose three robust deep neural network (DNN) architectures to perform feature extraction and classification of a given two second ECG signal. The first network is a Convolutional Neural Network (CNN) with multiple kernel sizes, the second network is a Long Short Term Memory (LSTM) network and the third network is a combination of CNN and LSTM based feature extractor, CLSTM network. The proposed networks are end to end networks which can be directly trained without any preprocessing. The networks were trained and tested with the MITDB ECG dataset on three classes Normal (N), Premature Ventricular Contraction (PVC) and Premature Atrial Contraction (PAC). The best model CLSTM gave an accuracy of 97.6%. Further, transfer learning is showcased on the best performing network for use with multiple ECG datasets requiring training only on the final three layers. The results showcase the potential of the network as feature extractor for ECG datasets. Our results outperform the state-of-the art works on ECG classification on several metrics. © 2018 IEEE.