Chest radiography is the most preferred non-invasive imaging technique for early diagnosis of Tuberculosis (TB). However, lack of radiological expertise in TB detection leads to indiscriminate chest radiograph (CXR) screening. A modest classification approach based on the local image description to detect subtle characteristics of TB using CXRs is highly recommended. In this work, an attempt has been made to classify normal and TB CXR images using Bag of Features (BoF) approach with Speeded-Up Robust Feature (SURF) descriptor. The images are obtained from a public database. Lung fields segmentation is performed using Distance Regularized Level Set (DRLS) formulation. The results of segmentation are validated against the ground truth images using similarity, overlap and area correlation measures. BoF approach with SURF keypoint descriptors is implemented to categorize the images using Multilayer Perceptron (MLP) classifier. The obtained results demonstrate that the DRLS method is able to delineate lung fields from CXR images. The BoF with SURF keypoint descriptor is able to characterize local attributes of normal and TB images. The segmentation results are found to be in high correlation with ground truth. MLP classifier is found to provide high Recall, Specificity (Spec), Accuracy, F-score and Area Under the Curve (AUC) values of 87.7%, 85.9%, 87.8%, 87.6% and 94% respectively between normal and abnormal images. The proposed computer aided diagnostic approach is found to perform better as compared to the existing methods. Thus, the study can be of significant assistance to physicians at the point of care in resource constrained regions. © 2019, Springer Science+Business Media, LLC, part of Springer Nature.