In this paper, a novel approach to assess the severity of the dysarthria using state-specific vector (SSV) of phone-cluster adaptive training (phone-CAT) acoustic modeling technique is proposed. The dominant component of the SSV represents the actual pronunciations of a speaker. Comparing the dominant component for unimpaired and each dysarthric speaker, a phone confusion matrix is formed. The diagonal elements of the matrix capture the number of correct pronunciations for each dysarthric speaker. As the degree of impairment increases, the number of phones correctly pronounced by the speaker decreases. Thus the trace of the confusion matrix can be used as objective cue to assess different severity levels of dysarthria based on a threshold rule. Our proposed objective measure correlates with the standard Frenchay dysarthric assessment scores by 74 % on Nemours database. The measure also correlates with the intelligibility scores by 82 % on universal access dysarthric speech database.