In this paper, we propose a hybrid framework that first uses an adapted Gaussian mixture model based method to represent a varying length sequence of feature vectors as a fixed length pattern and then uses a discriminative model for classification of varying length patterns of long duration. In the conventional GMM-UBM (Gaussian mixture model-Universal background model) based classifier, a UBM is built using feature vectors of all classes. In the proposed approach, a GMM is built for each class using the feature vectors of all the patterns of that class. Then an adapted GMM is built for each example in the training data set using the GMM built for the class to which the example belongs to. The log-likelihood of a pattern for a given example-specific adapted GMM model is used as a score. A similarity based score vector is obtained by applying a pattern to the adapted GMMs of the patterns in the training set. A test pattern is also represented using a score vector. Support vector machine is then used for classification of score vector representation of varying length patterns. Our studies on speech emotion recognition and audio clip classification tasks show that the proposed method gives a significantly improved classification performance compared to the conventional GMM based classifiers. © 2010 IEEE.