In speaker verification task requires some sort of background model for the system to make decision. Most of the cases, a speaker independent large Gaussian Universal Background Model (GMM-UBM) is used. In this paper, we propose to use a Speaker Cluster-wise UBM (SC-UBM) for a group of target speakers. In this method, the target speakers are clustered into group based on their similarity in Vocal Tract Length Normalization (VTLN) parameter. The VTLN parameter depends on the physiological structure of human speech production system. Hence, the group of speakers with same VTLN factor represent a speaker with unique characteristic. The SC-UBMs are derived from GMM-UBM with Maximum Likelihood Linear Regression (MLLR) by pooling data from the specific group of target speakers. The speaker dependent models are then adapted from their respective SC-UBM using Maximum a Posteriori (MAP) method. During verification, the log likelihood ratio for the claimant is calculated with respect to the corresponding group specific UBM. The comparative study are performed on NIST 2004 SRE in core condition. The SCUBM system reduced equal error rate (EER) by 9% over the GMM-UBM system. ©2010 IEEE.