We extend the methodology introduced for the initiation of damage within the context of a class of elastic solids to a class of viscoelastic solids (Alagappan et al. 2016 Proc. R. Soc. Lond. A: Math. Phys. Eng. Sci. 472, 20160231. (doi:10.1098/rspa.2016.0231)). In a departure from studies on damage that consider the body to be homogeneous, with initiation of damage being decided by parameters that are based on a quantity such as the strain, that requires information concerning a special reference configuration, or the use of ad hoc parameters that have no physically meaningful origins, in this study we use a physically relevant parameter that is completely determined in the current deformed state of the body to predict the initiation of damage. Damage is initiated due to the inhomogeneity of the body wherein certain regions in the body are unable to withstand the stresses, strains, etc. The specific inhomogeneity that is considered is the variation of the density in the body. We consider damage within the context of the deformation of two representative viscoelastic solids, a generalization of a model proposed by Gent (1996 Rubber Chemistry and Technology 69, 59-61. (doi:10.5254/1.3538357)) for polymeric solids and a generalization of the Kelvin-Voigt model. We find that the criterion leads to results that are in keeping with the experiments of Gent & Lindley (1959 Proc. R. Soc. Lond. A: Math. Phys. Eng. Sci. 249, 195-205. (doi:10.1098/rspa.1959.0016)). © 2018 The Author(s) Published by the Royal Society. All rights reserved.