We have studied self-assembled InAs quantum dot (QD) ensembles on GaAs(001) substrate using polarized photovoltage spectroscopy. The photovoltage spectrum shows four prominent QD related features whose nature differs for probe light incident along [001] and polarized parallel to [110] and [1 1- 0] directions. The polarization anisotropy suggests that for the lowest energy transition there is only a change in the oscillator strength with change in polarization, while for the higher energy transitions there is also an apparent shift in the transition energy. By comparison with simulations of the absorption spectrum, we show that the main features of the observed polarization anisotropy can be understood on the basis of a model where an anisotropic two dimensional harmonic oscillator potential represents the lateral confinement of the carriers within the QD in the (001) plane. © 2005 American Institute of Physics.