The onset of convection instability in a differentially heated layer consisting of gray and non-gray gaseous mixtures is studied numerically. The conditions investigated cover a wide range of Planck number values (Pl=κkT4σT03), from the conduction-dominated regime of Pl≫ 1 to the radiation-dominated regime of Pl≪ 1. The linear stability theory is applied to mass, momentum and energy balance equations and the resulting linear stability equations are solved by Chebyshev spectral collocation method. The divergence of radiative flux is solved by the finite-volume-based discrete ordinates method. The Spectral Line Weighted sum of gray gas (SLW) model is used to represent the fine spectral variation of absorption coefficient for a non-gray gas medium. The results indicate that the critical Rayleigh number (Rac) for the onset of convection increases with mean temperature (T) in the conduction-dominated regime at low values of T. In the radiation-dominated regime (Pl≪ 1), Rac decreases with T for gray media. If the medium is non-gray, the critical Rac reduces to even lower values (as compared with those of gray gases) due to the dependence of gas absorptivity on temperature T. A reduction in the wall emissivity value increases the stability of the fluid layer due to reflection of radiation from the wall, in the radiation-dominated regime. The reverse trend is seen for Pl≫ 1. The critical parameters also significantly depend on the concentrations of radiatively participating gases in the mixture. The temperature profile in the fluid layer transforms from a linear profile in conduction regime to a stratified profile with steep gradients near the walls, in the presence of non-gray participating gases. © 2021, Indian Academy of Sciences.