The multivariate Ornstein-Uhlenbeck process is used in many branches of science and engineering to describe the regression of a system to its stationary mean. Here we present an O(N) Bayesian method to estimate the drift and diffusion matrices of the process from N discrete observations of a sample path. We use exact likelihoods, expressed in terms of four sufficient statistic matrices, to derive explicit maximum a posteriori parameter estimates and their standard errors. We apply the method to the Brownian harmonic oscillator, a bivariate Ornstein-Uhlenbeck process, to jointly estimate its mass, damping, and stiffness and to provide Bayesian estimates of the correlation functions and power spectral densities. We present a Bayesian model comparison procedure, embodying Ockham's razor, to guide a data-driven choice between the Kramers and Smoluchowski limits of the oscillator. These provide novel methods of analyzing the inertial motion of colloidal particles in optical traps. © 2018 American Physical Society.