In this work we give a sense to the notion of orientation for self-similar Gaussian fields with stationary increments, based on a Riesz analysis of these fields, with isotropic zero-mean analysis functions. We propose a structure tensor formulation and provide an intrinsic definition of the orientation vector as eigenvector of this tensor. That is, we show that the orientation vector does not depend on the analysis function, but only on the anisotropy encoded in the spectral density of the field. Then, we generalize this definition to a larger class of random fields called localizable Gaussian fields, whose orientation is derived from the orientation of their tangent fields. Two classes of Gaussian models with prescribed orientation are studied in the light of these new analysis tools.