We consider complete noncompact Riemannian manifolds with quadratically decaying lower Ricci curvature bounds and minimal volume growth. We first prove a rigidity result showing that ends with strongly minimal volume growth are isometric to warped product manifolds. Next we consider the almost rigid case in which manifolds with nonnegative and quadratically decaying lower Ricci curvature bounds have minimal volume growth. Compact regions in such manifolds are shown to be asymptotically close to warped products in the Gromov-Hausdorff topology. Manifolds with nonnegative Ricci curvature and linear volume growth are shown to have regions which are asymptotically close to being isometric products. The proofs involve a careful analysis of the Busemann functions on these manifolds using the recently developed Cheeger-Colding Almost Rigidity Theory. In addition, we show that the diameters of the level sets of Busemann functions in such manifolds grow sublinearly. }