Let \(q\in [1,\infty )\) , k be a positive integer, X a ball Banach function space, and \({\dot{W}}^{k,X}(\mathbb {R}^n)\) the ball Banach Sobolev space. Under some mild assumptions on X, the authors find two positive constants \(\gamma _{(n,q,k)}\) and \(\Gamma _{(n,q,k)}\) (both are sharp in some sense) such that, for any \(f\in {\dot{W}}^{k,X}(\mathbb {R}^n)\) , where \(P_{B(\cdot ,r)}^{(k-1)}(f)\) denotes the \((k-1)\) -th minimizing polynomial of f; moreover, \(f\in {\dot{W}}^{k,X}(\mathbb {R}^n)\) if and only if f is locally integrable and the above limit exists and is finite. In addition, the authors also show that, for any \(f\in W^{2k,X}(\mathbb {R}^n)\) , where \(B_{k,r}\) is a generalization of the ball average operator, \(a_{k}\) is a positive constant depending only on k and n, and \(\Delta \) is the Laplace operator; moreover, \(f\in W^{2k,X}(\mathbb {R}^n)\) if and only if \(f\in X\) and the left-hand side limit of this equality exists and is finite. This characterization relies only on the Euclidean metric and the Lebesgue measure of \(\mathbb {R}^n\) and hence can be used as the definition of the corresponding Sobolev spaces on metric measure spaces. All these results are of very wide applications, which are also new even when X is the Lebesgue space. To obtain these results, the authors overcome some essential difficulties arising from the lack of the explicit norm expression of X via skillfully using (modified) Poincaré’s inequality, the uniform boundedness of ball average operators on X, and Alaoglu’s theorem.