While theoretical solutions to many stochastic control problems are well understood, their practicality often suffers from the assumption of known dynamics of the underlying stochastic process, which raises the statistical challenge of developing purely data-driven controls. In this talk, we discuss how stochastic control and statistics can be brought together, which we study for various classical control problems with underlying one- and multi-dimensional diffusions and jump processes. The dilemma between exploration and exploitation plays an essential role in the considerations. We find exact sublinear-order convergence rates for the regret and compare the results numerically with those of deep Q-learning algorithms.