Even in ancient Greece:
The so-called Pythagoreans, who were the first to take up mathematics, not only advanced this subject, but saturated with it, they fancied that the principles of mathematics were the principles of all things.
– Aristotle, Metaphysica
For thousands of years mathematics grew in a way that was tangible and “real.” Newton made great advances in physics by incorporating and expanding upon the differential calculus of his day. Einstein’s physical theories of space and time have their most natural exposition with the nineteenth century differential geometry of Poincaré. In the twentieth century, the gauge theory of particle physics and the mathematics of vector bundles grew side by side. The physicist Eugene Wigner called mathematics “unreasonably effective” in its ability to describe physics.
But developments of the twentieth century also fractured the math-physics bond. First was a mathematical tendency toward general, abstract mathematics (logic, topology, algebra, algebraic geometry). Second was the purely mathematical progress in fields which were originally tied to physics such as differential equations or geometry but were growing independently. Third was the maturation of particle theory (requiring no new mathematics) and eventual development of the so-called “standard model.” In 1972, the physicist Freeman Dyson had this to say:

I am acutely aware of the fact that the marriage between mathematics and physics, which was so enormously fruitful in past centuries, has recently ended in divorce.
– Freeman Dyson, Missed Opportunities, 1972.