Nobody really thinks about these things. Those who do can't obtain the title of "scientists".The lower the gravitational potential (the closer the clock is to the source of gravitation), the slower time passes, speeding up as the gravitational potential increases (the clock getting away from the source of gravitation)
If time goes slower at sea level than on a mountain, they could place 2 (or more) clocks, at sea level and on a mountain. Geographically close. Check them after some years. Easy experiment. (Remember the Hafele-Keating experiment that is said to be valid, they did not need years for that.)
Let's say - the experiment succeeds. Atomic clocks, mechanic clocks, electronic clocks, all run faster up on the mountain. "Time" is such a general force, that moves everything...Everything is moved by this "time force" - nevermind any other physical forces. (Except hour-glasses, those should run slower up the mountain. But never mind that.)
Let's say - in 1 million years clocks up the mountain gain 1 day difference compared to the clocks at sea level. Big problem - the top of the mountain has not moved into the future compared to the base. It's still there. How come? "Well time just accelerates things, it does not move things into any future" the scientists could say. Accelerates what? I am sure there are tons of physical and chemical processes that will not perform faster on top of the mountain. Does "time" work selectively?
Also it will never be night on top on the mountain and day at the base, no matter what clocks do on top of the mountain, and that is another problem. In the end they will say "it's all relative, no real physical effects", because there will be too many inconsistencies. Still, they will search for black holes and waves in "space-time".
I just proven "time" does not exist, by "reductio at absurdum". "Time" is just a count. Us humans counting things - we chose to count cyclical processes, and that is useful to us in many ways. "Time" is not a physical force, it's a concept.