Hard Question: Why does Joules in energy have about the same units as Torque but are very different?

Example, from how i understood it, joules is defined as the number of newtons to move an object to a distance. Breaking it down, Joules is simply N-m (Newton-meter) which happens to be the SI designation of Torque. Now why is it that in torque, the distance is not used in how far a point on the cirference of of an imaginary circle goes, but rather the distance is used to describe the lever arm or how far a force is applied from the central pivoting axis?