The "go to location" feature DOES point back directly to where you started - if of course you saved your start as a location!