I believe that something very close to what you describe here is the fastest line.
Immediately after the stream crossing, head up right and then swing left on a small path. This climbs gently for a short while, passes close to a huge boulder then dips down to cross a cart track, climbs out of the dip and takes you across to Pancake.