There are two places where people seem to go wrong. The first is the church after the golf course - it's very easy to miss the footpath dropping down to the right. First time I did the race I went straight on, and last time prevented a small group of others doing the same.
The other place is right at the end, going through the town to the finish. Unless you either know the route or follow someone who does, you'll almost certainly go wrong. I'd strongly recommend checking this bit out before you start.