Never found it that difficult to be honest. Back in the mid /late 60's, as a teenager, I converted my 10 spd Carlton Catalina to (ss) fixed and took to it like a duck to water. One memorable ride was Lancaster, Kirkby Lonsdale, Ingleton, Settle, Ribblehead, Ingleton, Burton-in-Lonsdale, Lancaster in one afternoon; the day after finishing my 'O' levels!
In my twenties I bought a Pennine Re Della Corsa Sprint track / road bike and covered lots of miles on that. The only accident on the Pennine was not due to it being fixed, but my own carelessness - "honking" out of the saddle up a slight incline with my head down, failed to notice a new Ford Capri Mk2 parked by the kerb, ended up sprawled across the huge rear window of the hatchback, with the handlebars wedged under the rear diff and my feet still clipped into those super smooth Campag Record track pedals!!