Gambatte, I think you've bought into propaganda there. The populist press whipped up a frenzy about this because it sells newspapers. Self determinism was probably a key driver for most people.
Before we joined the Common Market, there were perfectly adequate mechanisms for movement of people between european countries. No reason to think that will not continue despite what the scaremongerers would say.