Easy peasy, and I’m surprised you even needed to ask
It’s not the being in a Union that’s the problem, it’s who there in a Union with. There is no problem with handing sovereignty over to the EU, but there is to the U.K. That only counts for those that voted yes and remain. There’s a decent chunk that voted out of both unions.