Bit of a flaw in that argument. Trying googling "UK unemployment rate 2016" You will find it is about 5%. When it gets that low you are starting to get into very difficult to employ people. Many (not all) of these people are not interested in working/incapable of providing good value for money to employers. Without immigration business would suffer due to lack of good people.
I guess a large proportion of these people who bothered to vote; voted brexit because they are looking for somebody to blame. Possible they shoudl try looking in the mirror.
I have had two stints of unemployment of over a year and I can assure you it is some of the most depressing things that can happen to a person. The last time I was unemployed (with my PHD) and signing-on the woman at the job centre was apologising to me for making me provide evidence of my job applications as they (quote) "don't get my type of people around here".