Respectfully, I'm going to disagree with what you've said. It's more than a case of simplistic maths, with calories in equalling calories out. Some people hold onto calories better than other people, which is why diets are more effective for some people than others. For people who are fat, and are dieting almost all of the time, their bodies get trained to be really good at holding onto calories.
Not all calories are equal, either. I'm sure we all know how much more difficult it is in extracting calories from salad vegetables than a liquid such as beer. Meat or fish takes a lot of energy to break down, massive compared to the amount required for cake, so it's not just the number of calories that you consume.
Your metabolism doesn't have to slow, either. Mine hasn't, probably because I got into HIIT before it had a name, and it's something I've kept up as I find it works well. One of the things that happens to a lot of people is that they lose muscle - and muscle burns a lot of calories, even when you're asleep. If you maintain high resistance training, such as weights or running up very steep hills, then it is possible for men* to maintain or even increase muscle which will increase your metabolism
* It's a lot harder for women to build muscle
If you follow much of the current research in biochemistry, a lot is talked about time-restricted eating, (TRE), where you attempt to consume all of your calories in a short 'window'. The principal is that your body is able to spend more time in 'maintenance mode' when it is not doing the hard work of digesting food. I've been following a 12hr eating window for some time and I find it works well for me, although some experts recommend a 10hr or less window. This is for general health, rather than weight control, but you have highlighted the fact that digestion is better at certain times of the day.
It goes without saying that this makes having a social life difficult!