I've not got very much experience with ultras, but I do a lot of events (often LDWA) in the 20-30 mile range. I've been doing this for around 3 years, before that I did a lot more road races and shorter distances.
My 10k best is from about 3 years ago when I'd train for that distance, and I've not been very close since. But for all out speed, my mile and 800m time haven't dropped much - distances I have never trained for. In that time I've knocked a couple of minutes from my half marathon best, and in the midst of BG preparation set by a small margin a new marathon PB.
I think I could keep up doing the long distances a little slower than now, and bring down my 10k time at the same time if I trained specifically for that, but that's not where my motivation lies. Training for a certain distance will somtimes have some knock on benefit for other distances but once you're at one end of a scale it's not likely to help the the other end much.
Running a 10k in x minutes, or completing x miles in an ultra are both valid goals for runners, it's not a level comparison. Are oranges or apples better? The important question if one area has to suffer a little, is what do you enjoy more, long running or fast running?