i seem to be hitting on your stuff today! I also began late in life, but advanced quickly. I too had a thing with speed, even though i had been a motorcycle rider and was used to speed! I am 5'8", 130 pounds, and ski metrons at 152 and 164 lengths, depending on the day, etc. I think the other posters have excellent advice about chatter--be sure your skis are solid (i love the atomic metrons...).
My suggestion? One year, i decided that i would tackle speed as my issue. I didn't head for the steeps for this; i simply stayed on the blues when i wanted to tackle this, usually a couple of times a day (never a lot of runs with speed in a row, or I was afraid I would have a heart attack!). I just bumped up the speed a little bit on each run, did that until i was comfortable, then bumped up the speed a bit more, etc, until i was fine with speeds. It might take an entire season, but so what? Don't scare yourself silly...
Now, as for the question why would you want to go faster? Simply put, for control. I don't like to be a speed skier, but I want to be able to handle the speed I need with powder with comfort and, more importantly, if i EVER end up in a condition where i suddenly speed up (like on ice and a double black), I don't want to freak out. Also, there are those moments, like after a cornice jump, when the landing is very fast until you turn--I didn't want to bail on that, simply out of fear.
Just have fun with it. If it isn't fun (i.e. anxiety), dial back a bit until it is. Take your time to gain speed (oh, the paradox in that. how zen of me)