Unless there is a truly all-time weather event, you won't be backed up for hours getting from Silverthorne / Dillon / Frisco to Keystone or Breck. If it's a powder day (even midweek) or a nice weekend, traffic can be heavy, but not intolerable. You do want to arrive early (or after noon) on weekends and powder days if you want guaranteed parking - the lots tend to fill up between maybe 10 and noonish on big days.
Taking the Summit Stage (free bus) avoids that, but the routes aren't that convenient. If you stay in Summit Cove, there's an hourly bus that goes direct to both Keystone and Breck and it's really nice experience to roll in at peak times and not worry about parking.
Silverthorne / Dillon / Frisco to Vail and Beaver Creek can be quite slow (an extra 45 min+?) on big weekends and powder days because Copper is on the way and the off-ramp for Copper gets very backed up while everyone is trying to park. That's new this winter, really, and it kind of sucks.
The drive up from Denver to Summit County can be a multi-hour ordeal in weather or at peak times, yes. Not sure there's a good way around that, sadly, beyond being flexible.
ANYWAYS, if you're staying in Summit County, and you have an Epic Pass and don't mind being flexible, you'll be fine. Check Google maps in the morning and see what the roads look like and decide where you want to go, or take the bus. Midweek you really don't have to worry much about any of this (usually
).
We have a condo in Summit Cove and I ski Keystone and A-Basin (mostly on weekends) all winter and this is my experience.