Without a boot fitter to make the needed adjustments, I don't recommend sizing down. I did size down from a 25 to a 24 to a 23, but I worked with boot fitters along the way.
@SnowHot and her husband had to do a lot of work on my boots to get them to fit as well as they do. I noticed that you don't have custom footbeds or plan to get any. A custom footbed is fitted to your foot, supports the arch and instep, and effectively shortens the length of the foot as compared to a flat footbed. If you do decide to get custom footbeds, they will last a REALLY long time and transfer boot to boot in the future. Without them, you may not be able to get into the smaller boot.
If a boot is too stiff for you to flex it, it will keep you from progressing as a skier. It's not a simple function of higher flex number is a better boot. Yes, more advanced boots often have better buckles, liners, etc.; but if you can't flex it properly, it's no better than a piece of junk.
@WaterGirl is right ... since you will be in Tahoe and Mammoth, there are some really good boot fitters.