Just a little farther out are less expensive areas. Windsor, Weathersfield, the Springfield area have some far less expensive homes and apartments. Since you are renting, you could put up with the distance for one ski season. (I suppose this is the same issue with Rutland.)
The old animosity about flatlanders had settled down considerably before the pandemic. After the pandemic started, people got frightened. Vermont had, as you know, the lowest rate of infection around. We felt comfortable and safe, for the most part. Mask compliance was pretty good (even compared to going to NH for groceries). Just like some want to blame China, others want to blame anyone who is an "outsider." It will calm down when the pandemic is mostly over.
Now I live in Beacon, NY, about 80 miles from NYC. People here complain about those "New Yorkers" who come up from the city on the weekend, crowd into restaurants, park impolitely, and are sometimes in poor mask compliance.