Do you HAVE to invest in ski-specific gear? No, of course not.
Should you, at some point? Yes you should.
Ski-specific gear is high dollar gear because it must perform reliably though a larger range of conditions than most gear. We ski in temps from way below zero into the 60s and 70s. In conditions from bright sunshine through rain and puking snowstorms. But worst of all, through periods of intense activity (actually skiing) to periods of total inactivity (sitting on the chairlift), and we require temperature and moisture regulation through all of it.
Few types of gear will excel through that massive range of temps, activity, and conditions, and ski gear is specifically designed for that range. This means it usually has superior waterproofness and breathability to keep our core temps and moisture levels steady throughout the day, and ways to allow moisture to exit like extended pit or thigh zips. It usually also has pockets specific for ski gear, like goggle pockets, media pockets, hand warmer pockets, and often little extras like goggle wipes, key hooks, SAR whistles, RECCO patches, helmet-cut hoods, waterproof zippers with wind block, reinforcement in the knee and butt areas, soft material around the neck and face areas, and season pass/lift ticket attachments. It is usually also cut to allow the wide range of movements required while skiing without binding in key areas or riding up/sliding down around the waist area.
High end climbing or hiking gear will work with a variety of layers underneath. But once you've decided you really love skiing, you'll want to treat yourself to the gear that will allow you to be out there in the full range of ski conditions. Wear what you have and be on the lookout for the pieces that make your heart sing at the right price.