I'm 6'3", left handed, and forever afraid of looking bad at something in front of other people, so basically avoided the whole thing (even as a caddied my way through high school and college) until about 5 years ago, when I bought one of those Strata bag-in-a-box sets off Amazon for like $200. Took about 3 years to even flirt with 100, playing 10ish times a year. Like you, finally upgraded my irons to clubs that fit me/aren't made out of basically plastic, and have managed to break 100 a few times and 50 in 9 a few more times. I realize that I started about 15 years later than my friends, I'm not going to have the time to commit to this to make any serious progress in the near future....it's still frustrating as hell to duff an approach after a rare good drive or have that stretch where you can't get the ball off the tee, but man just being out there and walking the course with a beer in your hand beats pretty much anything else I do socially these days.
My friends who are super into have dragged me to courses I have no business playing on....Sand Valley a couple times, Streamsong a couple months ago. Yeah, it's not cheap, but when you're paying that much for a good concert ticket or a really nice dinner these days anyways, just enjoy the fun of it, the scenery, the 4 good shots you have each round, etc. There are golf shots I hit 3 summers ago I think about weekly.