I guess it really depends on whether you want to live your life doing what you love to do or doing something that doesn't satisfy you, but gives you the free time to do what you enjoy. I know many people in both circumstances. I've been doing animation professionally since 1995 and am very happy with it. Though I still miss being a paratrooper sometimes, I'm also amazed that I can be paid to have fun (90% of the time) and make cool stuff (again 90% of the time) all day.
There are also plenty of people that fall into the other catagory. An ex-girlfriend of mine is a lawyer that absolutely hates her work, but it gives her the financial freedom to travel the world, do exciting adventurous sports and live where she wants. Again, she hates her daily work, but it allows her to do what she loves. There's nothing wrong with that, it just depends whether you can handle that.
Now most people won't be satisfied with doing 3d as a hobby mostly because it's not the single person creative process working on personal projects that inspires most of us. I think the majority of folks out there want to work in a team on cool blockbuster stuff everyone will see - whether it's a best selling game or A-level movie.
What it may come down to you is risk. It often requires big risks to toss everything aside and become a full time 3d artist. For me it was without a question a very scarey proposition, but it was something that I absoutely had wanted to do all my life and never imagined that it would be possible.
Getting in to the industry can be a motherf@cker. No doubt I struggled for years before I could be somewhat comfortable. In some ways I'm still struggling, but I'm much better off than when I first started.
The industry is also very subjective and localized. It's subjective in that there is no set way to make it, no combination of experience, talent and education that guarentees you a job. Much of it is slugging away, waiting to be there at the right place at the right time. It's also localized because unless you're in a major city like LA, SF or NYC, you're probably going to have limited job opportunities. In fact, a majority of the CG work is in California - though this is starting to slowly change. Game work is everywhere and is also a very good place to start a 3d career - depending on what you want to do.
I seriously think you need to ask yourself a simple question: if I was an old man sitting on the porch, looking back at my life, would I regret not persuing a career as a 3d artist doing the stuff I love?
If you can be content without it, then I think your current situation is fine. Otherwise, take a stab and go for it. A simple word of warning though: Like I said, getting in to the industry can be a bitch - if you get discouraged easily then it may not be for you. It's like anything in life - you've just got to work hard for the things you want most.
