Peter, "The Hand Sculpted House" lit a fire in me too. That is one of my favorite books, along with "One Straw Revolution."
To answer your question, having a college degree lets you work for other people. It buys you stability. If you did not have a clear passion (natural building, homesteading), then I would say it might be a good path for you. I've worked with many directionless people who were perfectly happy to come sit in their office all day and go home to watch TV while raking in that steady paycheck. It strikes me as odd, but it works for them.
You, on the other hand, have a fire burning in you to learn natural building and start a homestead. I can tell you from experience that if you run away from that passion and follow a more traditional path (college, salary job working for somebody else), that passion will continue to gnaw away at you until you finally cave in and do something about it. Either that, or it will slowly die, which is even worse.
But your question is about practicality, balancing what you're interested in with the very real need to earn money. By the way, that's wonderful that you have no debt and are good at living frugally. You're already doing great.
If you are interested in any state or federal government jobs, then I think the BA may be worthwhile. Unlike private companies, where you can weasel your way into a high ranking position based on (gasp) your knowledge and skills, of which it sounds like you have plenty, government jobs are highly prescriptive in their requirements and do not make exceptions. Most require a college degree. They will also determine your pay based on your years of experience calculated, once again, following a strict formula. My experience working in state government years ago was generally positive. It's very stable and predictable, you get decent benefits, and you're (in theory) serving the greater good. You could get a fun job in an environmental or fish & wildlife department. It would likely have to be full-time. On the downside, the inefficiency and apathy in government can drive you nuts.
If you're not considering a government job, then I don't think the college degree is worth it unless you're confident it will lead to a lucrative, well-paying job. No doubt, if you pursued a BA you would learn new things and gain some skills - a college degree is not as worthless as people make it out to be. However, a college degree is often not worth the investment of time and money as a learning experience. A college degree can be worth the time and money as a hoop to jump through if it lands you in a career you want (not likely based on your interests) or a high-paying job. With a high-paying job, if you continued to live frugally, you could potentially build your homestead up slowly over time, tinker with natural building on the weekends, then retire early and live your dream life (still frugally) maybe starting in your 40's. Delayed gratification sucks, but with that plan you're getting the best of both worlds, especially if you get your food-producing plants in the ground early and have years for them mature while you're still slaving away for a paycheck.
You may want to do some math. Suppose you can find a job now or cobble together enough side gigs that you earn $30,000 a year. Could you live on $20,000 and stash away $10,000 each year? Awesome. Over the next 15 years you will save $150,000 plus interest. Suppose you go to college for 3 years and net $0 during that time - no debt and no savings. Then you get a job that pays $50,000. For the next 12 years you live on $20,000 and save $30,000 each year. In the end you have saved $360,000 by the same age living at the same level of spending. Just crunching the numbers for this entirely hypothetical scenario, going to college makes more sense. You may want to do some research and look at various realistic scenarios in the same way so that you can weigh the cold, hard numbers against the enjoyment and satisfaction you would experience in each scenario.
I wish I had a clear answer for you, but what I can do is tell you about my own personal experiences having college degrees. Maybe seeing how it played out in real life for someone else could offer you some insight to help you make up your mind. Since the day I graduated, I've worked full-time in jobs that required at least a bachelors: four years in state government, seven years teaching at a public university, and now one year at a private organization. In general, my jobs have been pretty cushy and low stress. I've always had enough money, usually more than I need. I've gotten to interact with tons of wonderful, intelligent coworkers from all different backgrounds. I've learned a wide variety of skills, many of which I never would have learned if I were just doing my own thing and not being forced outside my comfort zone. I've gotten to travel for business trips. In fact, I just returned from Toronto last night, where I stayed in a fancy hotel and went out for drinks every night with very smart, kind people from all over the world, all on the company's dime. There are perks to working at a professional job that requires a college degree beyond just a better salary. Most importantly, though, I've gained an understanding of what makes the world run - board meetings, policy development, office politics, committees, memos, organizational hierarchy charts - mind numbing things that make you want to bang your head against the wall. However, understanding how the business/government world works puts you in a much better position to create change in the world if there's something you're passionate about.
The downside of these jobs is that they were not completely fulfilling. Even worse, I feel like a slave working 8 to 5 every weekday all year long with only two or three weeks of vacation time that gets eaten up by sick kids or dentist appointments. I, personally, have not had much success finding part-time jobs or jobs with flexible schedules that pay well and are intended for people with college degrees, but that could just be me. So I am trapped in the rat race of working a rigid full-time schedule and never having a chance to pause and reflect on the what the heck I'm doing. I look back at the last 12 years and feel like I mostly wasted my talents and extinguished my inner creativity. As long as I've been working, I've felt restless and daydreamed about running my own small farm, which was my dream long before I ever went to college. Yes, having a comfortable lifestyle has been nice, but in my opinion it has not been worth selling my soul.
Soon I will jump ship and start building the life I want, which is made much easier because of my college degrees. The reason is that I will be teaching online (about 10-15 hours a week) to make just enough money to cover the bills. That affords me the opportunity to make a lot of mistakes while I learn how to eke out a living doing the things I actually love. Without my college degrees, I would not have the luxury of being able to make a smoother transition from working for the man to working for me. Having a steady paycheck, even a small one, brings me peace of mind, which is important to me since I have children and a family to think about. It sounds like you may not be as tied to stability and predictability as I am, given your description of your years sailing, rock climbing, backpacking, etc., which is why I lean a little more toward not going back to college in your case. The only exception is - you mentioned looking at Sterling College. If you could get into the next round of the Wendell Berry Farming Program, it looks pretty amazing. I'm sure it's competitive.
Best of luck to you! When trying to make a difficult decision, I have two pieces of advice. First is to remember that no matter which path you take, you can always change your course, and you will always learn something. The second is to flip a coin and say "heads" means you go back to college. If it lands on heads and you feel disappointed, then maybe that's not what you really want to do. If it lands on heads and you feel relieved, then maybe that is what you want to do. I know that's kind of stupid, but it has worked for me before. Good luck to you!