I am not a farmer. And I am on the opposite end of experiences from you: I've grown up in suburbia and am in suburbia now, contemplating a move into the outskirts of town to give this all a go. So I don't have any seasoned wisdom to offer, however I have been neck deep in research and what you've shared reminded me of some things I've been reading. The Independent Farmstead (Dougherty) has been hands-down the best book I've read so far. Two specific things from the book that might be helpful here:
1. The diversifcation advice shared above. This book made me realize how all parts of the homestead are interconnected and can provide cushion for failures in one area, as well as opportunities for bartering with others to help provide for you (and your future family) in a non$$$ way.
2. They emphasized very heavily the need for an in-person support system. Do you have close friends who live life with you, who understand your frustrations, who come to help when it's butchering time, etc? If not it seems like that's a critical next step, whether by being intentional with neighbors or other homesteaders/farmers in local organizations or guilds. Are you part of a local church? However you can grow very close with others it seems like it would pay off down the road.
I am very hesitant to share any of this because it sounds like an 8 year old telling a seasoned veteran what to do and I realize that's not the solution, but I am so sorry you've had such hard experiences and wanted to throw this out there.