I like the move somewhere nearby option.
My grandfather owned 2 100 acre properties within easy walking distance of each other. One was all swamp and gravel hill, grazed by cattle, trees wont even grow on a lot of it despite not being grazed or cut for 40 years. The other was a productive and fertile farm good for growing corn, soy, hay, etc. There might be somewhere better nearby?
I would personally prefer to own 5 acres of decent soil than 40 acres of difficult soil.
Also - I hear you about everything dying. The house i grew up in had maybe an inch of topsoil over maybe 1 ft of orangey subsoil over hard granite bedrock, 100 acres of rock and scrubby pine trees. Everything died, things froze, late frosts, etc. Dad grows nice tomatos... in pots. The apple trees are 12 + and have never produced anything. If I were you, I'd consider one or two nice raised bed for veggies, and stop stressing over growing anything outside of those beds.
I pity the original farmers, who left no sign of their presence other than a few incomplete rock fences.