I love trees. That said I have taken down a couple big ones in my time. Both cases were trees that were dead or dying and dropping limbs. I watched as one limb, which seemed smallish to me, fell directly onto my shed, crumpled the roof, and split the shed open. The reverberations could be felt through the ground. what if it had been a medium limb? shudder.
You're not exactly in that situation but your ability to produce food is being threatened. I consider that a valid reason to remove a tree, as long as you've thought through the ramifications.
Weeds are going to start sprouting like ... weeds once that sweet sunlight is there. You need to have an immediate mulching/cover plan or you are going to be waist deep in weeds.
Is the tree a windbreak? Is it holding a hill together, or preventing erosion? If so, when you get rid of it you may have wind damage, standing
water, or rivulets you weren't expecting.
Can you safely take down the tree?
Do you have a plan for the
wood? I hope this is an easy yes. Lumber,
hugel,
firewood, selling it, what have you. Processing a tree is tough unless you have big equipment.
If you are ok with all that and the only thing stopping you is guilt? For a tree that is blocking sunlight to your garden? All I can say is I cut down a tree a couple months ago for just that reason.