Brandon,
Worm casting are a great first step to improving the fertility of your
gardening soil. Actually you remind me of myself when I first started
gardening as I was always trying to find the perfect organic amendments to add to my clay soil (at the time it was strip mine tailings, so not too far off from the dirt you were planting in).
If you were adding one and only one amendment, worm castings would be an excellent addition. More importantly though, I would try to get life into that soil. Encourage all sorts of microbes and other forms of life (like worms) to grow and you will get better results than just adding an amendment. To do this, add organic matter and the soil life will thrive, and so too will your crops.
Bountiful soil fertility does not happen overnight though. It is the result of long term effort, but don’t get discouraged. The worm castings are a great 1st step, but the are really just a first step. If you build a compost pile on your garden beds over the winter, all the nutrients and soil biota will leach into the ground below and can be extremely helpful. I always keep a mulch on the surface and it does wonders for the soil beneath. The more life you add to the soil, the better things will grow. It is amazing.
If you want any other help, please don’t hesitate to ask. Permies is just a few keystrokes away.
Eric