My due date for child Three is in about two weeks, and as I had to give up gardening while pregnant, this is my kind of topic. I look forward to some digging and planting soon in the future!
Wihen our oldest was very small, I either parked him asleep in his stroller or carried him in the sling to garden. When he started walking, he just came with me and "helped." I really thinking having the kids with you and helping is important, as they learn more than just gardening--they learn self control, respect, and how to work--among other things. Yes, you could do it faster without them. But I think having them do it themselves, or help at least, is setting up a good foundation for the future. Our oldest son is now a (mostly) good worker and I can trust him to do all sorts of jobs around the house and garden; he's almost ten now. He sometimes wishes he could be like his friends and not have chores! But it's setting him up to be an adult, and he'll know how to do all sorts of things his friends won't.
Also, I see it as training him to be a good husband/partner/etc. I think this is especially important for boys, as I know it's hard on a woman (and on a marriage) to have to take care of not just the kids but the husband too; I think this expectation isn't as strong as perhaps it was in the past, and my husband isn't completely useless like some men I know, but I would certainly find it much easier if he took on an equal share of the house and garden work.