I don't mind the wasps, except some of them will invade beehives for the honey. As will hornets. In my experience yellow jackets have the most painful sting of them all.
Another problem we have is ground bees. The entire colony lives underground and if you happen to step on it or run over it with a lawn mower or tiller---LOOK OUT! If you don't bother them, they go about their business. We had a skunk once destroy an entire colony of them, leaving a huge hole and discarded pieces of comb all over the place. The bees never came back, but we keep a wary eye out just the same.