Generally not. You need to watch out for toxic molds that may be in your house or "off" compost (black molds are usually not a good sign, avoid breathing, can lead to aspergillosis). Rust on plants doesn't hurt people (as far as I know). Ergot on rye and some other species of related grasses can be a health hazard to people or animals grazing/eating contaminated seeds or hay.
Here's a site that has several links to info on mycotoxins: http://texasgrassfedbeef.com/science_underscores_grassfed_meats.htm Alt-F and search for myco and you'll find the links. Bottom line is they're a big problem with big-ag grain products. But I don't eat many grains or feed them to my critters, so I'm not too concerned.
'Science is the father of knowledge, but opinion breeds ignorance.' - Hippocrates
Poop goes in a willow feeder. Wipe with this tiny ad:
Limited Time 'Buy One Get One Free' Buddy Bonanza Event for the Permaculture Technology Jamboree