Generally not. You need to watch out for toxic molds that may be in your house or "off" compost (black molds are usually not a good sign, avoid breathing, can lead to aspergillosis). Rust on plants doesn't hurt people (as far as I know). Ergot on rye and some other species of related grasses can be a health hazard to people or animals grazing/eating contaminated seeds or hay.
Here's a site that has several links to info on mycotoxins: http://texasgrassfedbeef.com/science_underscores_grassfed_meats.htm Alt-F and search for myco and you'll find the links. Bottom line is they're a big problem with big-ag grain products. But I don't eat many grains or feed them to my critters, so I'm not too concerned.
'Science is the father of knowledge, but opinion breeds ignorance.' - Hippocrates
Of course, I found a very beautiful couch. Definitely. And this tiny ad: