I think it's mostly hysteria. Fetal alcohol syndrome occurs in pregnant alcoholics. When was the last time sushi gave you listeria poisoning when you weren't pregnant? And I would love to know what harm caffeine actually does. I guess I'm just too much of a skeptic.
Given my generation's mothers' behaviors, it's honestly amazing that any of us were born without severe problems. And that those of us who survived birth didn't all die in playground accidents.
I probably sounds flippant, but the "we don't know, so it's better not to risk it" approach causes too much stress on pregnant women to be perfect. Even I haven't had alcohol or sushi, even though I think the warnings are bunk, because there's so much societal pressure on me. Even I find myself being paranoid, even though I know better.