Toxo is actually rather difficult to catch, believe it or not. So, don't worry too much! However, like everyone has said, IF you clean the boxes, wear gloves. And wear gloves if you dig around in outdoor dirt.
I was a zookeeper for 6 years and worked with exotic cats (caracals, tigers, etc). I also worked with meerkats that were actively dying of toxo infections.
Because of this, I had a toxo titer done when I got my bfp and found out I have never been infected (despite the handling of animals and their wastes that had toxo). You practically have to EAT infected poo or meat to get it.
SO! Don't eat Yoshi's poo!! And don't cut in him to slices and eat him (her?) raw!!
Seriously, the only danger is if you get infected feces on your hands, don't wash them, then handle food or put your dirty hands in your mouth.
I have 6 cats and wouldn't dream of giving them up!!