I was vegan for more than a year and switched to vegetarian after the first few weeks of my pregnancy, exactly because I did not feel comfortable consuming soy any longer. I drink fortified almond/oat/hazelnut milk and eat cheese and eggs occasionally mainly for B12 vitamin and protein. The way I see it, there is no reason to consume soy and since the data is conflicting and inconclusive, I decided to keep it out of my diet as much as possible. The issue with soy is that it hides in many processed foods like baked goods and margarine as well, so you are probably consuming a small amount anyway, without realizing it.
If it is protein you are worried about, just increase your intake of beans, lentils, chickpeas (hummus, for example), mushrooms and kale. Of course, eggs and dairy will boost your protein intake as well, if you feel comfortable consuming them.
If you feel that you still want soy in your diet, consume it in fermented forms like tempeh and soy sauce.
Best of luck and don't worry too much. Many children have been born to vegan women who consumed soy products as a pp said and they look fine.