That really really really isn't bad for baby. Only if you spend a lot of time lying around on your back. And even then it doesn't have to be a problem. It is only a problem if the entire weight of the uterus etc interferes with the circulation from the vena cava (main vein). That rarely happens, or better said, it rarely causes any problems.
Other than that vein thing (again, isn't very common to cause issues), the extra weight can cause worsening of things like hemorrhoids and backache and interfere with digestion (more constipation). It's just not the best position to be in. But not dangerous, most of the time.
Just don't stay on your back all day, all night, every day and night.
Here in the Netherlands it isn't even a thing and my midwife looked at me as if I was insane for asking her.
Don't worry, you are not harming your baby by waking up on your back from time to time
Edit: see it like this, they advise sleeping on your left side because it is the optimal position for the best bloodflow, best digestion etc. But that doesn't mean that if you always sleep on your right side that your baby's health is ruined. Or worse than it should be/could be.