I had issues with my symptoms coming and going, one week I was so sick, my nipples hurt, the next week I wouldn't be sick, and I had to pretty much punch my boobs to feel any pain. It is normal for me to sway back and forth, and it usually comes back full swing. I would say that you should definitely call your doctor/midwife if you are concerned, it helps us ease our minds! Good luck to you!!!!
xoxoxo
P.s. Also noticed you did infertility and clomid, Do you suffer from PCOS? I also suffer from PCOS and endometriosis. Congrats on overcoming the odds!!