You absolutely can refuse, and if you don't feel comfortable, there's no reason to put yourself through that. It's definitely something that can wait until after pregnancy. The virus that causes cervical cancer (and the vast majority of abnormal cell changes) is incredibly slow progressing in all but a few very rare cases. In most people who even have it (which frankly, is pretty much all sexually active women), even when it causes abnormal cell changes on the cervix, the body is able to fight it off in time and clear all the bad cells, so that everything goes back to normal. I had a bunch of abnormal smears years ago and had to have a colposcopy and biopsy. They were pre-cancerous, but not incredibly serious, and two years later, my body wiped them out and my smears went back to normal.
I would say the thing to think about is what you would do if you got an abnormal result. I think the approach in the UK is to typically wait a year after the abnormal result and do another smear to see if it's still abnormal. Would you want to have that to worry about for a year while you should be enjoying your pregnancy and new baby? Or would it be better to wait and get the smear done later knowing they wouldn't do anything in the meantime but ask you to wait? I was living in the U.S. at the time I got my first abnormal results and there the approach is to to go straight to the colposcopy and biopsy (it's a very profitable procedure, so there's no waiting if they can make money off you). You probably wouldn't want to have a colposcopy done while pregnant (because you couldn't have the biopsy anyway, which could potentially weaken your cervix and cause premature dilation - actually it can cause risks during pregnancy even if like me, you've had it done way before you got pregnant, nor could you be treated for cervical cancer during pregnancy). So I guess the thing is, think about what your next move would be if anything abnormal did come back, and whether you would want that during pregnancy or if you'd rather just wait. In the small chance that there are abnormal cells, more than likely if you just wait without testing, they'd clear themselves out in a year or two.
And even if they were the particularly virulent kind that did progress to cancer, it takes on average 10 years for cancer to actually develop (which is why most guidelines only recommend screening every 3-5 years anyway). So there is plenty of time to attend to getting a smear and worrying about this stuff once you have your baby. Knowing my history with abnormal smears and knowing what I know about medicine, I wouldn't worry about getting one done during pregnancy. It's not necessary, unless you were having lots of unusual and worrying symptoms, like bleeding and pain during sex, before you got pregnant. If you are worried about STIs or BV, they don't test these during a smear anyway (though many people do get them done during the same vaginal exam). These are actually tested using a cotton swab and not the wire pipe cleaner thing used for smears, and are easy to do during pregnancy if it's something that concerns you.