It is optional. They can't force you. With my first two pregnancies I was with an OBGYN and not knowing much I went for the tests each time and was fine.
Hated the drink though.
My 3rd and now 4th pregnancy I am with a midwife. Love their relaxed approach!!! It's totally optional. With my 3rd pregnancy I skipped the test and with this 4th pregnancy I will skip the test as well! My midwives have assured me both time so far that there are other red flags for GD if I do develop it, and at that point they would require me to get tested if there was concern.
So if you want to skip it go for it, no one can force you to do anything! Just make sure to remain healthy through your pregnancy, no processed sugars (or very minimal), protein and lots of veggies! You will be fine