I noticed a difference at 10 weeks, but at that point, you probably wouldn't know I was pregnant. I still fit in most of my clothes now, but I mostly also don't care to wear them because they're uncomfortable.
I've gained only a pound up to this point, so the doctor says my bump is just progesterone doing its thing (even moreso because I was taking supplements for a while, too).