From your photo, it looks as if you were quite slim to begin with and would therefore need to put on some ample weight. I too am concerned with your doctors' criticism.
That said, some healthcare professionals are just weird about weight. I had gained about 20 pounds by 30 weeks. My practice has multiple midwives and when you go in for your appointment you just see the person available. I would sometimes get this one midwife that would tell me I was putting on weight too quickly. When I would see the other midwives, they wouldn't even mention it - I'd ask and they say I was right in the healthy range. So as you can see, there are a wide variety of opinions on weight gain during pregnancy.
For the woman who has only gained 3 pounds, I wouldn't worry either. If your BMI is a bit higher, you typically don't need to put on as much weight (not to say you shouldn't). Also, the main thing is how your baby is measuring. It's completely possible for you to lose weight and for the baby to gain, kind of evening everything out.
I think we all need to be a little easier on oursevles. This is a vulnerable time.