Do a lot of people see and hold him regularly? Like family? He might be super attached to you since you are home with him all the time and your DH is away at work. I'm always with Erica, but both her grandmas, her aunt and uncle are around a lot, so she is kinda used to other people holding her.
Someone told me tho that babies go through a phase where they will only let their parents hold them, maybe Josh is reaching that stage already? He is such a cutie too btw!! I love the pic in your avatar!