Generally, you only need to offer extra water if there is a specific reason. For us it was needed because he was constipated.
If your LO is dehydrated their fontanelle will sink in, but this is also an indicator of illness, so you should get that checked out rather than just giving water.
If you do offer extra water, don't give too much - it will fill them up and stop them wanting as much milk. The odd ounce or two is all that should be needed. I found that Dylan took in a lot more wind with water too, so make sure you get some burps, or you may get it all back with interest!