It is highly unlikely that any damage will have been done to your baby as of yet. The fact that you have now stopped, is the main thing. You have to remember that in the first 10-14 days or so, your baby is not even a part of your system. It is it's own little being, floating along and multiplying. It will have no direct link to your blood supply at this time, which is why even blood pregnancy tests may not pick it up at this time.
Many women realize that they had alcohol, fish, etc... before they knew they were expecting, and it would be fair to say that 99% of those pregnancies were fine.
The main thing is that you stop now. As for caffiene, they estimate you can safely have 250-300 milligrams a day, even after you have found out you are pregnant, without causing significant effects; over that amount, and they have found links to miscarriage.
Good luck. Relax. You are fine.