Personally, I would just persist with the water. As long as they are getting milk feeds on demand still, they'll get all the hydration they need from that. They don't *need* to drink with meals, but it's good to offer it in case they do want it. My daughter didn't drink any water at all until 9 months. We just stuck with water though and she drinks it just fine now. I would assume that oat milk should be treated like cow's milk for a baby with a dairy allergy and it's recommended not to offer cow's milk as a drink before 1, as it can replace breast milk or formula (which is more nutrient dense). It's probably not doing any harm in small amounts, but personally I'd persist with the water. She'll drink it one day when she's ready.
There are tons of dairy free finger foods - any sort of fruit is great. She probably won't have her pincer grip yet, so apple or pear slices, big chunks of banana (I used to cut a whole banana into thirds, so eat piece was a bit longer than my daughter's fist), whole strawberries, avocado slices, toast fingers with mashed banana, nut butter, hummus (just make sure bread is dairy free or make your own), rice cakes, plain oat cakes (not the cheese ones obviously), cucumber sticks, soya or coconut yogurt or dairy free cheese. Also, plain pasta or with homemade/dairy free sauce (some of the store bought ones have added cheese), scrambled egg, porridge fingers or balls made with whatever dairy free milk you use, roast chicken pieces (usually strips are best at first). Once her pincer grip is more developed, blueberries, grapes, peas and sweetcorn are great.