I have also given up all dairy except butter,cheese, and some heavy cream occasionally. I do eat alot of cheese, probably more than i should because it has little effect on my bg and is delicious. Milk is not good for the bg, i have seen it recommended for use as a treatment for hypoglycemia(similar to glucose tabs). In my pre-paleo days, i would eat a big bowl of cereal and milk in the morning and wonder why i couldnt control my bgs even with careful carb counting. lol!
I too drink almond milk, silk unsweetened vanilla flavor from walmart. 3.50 for 1/2 gallon and 1 carb/8 oz serving. tastes pretty good, takes some getting used as it is very different from cows milk