I think the main reason you want to exclude milk is because the protein in conventional milk (lactose) is hard for the body to digest because of the pasteurization process which removes lactase (the enzyme needed to properly digest lactose). If you stick to raw milk or raw dairy products you would be a lot better off. Pasteurized milk does show to cause inflammation in many individuals who have lactose and casein intolerance, even people with no known dairy allergies have shown in studies to show a rise in inflammation after consuming dairy. A lot of people in general and on this forum have said that after dropping dairy they lost a lot of puffiness around their body and bloating around the stomach.
I prefer coconut milk over almond milk. Unless you have a specific allergy to coconut I don't see it being too inflammatory. Coconut has many anti-microbial and anti-fungal properties that is normally very good for the gut. It is also filled with medium chained fatty acids which are pretty healthy saturated fats. Coconut milk and oil has been used in traditional diets all around the world most notably southeast Asia for hundreds of years, while cow's milk hasn't. Cow's milk has been used longer in countries where herding is in their culture, like the Swedish and Irish for example. Almond milk isn't too bad but then again it is derived from almonds. Too much nuts in the diet can skew your n3:n6 ratios.
To answer the last part of your question. I don't think whole cream in your morning coffee is a huge issue. But overdoing it with a few glasses of milk and cheese on everything might be a lot if you are really trying to lose weight.
"If man made it, don't eat it" - Jack Lallane
People say I am on a "crazy" diet. What is so crazy about eating veggies, fruits, seafood and organ meats? Just because I don't eat whole wheat and processed food doesn't make my diet "crazy". Maybe everyone else with a SAD are the "crazy" ones for putting that junk in their system.