When I drink skim milk I get a congested nose and my throat feels swollen - Just started month ago

When I drink skim milk I get a congested nose and my throat feels swollen - This started a month ago.  I've never had allergies or have been allergic to anything that I'm aware of.  I didn't think anything of it until I drank a Fappicinno from Starbucks and had the same reaction.  It feels like the onset of a cold or something but it goes away after a while.  Several people said I'm either allergic to coffee or milk but that I should have been aware of this since its rare to just all of sudden get allergic to something. They also said allergic reactions are like colds at all and maybe I have sinus issues.   I dont beleive that so that's why I'm asking this here. Maybe its in the weather because I went outside early and now I feel like my throat is closing again and my nose is stuffy.  This is irritating me.  I have an allergy blood test being done in a month but just wanted to ask here

I switch to Soymilk in my coffee thinking this might help but I think I'm gettin congested again.  Any suggestions to what this might be?