How long had it been since he last came inside you? Seman can have a soapy taste, but not so much, if at all, after its been inside a woman. My wife drips for about half a day after lovemaking, and we regularly have had oral sex after vaginal sex. However, during oral sex I do not notice any taste even when its obvious she has my seed up her from minutes before. So even if you were with another man, as I assume he's claiming, it doesn't explain the soap taste.
Are you on any medications which may affect chemisty? Is he on any medications that can affect his taste? What had he been eating earlier that day? If he is claiming your taste is semen, I think he could easily enough see if you had semen dripping from inside you, unless you were in the dark then. Often semen has little or no taste as the female chemistry neutralizes it to more of a 7 pH base. So I doubt if he thinks you have had sex with another man, that you would taste soapy.
Or, perhaps he is not claiming he tasted semen, but that he thinks you just showered and your skin tastes a bit soapy? I find that a bit odd, if that is his angle. If you rinse off reasonably well, there is a soap smell but not a soap taste. But you are saying no shower for 12 hours.
Or, it could be that you have some type of vaginal discharge or infection? I'm not familiar with yeast infections and possible taste. If the taste persists, you might want to consult your GYN to see if you have something wrong that needs treatment.
I'm sorry about his trust issues. I'm sure that hurts. Good luck.