I was just diagnosed with genital herpes (HSV 2), so I'm no authority by any means, but as you can guess, I HAVE been doing A LOT of reading and learning since my diagnosis.
I'd always heard what your guy says: You can't spread it without some sort of outbreak. However, after reading up on herpes, I've learned about something called "asymptomatic shedding," where an infected individual CAN spread the disease, without having any symptoms.
According to the counselor I spoke with, there are different schools of thought on Asymptomatic Shedding (I'm just going to call it "AS" from here on out. It's easier.). Some physicians say that it's highly unlikely to spread the disease via AS, with no outbreaks. Others say that it's very possible. In my own estimation, I think it's safest to assume that you could contract the disease, but it's less likely if there's no outbreak. Your safest bet is to use a condom... the problem is that it only protects you if his outbreak is ONLY on his penis. Apparently outbreaks can happen anywhere in the "boxer short" regions, including the butt.
Which ties into my next point... outbreaks can sometimes be very subtle or in inconspicuous locations. Even if your guy is 100% right and AS is a crock and you can't spread it unless there's an outbreak, it's still safer to just be careful in case of a less-noticeable outbreak.
If using a condom during sex is a deal-breaker for the guy, you may want to question his priorities and if this is the best situation for you.