It is definitely possible to give genital HSV-1 back to your boyfriend, but less likely.
I would also be careful when your boyfriend has a oral HSV outbreak. It is possible to spread HSV-1(re innoculation) to different parts of your body. For instance, oral-anal contact can give you an infection on your anus, where you might have been previously infected on your vulva.
Good luck!