This is a good question and one that encites a lot of controversy. However, once the sperm is dried it is not able to impregnate you.
I wrote at length about this topic with an article on the misconceptions of conception:
"This is one of those questions where there are many variations on the theme. Someone will ask if they masturbated using their hand and some ejaculate got onto their hand, they washed their hands, played basketball and then saw their girlfriend and touched her vagina and then want to know if she will become pregnant. Or someone might ask if dried sperm can come back to life when moistened.
I will do my best here to answer the many versions of this question as it comes up so frequently on our site.
The American Pregnancy Association is pretty clear in stating that once sperm is dry it is dead. Sperm cannot be brought back to life by being re-moistened. So if the sperm has dried it is not going to get anyone pregnant.
Sperm can live up to five days if it stays within a warm moist environment such as a woman's cervix.
But outside of the human body sperm have a much shorter existence.
There is much variability in the answers given as to how long sperm can remain alive outside of the human body such as on a hand. It seems to depend upon how exposed the semen is to air and humidity and this pregnancy site tells us that sperm can last from twenty minutes to an hour outside of the body.
But then there are sexual health organizations such as this one which say that sperm are able to live for up to four hours outside of the body.
So basically if the sperm is dry then there is no chance for pregnancy. But if you have ejaculate on your hands which is not dry or washed off completely and you get that ejaculate anywhere near your female partner's vagina then perhaps there is a slight risk for pregnancy.
Your best bet is to always wash your hands thoroughly if you have any seminal or pre-ejaculation fluids on them before touching your female partner's vulva or vagina."
Hope this helps some!
One way to resolve this worry is to use spermacide in addition to a condom or use additional forms of birth control.
Thank you for your question.