PSA

I am a 51 year old male. Had a PSA test by internist in November of 2007, which was at 4.0.  I saw my urologist one month later and it was down to 3.2.  I have a 20 year history of prostate infections and my urologist told me eight years ago my prostate was large for my age. I have had to urinate frequently since my late 20's. And, my father went through the exact same scenario when he was younger as well(never had prostate cancer). So, although my urologist said the "literature" indicated a PSA of 2.6 or above would lend itself to a biopsy, he also told he did not believe I have cancer. So, I've been putting the biopsy off based on the fact that my PSA went down almost a full point in one month(& I understand that a fluctuating PSA can be a result of enlarged prostate issues) and my Dad's history of the same thing, my urologist's comments etc.

Any thoughts, comments?