Are height/weight and body fat charts always accurate?

I am a 38 year old male.  I look and feel fit and my doctor says that my weight is actually on the lower side of where he thinks it should be.  People have commented that they think I look a little TOO THIN since I've recently lost some weight.  Yet according to every height/weight/body fat chart I read, I am considered obese - anywhere from 30 - 40 lbs overweight.  My waist size is around 31/32.  If I lost 30 lbs I would look like like a refugee from a concentration camp.  The numbers on those charts are completely unattainable and unrealistic for me.  What is wrong with me???