Simply put, yes, you should be concerned.
High blood pressure, or hypertension, is a widespread and serious illness. The condition predisposes you many health problems, including (but not limited to!) heart disease, kidney disease, and strokes. You should aim for a blood pressure of 120/80, but at least under 140/90.
You should consider several things about controlling your hypertension. First, weight loss and exercise can reduce blood pressure significantly; as little as a 30 minute walk four times a week will help lower your numbers. Also, there are many different medications that can be prescribed that can help better control your blood pressure. They mostly all very safe, and have few side effects; and of course, if you encounter a side effect of a medication, your doctor can tailor your regiment to suit your needs.
I would keep a BP diary for the next week and take it, if possible, twice a day. Make an appointment with your physician for the following week and bring your readings. If your numbers are above the goals I listed above and doctor remains unconcerned, YOU NEED TO FIND ANOTHER DOCTOR! High blood pressure is a supremely treatable condition, and lowering yours to a normal pressure should help you live a healthier, longer life.
To your health,