It's really common with bipolar depression to isolate. I used to do this all the time when I was very sick. It has helped me to recognize that I was doing this and to plan ahead ways to get out of isolation very early on. Here are some of the things on my list.
Call a friend and talk for 5-10 minutes.
Get out of the house, if only for a few minutes to go for a walk.
Attend my regular groups (Toastmasters, church, support group).
Make a business call. (This is the hardest because it is the biggest step into the real world.)
Go swimming and talk to other swimmers in the dressing room.
When I feel like isolating, it helps to do small things that get me out of the house, not all-day things that may break the isolation but exhaust me at the same time. If I act as soon as I notice myself isolating, it usually doesn't take much to break the cycle. If I have been isolating for a long time, it takes more effort to break the pattern.
Isolation is both a symptom and a trigger of depression. When we change the pattern, it can help us get away from depressive thoughts that are easier to think when we are alone. It is also dangerous. if we are feeling suicidal, to isolate. Days of isolation can also add up so that we lose much or all of our social life. We soon start to feel terribly alone with our illness.
Of course, your family is worried about you. Becoming isolated is obviously not what you do when you are feeling well. It would, of course, be nice if they didn't call you "crazy" but that may be the best they can do at this time. You're certainly not "crazy" but you are showing signs of an illness that is serious. They may not know how to respond to this, but perhaps they could help you at these times, if you ask them to do something simple to get unstuck from the isolation.
Don't forget, too, that many people with bipolar disorder have tried to go without medication, and it hasn't been productive in the long haul. You might want to read some of the blogs on this site and see for yourself what others say about medications and their usefulness in getting to feel better.