Hi BGS and welcome to ADHDcentral.com!
One of my favorite ADHD experts, Dr. Ned Hallowell, who co-wrote "Driven to Distraction" and authored a lot of other ADHD related books, points out something that might be helpful to you. He said that if one has ADHD, then the best co-morbidity to have is anxiety, because the anxious ADHD adult is more likely to have some control over the clutter and chaos. Now, that's just a paraphrase and he surely didn't say that this makes life any easier, but what he IS saying is that anxiety often helps people with ADHD keep the clutter chaos under control.
I know of many adults with ADHD whose homes are not at all cluttered or disorganized. What I DO tend to see, though, is that they take heroic measures to keep their homes neat because it does not come easy to them. They may spend way more time and energy keeping their homes up. Or they stress over it, possibly as you're describing it, because of the challenges they face in trying to keep the order while struggling hard to do it.
Most with ADHD have problems with disorganization, time management and simply figuring out systems and strategies for making their homes and offices clutter-free. Then there's the whole issue of procrastination and fear of boredom. It's very hard to keep up a house when we often procrastinate on taking care of, say, the laundry, dishes, bed-making, and putting clothes away.
So again, to answer your question...no. Not all people with ADHD have cluttered homes. Like you, they find it visually distracting and stress inducing. They may have anxiety or OCD tendencies that force them to take care of the problem. But in most cases, though their houses are tidy, they more than likely work doubly hard at keeping it that way because it's so difficult to manage when ADHD is part of the mix.
Hope this answers your question.