Hey Paul,
Is it mainly your thumb that gets tripped up when the strings aren't muted? For a lot of people the thumb getting snagged is a bit of a problem. What I recommend is just going up and down with your thumb (forget about the fingers for a minute) and get a feel for the distance and pressure that works for you on open, unmuted strings when you're just playing with your thumb strumming up and down. This seems to make it a little easier to concentrate on just this aspect so you can learn to regulate it.
In general, learning to really isolate the actual technical problem - in this case the thumb getting caught in the strings in the middle of - is a great way to fix issues. When we leave the issue in its normal context there can be too much 'noise' to really concentrate on the problem and so it takes longer to figure out. The better we get at identifying the particular issue and then isolating it the easier it becomes to resolve stuff as it comes up.
Cheers,
Kai