In my opinion (which may not be worth much) the follower has a tougher job than the leader. Yes, the man (leader) has to think of what to do next, know how to dance the figure, and lead his partner. But the lady (follower) has to be able to "read" what may be some pretty subtle cues, and she has to know the figures as well--unless she is a VERY good follower, it's next to impossible for her to dance some of the more involved figures "cold."
My regular partner is my wife, even after quite a few years of dancing. Yes, we have our disagreements. She is a more "intuitive" dancer than I am, and she gets frustrated sometimes when I don't pick up something new as fast as she does, or when I can't figure out what pattern she would like to do (if she suggests one) from her description. We also probably know each other's mistakes and compensate for them. But we still enjoy dancing together. That doesn't mean that we shouldn't dance with other partners, or that you shouldn't. I'm glad you say you still enjoy dancing with your wife.