You make perfect sense to me... I have also battled these feelings in the past. The first few years were the worst for me regarding this topic. When the ex moved out to be with the OW, he participated in visitation pretty regularly the first year. Of course, I had no idea the OW existed at the time, as he swore it had nothing to do with anyone else and the kids never met anyone when with him. I didn't find out she existed and was a co-worker of his in the NYC office, until he took her home to his family's for Christmas, just 6 months after we separated. Then the truth came spilling out.
The second year of our separation/divorce preparations, he slowly backed away from visitation, until he was not seeing them at all. I realize now that it was because he was traveling to her in NYC, or she down here, so they could see each other. The kids were not his priority list at all, only own selfish needs mattered.
The third year he was still living here, but had no visitation at all. I remember talking to my lawyer about this, asking if he couldn't be forced to participate in visitation and she said "No, you can't force someone to be a good parent, you can only make sure they support their children financially". I was livid! At the time my mom had just had two strokes, my oldest son was going through intense problems, I was in college and he would not see or help me with the kids. I couldn't understand why he was allowed to not take any responsibility in raising them too. I desperately needed help, needed some time to take care of others things and just a little break too. Never got it!
So, I understand and relate. It is extremely frustrating, aggravating, and unfair. The thing that got me through it is remembering that I at least wasn't a parent squabbling with the other parent over custody. That would have broken my heart to go through ever thinking I might loose my kids in all too. I was very, very, very thankful there was no custody battles.
As for people keeping their vows... I think what happens in divorce cases is one person ends up being more committed than the other, loves the other more they are loved back and the person who leaves never took their vows seriously. The marriages that last are committed, two way partnerships. They may not always get along, may not always even like each other, but they honor their commitments and they work it out. Sadly, if we are here on the board, we didn't have that type of a relationship with our spouses. If we did, it was only because we were the ones willing to make it work, do whatever it took, they just weren't willing in return. It's very sad, but true.
"Life isn't about waiting for the storm to pass...it's about learning to dance in the rain."