What was the father's explanation to his son about the way he sat in the car and watched his wife in all those places? "If she saw me, my family would accept our son"?? or my family would take him back? I didn't understand that, but it made a big difference in how the son felt about his father. It seemed that he felt sympathy for him and reversed his mental image of the past.
The way I understood it was the parents got divorced and the mother got custody of the son, but only if she agreed never to see the father again. If that happened, the family had the right to reclaim the son.
So his father, by not seeing the mother (which meant not seeing the son either) was doing them a favor. Up until his explanation, I think the son had always felt he betrayed and/or abandoned them.