Lila, all the things you say make sense and are well-reasoned---and I love your PeeWee/Brigadoon references---but I still disagree. I think Grandpa acts too entitled, and this pre-dates his relationship with his girlfriend. If he doesn't like the way his children act, as their father he's responsible, to some extent. (Even though they're all over 60 now.) Just sitting back and acting annoyed by them, ignoring their needs and so forth, isn't very nice. They're a family, after all.
It's not like one of us is right and the other is wrong, on the other hand. There's room for both Grandpa-approval and Grandpa-disapproval here.