Presenting the children factor is a nice way of saying, "I'm prioritizing in my life and you or this job are not a priority" which is fine. Because that's what it means to make the decision to have children, they have to come first. Just make your children the priority and so "no." It means missing out on something, but that's the nature of prioritizing....do it guilt-free and you, the kids and us single folks will appreciate it a lot more....
We give up one thing to have another. That's sort of the balance of the universe.