Does saying yes make you feel uncomfortable? Why?
And why do your kids not want to go? Is it because it's yuor year, or is there a dynamic at play in the other situation? And is that dynamic a problem or just new and uncomfortable?
If your ex pushes a lot of boundaries, say no.
If your ex is generally okay, and usually pretty good, then I would consider why the kids don't want to go. AND I would consider if I WANT the kids to go.
Personally, in your shoes, I LIKE having my kids on the holidays. I would say no because I want to see the kids.
And don't worry about the grandparents. If you really want to be nice, but still keep the kids, you could always invite them to your house. We have done that before, and it addressed the grandparent frustration without giving up boundaries.