Would you go on vaction if?

sorry so long! me and my husband are having money trouble and we seldom get to have any fun outside of taking our four kids to the park every other weekend. my husband works and i stay at home with our children ages 7,2,1 and 3 months old; the problem is my sister wants me to go on vacation with her and her husband to california and but my husband has no vacation time and cant go. my mom has offered to watch our kids if i decide to go but i feel soooooo guilty that he wouldnt be able to go. he said it would be ok for me to go but i can tell that he would be hurt. he swears that he wants me to go and have fun. what should i do?