It's generally uncomfortable to share meals with people who aren't eating. It's deeply ingrained in many cultures to "break bread" in a way, so I think that's why it can be perceived as rude.