I'm pretty certain it's the Xbox. Did everyone port forward their router? I've never had issues with my group. Secondly, Xbox likes to switch you game chat, or party chat randomly. If you can't hear, or can't hear someone, check those options first. Thirdly, are you in a clan?