Does anyone else have lags while playing with a friend on xbox?

Since this Season started my friend and I have a big problem, whenever I join his instance my ping spikes crazy between 50ms and 1000ms. When I enter the next zone before him, my pings is a stable 16ms and everything is fine. But as soon as he enters a Zone before me the same thing happens again. We literally only can play on my instance because of that, its really frustrating. Anyone else have this issues? Are there fixes for it? Thank you for the help :)