Check the mag cap that you have to remove to swap barrels. If it's loose it can let the barrel move a bit.
Happened to a friend of mine. We wasted a load of slugs and beat our shoulders to hell before we figured it out.
Bad thing was that it would loosen up after a couple shots.
I’ll check this out. It is certainly worth a try. Realistically I would be happy with a 4 inch group at 100 yards, but I’m far from that