If you're using the web version and frequently browse your following feed, this issue might be hard to resolve. The following feed needs to connect to dozens (or even hundreds) of relays to fetch notes, and browsers have strict connection limits that can interfere with your bunker signer connection.
I'd suggest trying the desktop app instead to see if that improves things.