To nit-pick a tad more, when they have access to my key and have my passphrase so they can sign with it…

That’s why you set the passphrase on keys, gpg, ssh,… Never use a encryption without a key. That way you need posession (key) and knowledge (passphrase) to identify yourself. When you use ssh, use the ssh agent, when you have automated login which would be better to use without keyphrase, use a different pair (specify wuth -i option) and limit access with that to a fixed ip.

And always protect your key. No cloud backup…