Disabling ssh-agent was as simple as removing use-ssh-agent from/etc/X11/XSession.options.

To be sure that /etc/X11/Xsession.d/90gpg-agent does the right thing, you need use-agent in $GNUPGHOME/gpg.conf and enable-ssh-support in $GNUPGHOME/gpg-agent.conf.

(FTR, this was done on debian stretch)