Subversion's passphase caching (and its interaction with gpg-agent) might be an issue here too -- I'm not sure exactly which of the many moving parts is the one causing the problem, so I can't say for sure yet.