I have to agree with Diane on two fronts.

Though it's just an impression, because of it being Java, you could have trouble not only because of dependency hell, but because of bundling, with Jitsi.

And I would much rather suggest something that is a de-juri standards, mostly using XMPP, but also SDP and SRTP based.

But I would like to suggest using Mumble, as a kickstart thing. Though I just noticed it uses C++ and some QT dependencies, I suppose those are not as a challenge as some bundled Java libraries. It seems to use protobuf and speex, but it stlil needs its own server. As a positive thing, you can set it ad-hoc and share the connection details out of band, so you won't have some of the issues of federated networks, like credentials and metadata that bad agents could sniff upon. I know this last comment is controversial, but we can start a new thread about it.

At last, I know some folks who have used Mumble for recording a podcast with multiple people, so at least it's good for the purpose at hand.