AFAIK, the permissions are just "basic and complete" like that. Those can be refined in the future, of course.

If it helps, if you use a non-web client, you can add people without logging into (and therefore, authorizing) any other servers.