Turns out liboggz's seeking doesn't seem to be optimized to use ogg skeleton data anyway. :) Managed to find the well-hidden 'libskeleton' -- https://git.xiph.org/?p=libskeleton.git;a=summary which actually parses this data and includes a function to give you the last-keyframe offset for a given time position. Heyyyyy sounds handy!

(Note it doesn't clone over https. Bug filed.)

(And it doesn't compile with clang. One-line fix patch submitted.)