Possible to use git over custom ssh libraries?

I'm working on a set of network services using SSH, like SFTP, backup,
fsnotify, (text/video) chat, connection forwarding using my own SSH
implementation. See:

https://github.com/stefbon/OSNS

Now I want to also provide support for browsing remote git
repositories using my ssh libraries, which will provide the transport
layer.

Is it possible to use git in combination with my own libraries? Or do
I have to use libgit2 for that or something else?

Thanks in advance,

Stef Bon