git submodules and custom work tree / git dir

I am working on a setup for managing my personal config files 
(shell settings and vim settings and such) using git. 

I have a git repository which ignores all files by default, so they don't 
cloud status. I have the following aliases:
alias config='git --git-dir=$HOME/.config.git/ --work-tree=$HOME'
alias config-add='git --git-dir=$HOME/.config.git --work-tree=$HOME add -f'

This works pretty well. It means that the config repo is separate from other 
git repos, and I won't accidentally touch it by just typing "git."
What I'm trying to do now is to track vim (bundle) modules inside this git 
repo, which requires the use of git submodules.
It appears that submodules don't work with this weird configuration.

By calling git as in the "config" alias above, except removing the --work-tree 
argument, I was able to add submodules successfully with git submodule add. The
submodule info is written into .gitmodules and .config.git/config as expected. 
However,  the submodule doesn't seem to "stick." It displays when I type  
"config status" as "untracked content" no matter what I do, even though 
there is no untracked content and no change in the submodule.

Is there a way to make submodules work reasonably with git when using the 
--work-tree and --git-dir arguments?