Problems after deleting submodule

I included a submodule in my project then decided I didn't like
submodules and deleted it again. I followed the advice of delting
.gitmodules, the bit from .git/config and then git rm'ing the
submodule. Seemed to work. I then copied files with the same directory
name into where the submodule was. However, I can't add them.

Doing git add /path/to/old/submodule - does nothing, files are not
staged, no error messages no nothing.
If I try to git rm /path/to/old/submodule - it just says 'did not
match any files'.

It simply does not seem to want to add anything to the old submodule
location. I had a grep around and could not see any obvious references
in the repo.

I'm a bit stuck... any suggestions for things I can try much appreciated.