git clean is not removing a submodule added to a branch when switching branches

If I add a submodule to a branch and then switch branches, git
checkout warns it cannot
remove the submodule. If I then issue a git clean - it says it removes
the submodule but
in fact does nothing at all. Is this a bug or expected behaviour.

TIA

Adrian

Below are cut&paste steps to reproduce.

git --version
#git version 1.7.8.4

git init submod
cd submod
echo "This is a submodule" > README.txt
git add .
git commit -m "Initial commit"
cd ..
git init prog
cd prog
echo "This is a program" > README.txt
git add .
git commit -a -m "Initial commit"
git checkout -b topic1
git submodule add ../submod
git commit -m "Added submodule"

git checkout master
#warning: unable to rmdir submod: Directory not empty
#Switched to branch 'master'

git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       submod/
#nothing added to commit but untracked files present (use "git add" to track)

git clean -fd
#Removing submod/

git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       submod/
#nothing added to commit but untracked files present (use "git add" to track)