after having committed folders with lower case naming, I decided to rename them
to upper-case names. Expecting git to detect them as renamings, it started a
new parallel hierarchy with new files, which I had to add/commit.

It was a kinda strange behavior, which I fixed by rename the folder to
something completely different, commit and rename the folder again to the
desired value.

Is this an actual desired behavior or is it a bug?

It is expected (but may be unexpected), I hope these hints are not too harsh:
You can blame the vendor of the OS,
who decided to design a file system which ignores the case of files.
(Or yourself, because you use it)
(Or yourself, because you can install a HFS+ partition under OSX which
is NOT case insensitive, but few people seem to know about this or use it)
(harsh mode off)

The correct way to rename a file under Git is to use Git:

git init
Initialized empty Git repository in /private/tmp/ttt/.git/
user@Mac:/tmp/ttt>mkdir dir1
user@Mac:/tmp/ttt>echo File >dir1/file
user@Mac:/tmp/ttt>git add dir1/file
user@Mac:/tmp/ttt>git commit -m "add dir1/file"
[master (root-commit) 14d3862] add dir1/file
  1 file changed, 1 insertion(+)
  create mode 100644 dir1/file
user@Mac:/tmp/ttt>git mv dir1/file DIR1/FILE
user@Mac:/tmp/ttt>git commit  "mv dir1/file DIR1/FILE"
user@Mac:/tmp/ttt>git ls-files
DIR1/FILE

Robert