Now, you're using Git submodules in your setup. This, yes, is a nasty and dangerous thing to deal with.
As for symlinks, heh, yes I do know they are not silly and actually extremely useful and yes I know how to use them. I simply think they are silly for managing dotfiles. If your intent is to version your dotfiles and have different 'profiles' across your systems, using the tools at hand -- well, in theory you can write a bunch of script to create the symlinks and 'manage' them as well -- but I just find /that/ silly.
My approach: git repo with a very simple setup script that creates symlinks to the dotfiles/dotdirs. It also manages some sensitive things like gpg/ssh keys.
So you can have a ~/dotfiles working tree that points to ~. And perhaps a ~/mydata working tree that points to ~ as well. Both will index different files as manually added by the user.