True, the Shadow IS partly fears, but it's also desires. Everything the Shadow feels is also part of what its analogue feels (as has been made abundantly clear), even if they won't necessarily admit to it at first.

And let's face it, Kanji's Shadow was acting a lot more like it represented desires than fears throughout most of that dungeon. XD