Sounds very similar to what happens with NVCS on RB25DETs, which is a very similar system. I've tuned one "totally" with VCT on and off to get ensure optimal results either way and when overlayed the difference was really noticeable:
Torque only really fell below where it was with NVCS somewhere decently past 5000rpm from memory - and again, with RB25s a LOT of people don't bother at all with playing with valve timing... when an engine is designed to utilise it, its a huge sacrifice in low down/midrange torque for no gain if you don't use it. Partly due to the common laziness in this area RB25s get a bit of a dud rap in regards to their turbo spooling/torque.
If you have VCT from factory, and you haven't massively changed the breathing/combusion behaviour of the engine from factory then IMHO having adjustable cam gears is definitely not a good replacement. At least with the RB, getting an adjustable exhaust gear and leaving the variable intake cam system can result in awesome power delivery