if that were the case, then wouldn't i not be having issues with my current setup where the V-Portal and the DGL are fed from the gigabit switch?
No. there is no way for either the v-portal or the DGL to prioritize traffic when they both come off the switch. Any QoS settings would have to be done on the switch config.
By running it from the modem -> V-Portal -> DGL all network traffic must pass through the v-portal. The v-portal should handle QoS during calls. The DGL doesn't need any configuration at all.