Attach the speakers to the xbox instead of to the computer?
Ummm.... I don't think connecting the audio out from the xbox to the mic port would work. I can't say for 100% though. If you play the xbox and you don't hear anything from the speakers though, then you will know 100% that that will work or not.