This is the same problem I've been having. I've used my XBOX 360 in the same setup you described and it's always worked on my computer with Windows XP. However, I just recently reformatted and switched to Windows 7 and my audio through my line in is really distorted. It seems like the levels are set too high but I can't find any way to turn it down.
However, on my XBOX, all the sound is bad. Even in the dashboard the sound is distorted.
Sorry I don't have an answer for you, but I just wanted to at least say that this setup did work on a system with XP.