The only way to get true surround sound output through a digital connector on a sound card is by sending it as a Dolby Digital or DTS stream -- there's no way to get uncompressed surround sound signals through a digital cable (I think the bandwidth is insufficient, but at the very least, there's no standard that allows it). Thus, if you enable digital output on your sound card and set it to 2 channels, you should get sound, but it won't be 5.1. To respond to allegations that PC digital output is incompatible with home theater hardware, this is untrue. If you're watching a movie with a surround sound track, you should be able to set your player software to a pass-through mode where it sends the full 5.1 sound through the digital output. In other arenas, well, most sound cards don't encode to DD on the fly, so you're stuck with stereo in games if you use the digital output.
What this means is that the ONLY way to get true surround sound in games is to connect via analog audio cables. You'll need to use all 3 mini stereo jacks on your sound card (I'm assuming the card does support 5.1 in the first place), so that means 6 RCA plugs on the other end. Ara's warning about bass is more or less correct as well. My receiver (Yamaha HTR-5790, a reasonably high-end model) automatically redirects bass received in any channel to my subwoofer, but I believe many other receivers won't do this, especially for the 6 channel analog input. You'll probably need to force the computer to do the redirection.
After re-reading your post, it appears that you have a surround sound system intended for TV use, with only two RCA plugs. One telltale sign of these systems is when the inputs are on the same box that plays DVDs. I installed one of these for a friend once and didn't believe my eyes, because there's absolutely no way to get surround sound to the system from a PC (except in the aforementioned case when you're watching a movie in surround sound). When you used your regular audio output, the sound you heard from the rear speakers would have been a cheap upmix trick performed by the speaker system, which would explain the low volume in the rear speakers. My advice is to return your crippled "5.1" system and build one from components using a real home theater receiver. Either that, or you may be able to find a prepackaged system with 6 channel analog inputs.
-Brian