I will assume that you are following the instructions for the color codes on the speaker jacks. But I have two other suggestions to try.
1. On my Realtek system (and many others) there is a special feature designed to allow you to re-assign output ports to different functions. However, it comes with a condition - you MUST follow particular procedures when connecting speakers. If you don't they get poor or non-existent signals and don't work. So, to clear any previous errors and start over, follow this sequence. You will need to set up your system so you can have access to the back panel for plugging in speakers AND be able to see your monitor and use your mouse.
(a) Unplug all three sets of speakers, then reboot your machine. You already have it configured for 5.1 sound, I see.
(b) Plug in ONE speaker pair (say, front L/R) to the correct port. WATCH the screen for a window that pops up. It should say it detected something plugged in and suggest what that might be. It might even be correct. But do NOT ignore that - you MUST answer, either by confirming the suggestion or choosing another. If you don't the window will disappear BUT it will NOT use the default suggestion it had shown you. With no answer, it will send NOTHING out of that jack!
(c) OK, with one set connected properly, repeat for a second speaker set on its port, and answer the window question. Do the same for the third.
(d) Now you should have sound from all your speakers. Probably there is a speaker test demonstration in one of the screens of the Realtek sound manager on your system tray at bottom right.
2. Now, the odd volumes. Maybe that's a problem with the jacks on the back panel. Check the plugs you pushed into them to be sure they are all the way in. Sometimes right-angle plugs get snagged on nearby fame or jack bodies. If that appears OK, look closely inside the Realtek Sound Manager. First, verify that it also shows that you are using 6 speakers for 5.1 sound. Now go to where you can adjust each speaker to achieve left/right balance and front/back balance. You probably need to be playing some 5.1-channel recording for this.Those balance settings may be set wrong. See if you can get the sound working properly there.