You can try reversing the two rear fans so that they pull air out, because you're pushing air in from the front, rear, and side and only pulling out air from 1 80mm fan.
As for having too many fans, it shouldn't matter how many you have, but how good your airflow is. I have an Antec 902 case, so I'm using 2 120mm in the front as intake, 1 120mm in the rear and 1 200mm on the top as exhaust. Aside from that, I my PSU from the bottom has a 120mm intake fan, my hsf has a 120mm fan for the heatsink, and my GPU has a 90mm fan. So I have 7 fans in total and that's without adding any extra fans or mods. My case allows me for another 120mm on the side window and one 120mm inside the HDD cage and my heatsink allows for another 3 120mm fans to be attached on it, so I can get up to 12 fans.