In theory you have enough bandwidth for your games, but games are not the only thing that uses bandwidth. As soon as someone tries to browse some web pages or a game tries to patch itself, the network is going to get flooded.
You should really have at least 10Mb/s, then you can do proper QoS with the correct setup. The reason 10Mb is important is because some packets are larger than others and a 1500 byte packet takes about 1ms to be transmitted at 10Mb/s.
If you control the computers, there are some tricks you could do to make 5Mb seem fast since you have a separate 32Mb/s connection. At least in Windows, you can configure certain programs to mark packets with P802.1p, allowing your firewall/router to potentially use your different...