There can be many factors but...
If my guess is right, you have packet collisions problem due to broadcast packets. Making things short, in your switch network, PCs send out broadcast packets that go out to all switch ports except the one originally came in. The more PCs you have the more broadcast it generates. The amount depends on application in use too, your game for i.e.
Primary solution is to divide the network into multiple collision domains with the use of VLANs. You need VLAN capable switch. Now it works like this: when a broadcast comes in, it only goes out to ports that are in the same VLAN group instead of all ports, reducing collisions. When you set up VLAN, you also need to set up inter-VLAN communication/routing/forwarding. This inter-VLAN routing can be done by
1) the layer 3 VLAN capable switch OR
2) router-on-a-stick scenario
I know some work involved but when you do it, you'll learn a lot of new things. VLANs are not layer-3 concept.