problem with network over different switches

I have the following set of equipment:

Dlink DGL-4100 x1 (router)
Dlink DGS-1008D x2 (Switch A/B)

and connected as following:

DGS-1008D(Switch A)<->DGL-4100<->DGS-1008D(Switch B)

and the router connects to the cable modem/internet. All the cabling done from switch to computers are Cat5e and the cabling between switches are Cat6. No other computers are connected to the router (all computers are connected through either Switch A or Switch B)

The problem I am having is whenever a computer at switch A is taking ~4-10KB/s total bandwidth (2-5KB/s UL+2-5KB/s DL) the computers at switch B would have latency issues but not the other computers at switch A (the latency issues would happen at computers connected to switch A if similar bandwidth is taken at switch B)

I am sure I am missing something here, so any help would be very appreciated.

I believe those switches support flow control. See if your pc's nic card does to. If so turn it on an see of it helps. It's located under the advanced properties of the ports driver. Most default to off, and not all support it.