sub mesa :
You're probably using PCI as interface, PCI has high interrupt CPU usage so lots will go to waste. You need a PCIe adapter. It could also be bad drivers. Some drivers can use polling as interrupt method, which would lower the CPU utilization caused by interrupts.
Are you using gigabit networking, and how did you test the speeds?
sub mesa, this is an interesting piece of info. i have two test machine sets; one set with a pcie-pcie gig-e interfaces and one with a pci-pcie interface. i am working on a setup where the pci-pcie connection will lose packets (about 5-10%). where the pcie-pcie connection will have a packet loss of less than 0.00001%. for the pci-pcie conneton, i have tried 3 other pci gig-e cards and they all produce the same packet loss (about 5-10%).
these are controlled tests with no changes other then the pci cards. the only pcie gige cards i currently have are broadcom BCM5755 chipset based. the pci gige cards vary.
anyway, the point of this post. do you know where i could start looking up hard documentation for why the pci cards...umm..."suck" compared to the pcie cards? i need to find some soild justifications why this happens. ive looked around but dont think im searching the right words. do you know any keywords i should try?