Assistance with Underperforming 10GB Network

drhlis

Honorable
Jan 31, 2015
7
0
10,510
What can be causing my network to run at such slow data transfer speeds? Could it be the Cisco RV042 Router?

Router: Cisco RV042
Switch: Netgear XS712T 10GB 12-Port;
Server: Two (2) Intel Xeon E5620 @2.4GHz CPUs w/24GB RAM running on Windows 7 Professional 64-Bit, Intel SSD 480GB, SATA2, RAID 1
Network Card: Intel X540-T2 Ethernet Converged 10GB Dual Port Network Adapter
PCIe v2.1 (Each 10GB Port has been Link Aggregated (Teamed) on the X540-T2 Network Adapter and XS712T Switch.
--Flow Control = Rx & Tx is "Enabled"
--Interrupt Moderation Rate = "Off"
--Low Latency Interrupts = "None"
--Receive Buffers = "2048"
--Transmit Buffers = "2048"

Test xfer file: Windows 8.1 iso (size = 3.55GB)
Test file is copied from Server to Workstation with initial xfers speed of 500MB/sec but then it drops down to 300 MB/sec

Workstation: Intel(R) Core(TM)2 Duo T9500 @2.6MHz w/16GB RAM running on Windows 8.1 Pro 64-Bit, Samsung SSD 850 Pro 1TB, SATA2
Network Card: Intel X540-T2 Ethernet Converged 10GB Dual Port Network Adapter
PCIe v2.1 (Each 10GB Port has been Link Aggregated (Teamed) on the X540-T2 Network Adapter and XS712T Switch.
--Flow Control = Rx & Tx is "Enabled"
--Interrupt Moderation Rate = "Off"
--Low Latency Interrupts = "None"
--Receive Buffers = "2048"
--Transmit Buffers = "2048"

Test xfer file: Windows 8.1 iso (size = 3.55GB)
Test file is copied from Workstation to Server with a consistent xfer speed of 207MB/sec for the entire file.

I have two other workstations equipped with StarTech 10GB single port PCIe Network card which are also connected to the Netgear XS712T switch.

There is a Netgear G724T 24-Port 1GB switch that is daisy chained to the XS712T 10GB switch. The are approximately 12 workstations connected to the G724T 1GB switch. Each workstation has a PCIe 1GB network card.

 
Solution
SATA2 is only 3Gb/s, so 300MB/s sounds about right. As for the initial burst of 500MB/s, if you installed Samsung software, their newer drives support caching data in memory prior to writing to the SSD.

I see you turned off Interrupt Moderation. What is your kernel time and total CPU time while doing the transfer? CPU issues are common when you're talking about 10Gb rates. A Core2 Duo is a pretty old CPU design, there has been a lot of changes to modern CPUs to handle interrupts much better because of issues caused by 10Gb+ networks.

My guess is your issue is because your computers are old and you're hitting a mixture of CPU and SATA bottlenecks.

Kewlx25

Distinguished
SATA2 is only 3Gb/s, so 300MB/s sounds about right. As for the initial burst of 500MB/s, if you installed Samsung software, their newer drives support caching data in memory prior to writing to the SSD.

I see you turned off Interrupt Moderation. What is your kernel time and total CPU time while doing the transfer? CPU issues are common when you're talking about 10Gb rates. A Core2 Duo is a pretty old CPU design, there has been a lot of changes to modern CPUs to handle interrupts much better because of issues caused by 10Gb+ networks.

My guess is your issue is because your computers are old and you're hitting a mixture of CPU and SATA bottlenecks.
 
Solution

Kewlx25

Distinguished
I noticed flow control is on. The way flow control works is when a link anywhere on your Layer 2 network reaches capacity, it will send back a Ethernet frame to the seconding MAC address, and tell it to stop. This does not just stop the current transfer, it stops ALL traffic from your network card.

This means if your 10Gb NIC starts sending data to a device with a 100Mb port, your 10Gb nic may suddenly turn into 100Mb for all devices. Not entirely that simple, but similar.
 

drhlis

Honorable
Jan 31, 2015
7
0
10,510


Yes, my computers are old. I intend to make upgrades one workstation at a time when funding is available. I am unsure how to obtain kernal & CPU time. I have been out of the loop practicing optometry for 20 years. I resigned from McDonnell Douglas Space Company 25 years ago as an EE.

I have two other workstations in exam rooms 1 & 2 which have a StarTech 10GB Single Port Network adapter card which is realizing transfer speed of 150MB/s compared to 13MB/sec. There are 11 workstations, a NAS , server and four employees who are on the network at one time.

I also maxed out the Receive Buffers to 4k and Transmit Buffers to 16k which seemed to increased the xfer speed by a few more MB/sec.

Thank you for answering my concern.