Testing Throughput on a 100 Mbs Ethernet Line

[knudsen]

We have a bottleneck between two points on our net. It is somewhere between three switches, a dell 5324 and two 3024's. The middle 3024 is acting as a relay becaues we are over 100 Meters between the two outer switches. The middle 3024 has a VLAN setup with just two ports assigned, 5 and 6. All connections copper (CAT5e). I should also state the fact that I set none of this up and can't reach (and don't want to reach) the person who did. So it looks like this:

5324<---------><5>3024<6><--------->3024

Please check my math and troubleshooting thoughts:

The two outer switches are default config with all ports in the default vlan. I have eliminated tham as a cause by moving the CAT5 cable to another port. The middle switch has 4 of these two-port VLANS config'd, so I was easily able to eliminate them as the cause by moving the patch cables to a different VLAN.

Math:

100 Mbs should = 12.5 MBs (divide bits by 8 to get bytes)

I understand in reality there is some overhead and we might see closer to 10 MBs or a little less. Let's be conservative and say 8 for half duplex.

So, if I drag and drop an 8 GB file across this line as a test, it should take 8GB divided by the 8 MBs for a 1000 second transfer time, right? Div by 60 is around 16 minutes.

After 5 and weekends this line is very lightly used. I can't immagine there is 10% of the bandwidth used, so we should be able to get that file across in 20 or 30 minutes, don't you think? I'd even give it an hour and say I'm wrong on the usage, but the file is taking hours and hours. I think there is a serious problem.

There is plenty more that I can try for troubleshooting, including using different lines. I am mainly concerned with my calculations and expectations of a 100 Mbs Ethernet, not how to troubleshoot. But there are a couple of other symptoms I observed I will mention, in case someone says, oh, yup, you have this or that wrong, idiot jon.

One thing is port 5 on the middle 3024 glows yellow. The manual says that means it is running at 10 Mbs, which would explain things. It may well be, and that would probably be bad wiring (I will eliminate or isolate that by substitution Monday night). However, the configuration of the middle 3024 says it's running 100 Mbs half duplex. I believe the configuration I am reading, quoted below, is a reading, or should I say a report of what was autonegotiated by the switch, and not a setting. Am I right? Does anyone know if it is running at 10 as the lamp indicates or 100 as config does? BTW: When I moved the patch cables from port 5 and 6 to an unused VLAN on ports 1 and 2, the yellow light followed the wiring.

Unit 1 Port Manager

Port Link Admin State Rate/Duplex Flow Ctrl Comments
--------------------------------------------------------------------------------
1 Down Enabled Blocking (Auto ) (Auto ) Not Defined
2 Down Enabled Blocking (Auto ) (Auto ) Not Defined
3 Up Enabled Forwarding (100 Full) (Enabled ) Not Defined
4 Up Enabled Forwarding (100 Full) (Enabled ) Not Defined
5 Up Enabled Forwarding (100 Full) (Enabled ) Not Defined
6 Up Enabled Forwarding (100 Half) (Disabled) Not Defined
7 Up Enabled Forwarding (100 Full) (Enabled ) Not Defined
8 Up Enabled Forwarding (100 Full) (Enabled ) Not Defined
9 Down Enabled Blocking (Auto ) (Auto ) Not Defined
10 Down Enabled Blocking (Auto ) (Auto ) Not Defined
11 Down Enabled Blocking (Auto ) (Auto ) Not Defined
12 Down Enabled Blocking (Auto ) (Auto ) Not Defined

<SNIP>

Then on the 3024 on the end, I noticed in the config, all ports are set to auto negotiate, except the port the problematic line leads to, which has been manually set to 100 half duplex. I'm thinking this is an indication that someone saw a problem and was working with it, but gave up. I suppose the setting could cause some sort of a problem, but you should be able to set it up manually without a problem, except the predictable slower simpelx mode. I did try autonegotiate when I moved the line to another port for troubleshooting.

Anyone who read this far, thank you for putting some time and thought into this, regardless of weather or not you are able to understand my mumblings and provide an answer!

Cheers,

jon

 

[jjw]

Could have been a cable issue, or the previous box plugged in at that port had issues.

If it is taking hours and hours, then you are likely running at closer to 10 Mbit.

Try reseting to full duplex or auto, and enabling flow control.

 

[knudsen]

Thanks jjw. I appreciate confirming the math. I have to guess, the wiring is bad and always has been. We have good facilities people who don't realize it's bad to wrap a 100 meter CAT5e cable around a flourescent tube 29 times. Bad thing is the bottleneck is 7 or 8 years old the x-admin ignored it and ignored my advice, which is why he is no more.

I'm hoping to run fiber by the end of the year and GB switches are ordered! The vernable, respected

, but 100 Mbs PIX on the other end of the outer 3024 will soon be replaced with an ASA 5510. Yah, unh-hunh, that's gunna rock!

Anyone have Cisco ASA experiance? OK I admit it, I'm scared.

 

[Madwand]

On the subject of throughput testing and not the problem itself -- file transfer tests are good and fine for lower-end bandwidth, where the drive speeds / etc, aren't much of a factor, and good for application-level tests, but for higher speed networks and clarity, you should go to a pure network bandwidth measurement tool.

You probably already have access to some. If not, TTCP and its Windows port PCATTCP can do the job. This will factor out file transfer protocol overhead, file system / drive bottlenecks, etc., and give you hopefully fairly stable pure network performance numbers with less garbage to clean up -- only the test app to run on both endpoints. IIRC, no installation is required.

When you go to gigabit, you're going to have to go to such tools -- typical HD's will bottleneck and won't tell you what your underlying network is capable of. Assuming of course that you go to gigabit this decade, and HD's and file transfer protocols don't make gigabit trivial by that time

 

[fredweston]

knudsen, sounds like you have some budget to play with. Find a reputable consulting firm in your area and have them bring a Fluke in and run some tests on the various lines. You should be able to get this done for a few hundred bucks, totally worth it considering the time it'll save you. Make sure they can give you a printed report to keep on file.

I'm assuming your switches don't have any fiber or gbic ports. You could always get a pair of media converters for the time being and link them together with singlemode fiber. That will eliminate the need for the relay in between. Plus fiber is nice in the sense that it's immune to EMI. Then when you get your new switches, just put the media converters in the storage closet and plug the fiber directly into the new switches.

 

[knudsen]

Madwand, thanks for the tip on TTCP. I'll definately look forward to playing with that. Gb ether being trivial sounds funny, but so did a hard drive not only exceeding 20 MB, butnot taking up two 5 1/4" bays!

Fred, I think I'll skip consultants at this point as I could have the wire replaced for less than testing it. Good idea though. I do have a meter that can test that the pairs are all paired up right and the cable length. After stepping back from the problem over the weekend, I am pretty confident that it is the wiring. There are supposed to be four wires there, one unused, but I can't find the unused one. We have two seperate DMZ's and one seperate LAN on the three wires in use. DMZ1 is mainly just e-mail and doesn't have to be as fast. DMZ2 has application servers on it and it is on the problem line. I can't unplug them long enough to do much testing or the natives will get restless. I can uplug them long enough to swap DMZ1 and 2 around. I'm 99% sure that will solve the problem from a symptomatic perspective. If not, this is spooky weird.

Fiber is right around the corner! I have a one meter preterminated length and the SFP tranceivers are on order. I am to test the use of it between my 5324's, then order the fiber and tranceivers to eliminate the relay, but more importantly, electrically isolate the four (connected) buildings. We need to replace at least one switch too. Or like you said, get a converter. At somepoint soon the old PIX will get replaced with our new ASA5510. Right now hte PIX would be a 100 Mbs bottleneck as soon as we get these lines up to Gb.

Thank you both for putting some thought into this, for your tips and advice! It is all appreciated.

jon

 

[Madwand]

You're welcome; you seem to have a handle on the problem.

Downgrading to 10 Mb/s is symptomatic of auto negotiation, cabling, and distance problems. Distance can be a big problem -- for long cable runs, some people have even found wireless bridges to give better performance than copper gigabit 8O

To illustrate, I'm getting around 2.1 MB/s throughput on a file transfer here over a consumer wireless bridge -- this would put the entire transfer at a bit more than 1 hour, which is better than your throughput Of course this is a battle of snails; the same file can be transferred at around 75 MB/s (around 110s) using consumer GbE.

Fiber is of course the right solution when distance is the problem, and even in some cases where distance is not the problem, but the connections / cabling are, or electrical grounding, as you point out.

Good luck!

 

[fredweston]

The way you worded your question made it seem like these were runs between buildings and difficult to replace. Testing is always preferable to replacement, at least to me, because it lets me know if I'm going to be wasting time or not. A fluke can tell you so much more about a cable than a regular pinout tester. The latest ones can even give you educated guesses as to the problem I.E. the wire is too near a fluorescent light 37.3 meters away.

 

[knudsen]

Fred, it is dificult to run a new wire, I'm sure, but I don't have to do it or pay for it :twisted: :twisted: :twisted: Facilities dept gets the joy of pulling out all the false ceilings. This run is stretched across 3 of our 4 buildings. The buildings are connected together, but I don't envy the guy who gets that assignment! I'm thinking I can swap DMZ1 and DMZ2 around and it will work well enough until I get the fiber ran through. Fiber should happen this year, so I just need to fix this one problem, even if the solution would normally keep me up at night worrying.

Any of you guys worked with those SFP transceivers that plug into the 5324's for the fiber connection? The two I ordered for testing are from Dell. But I know when I order more, I can get them for half the price elsewhere. Do the other brands work as well? Reliable and compatible and all? I'll need 16 to 24 just for my part of the network, so the $50 ones are a lot more attractive than Dells $100 ones!

 

[fredweston]

I couldn't tell ya, Dell's $100 ones sure sound a lot better than Cisco's $400 ones though! I've stayed away from the generic ones just because Cisco (and I assume Dell) won't support them.

Care to tell how you get cable pulled for free? That's a good trick I'd like to learn!

 

[knudsen]

Oh, it's EZ, Fred, I just put a request in to facilities using our on-line form and some guy comes by and does it :lol: To me it's free if it doesn't come out of my budget.

Yup, I've seen the prices on those range from around $50 up to 400. Some of hte cheap ones are even Ciscos, but Murphys Law says it won't be the one you need. Dells list is closer to $200, but we get a pretty good discount.

 

[knudsen]

Thanks for the ttcp tip. I had it installed on my desktop PC and didn't even know it. Standard Fedora program, I guess. I'm too old and lazy to pick and choose the programs on install, so I just install it all and turn off unused services <G>.

I can't swap lines tonight as we have three exec's working and accounting is trying to close the quarter. But I'm calling this a 10 Mbs bottleneck. Is my math right:

[code:1:759dd0e26b][root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 13.63 real seconds = 1202.27 KB/sec +++
ttcp-r: 5792 I/O calls, msec/call = 2.41, calls/sec = 425.02
ttcp-r: 0.0user 1.1sys 0:13real 8% 0i+0d 0maxrss 0+3pf 5210+0csw
[root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 12.74 real seconds = 1285.91 KB/sec +++
ttcp-r: 6278 I/O calls, msec/call = 2.08, calls/sec = 492.73
ttcp-r: 0.0user 1.0sys 0:12real 8% 0i+0d 0maxrss 0+3pf 5447+1csw
[root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 12.05 real seconds = 1359.58 KB/sec +++
ttcp-r: 5834 I/O calls, msec/call = 2.12, calls/sec = 484.12
ttcp-r: 0.0user 1.1sys 0:12real 9% 0i+0d 0maxrss 0+3pf 5042+2csw
[root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 14.07 real seconds = 1164.16 KB/sec +++
ttcp-r: 5998 I/O calls, msec/call = 2.40, calls/sec = 426.19
ttcp-r: 0.0user 1.0sys 0:14real 7% 0i+0d 0maxrss 0+3pf 5323+0csw
[root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 10.99 real seconds = 1490.38 KB/sec +++
ttcp-r: 5994 I/O calls, msec/call = 1.88, calls/sec = 545.25
ttcp-r: 0.0user 1.0sys 0:11real 10% 0i+0d 0maxrss 0+3pf 5447+1csw
[root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 11.60 real seconds = 1412.31 KB/sec +++
ttcp-r: 5735 I/O calls, msec/call = 2.07, calls/sec = 494.36
ttcp-r: 0.0user 1.0sys 0:11real 9% 0i+0d 0maxrss 0+3pf 5134+1csw
[root@guppy ~]# ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 172.16.1.240
ttcp-r: 16777216 bytes in 12.35 real seconds = 1326.81 KB/sec +++
ttcp-r: 5676 I/O calls, msec/call = 2.23, calls/sec = 459.65
ttcp-r: 0.0user 1.0sys 0:12real 8% 0i+0d 0maxrss 0+3pf 5129+1csw[/code:1:759dd0e26b]

 

[fredweston]

Hmm, that's weird. 10mbps is about 1240kb/s in a perfect world, and you;re exceeding that. So it seems to me that it can't be a 10mbps bottleneck, but rather a severely impaired or overloaded 100mbps link.

 

[knudsen]

Does a 10 Mbs line duplex? So a perfect line would be around 2400 K ????


Only three people were on my end of the building when I did the tests, and one is an e-mail only type user. I ran the test many times, more than what I pasted in, similar results all the time. Never saw under 1000 Kbs. So impared may be true, but overloaded seems unlikely.

Swap lines prob. tomorrow night, and that should prove interesting. 10:1 odds it's wire.

 

[fredweston]

I guess it would duplex to 20mbps, not really sure though. If you go into the switch and do a show int, what does it say for bandwidth? I think it usually says something like 100mbps half/full duplex, but I never paid much attention to the duplex part.

 

[knudsen]

I've never seen a command prompt on either 3024, so I dunno I did plug a serial line in and saw "6 Up Enabled Forwarding (100 Half) (Disabled) Not Defined" for the line with the yellow light. Whole file is quoted in the first post. This line is what initially confused me, as the yellow light indicates 10 Mbs, according the the manual, yet this indicates 100/half duplex.

Half duplex is funny. Doesn't that mean half double? :roll: :? :lol:

I don't recall ever hearing full vs. half duplex when dealing with a 10 Mbs device. Back in the day, when we were all 10 Mbs, we only had a bunch of hubs daisy chained together. So I don't know what to expect. Probably swap lines around tomorrow night.

 

[fredweston]

Well, 1/2(2)=1 so technically it's correct.

 

[knudsen]

Yes, Fred, I agree on this double negative. It is not incorrect :lol: :lol: :roll:

 

[knudsen]

BTW: With our daytime traffic on the suspect line, I'm getting 600 to 700 Kbs.

 

[fredweston]

Something you might want to look at is running MRTG on you pertinent switchports. That'll tell you how much traffic is actually going through them, and you can compare that with the throughput you're getting to see a more complete picture of what's going on.

 

[knudsen]

Can that be run on a 3024? We have it running on the PIX, which is next to the right on the diagram, something like this:

5324<---------><5>3024<6><--------->3024<--------->PIX

Right off the pix:

The 3024 to the right splits the traffic in 24 directions, so the above graph is but only a clue. At the time of this snapshot, I'm getting 625 to 650 KBs.

 

[fredweston]

You should be able to run MRTG against anything that supports SNMP. I have a couple PowerConnect 3324s and I know they support SNMP. I haven't tried using them with MRTG, though.

 

[El0him]

You're numbers are skewed. You are assuming that there is nothing else on your network except those three switches and the two workstations that's sending data to each other, which probably isn't true. You need to unplug and turn off everything in the network except for the require ports. Put a sniffer on your network and you will be surprised at how much traffic you pick up without someone actually doing anything useful.

 

[fredweston]

That's why I recommended MRTG, you can take your test results and the MRTG results and do some math to figure how much bandwidth is being used that isn't part of your test.

 

[El0him]

If you just want to to look at the potential...

http://www.dslreports.com/calculator