Getting inconsistent results with TTCP

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Hi,

I've been doing some testing using TTCP (from
http://www.pcausa.com/Utilities/pcattcp.htm) between two identical
servers. Both servers are running Windows 2003, and have GigE
interfaces.

The problem that I'm having is that I am getting very inconsistent
results when running TTCP, even with the same parameters, and with the
same sender and receiver. For example, doing a 2048 packet test (the
default) from server1 to server2, I have seen results from 7 Mbytes/sec
to 114 Mbytes/sec.

I have the feeling that this may be because I'm not waiting long enough
between tests, but I'm not able to confirm this with my testing, so I
was wondering if anyone here has experience with TTCP, and might be able
to tell what the problem is?

I'd also be interested in any other network performance tool that might
possibly do a better job?

Thanks in advance,
Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya wrote:
>
> Hi,
>
> I've been doing some testing using TTCP (from
> http://www.pcausa.com/Utilities/pcattcp.htm) between two identical
> servers. Both servers are running Windows 2003, and have GigE
> interfaces.
>
> The problem that I'm having is that I am getting very inconsistent
> results when running TTCP, even with the same parameters, and with the
> same sender and receiver. For example, doing a 2048 packet test (the
> default) from server1 to server2, I have seen results from 7 Mbytes/sec
> to 114 Mbytes/sec.
>
> I have the feeling that this may be because I'm not waiting long enough
> between tests, but I'm not able to confirm this with my testing, so I
> was wondering if anyone here has experience with TTCP, and might be able
> to tell what the problem is?
>
> I'd also be interested in any other network performance tool that might
> possibly do a better job?
>
> Thanks in advance,
> Jim


Hi,

I forgot to mention that when I run similar tests at home between two
machines with 10/100 NICs, I get very consistent results, regardless how
long I wait between tests. This problem with inconsistency only seem to
happen with these servers with GigE interfaces. Also BTW, these servers
each have 4 Xeon 3.0GHz CPUs.

Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> Hi,

> I've been doing some testing using TTCP (from
> http://www.pcausa.com/Utilities/pcattcp.htm) between two identical
> servers. Both servers are running Windows 2003, and have GigE
> interfaces.

> The problem that I'm having is that I am getting very inconsistent
> results when running TTCP, even with the same parameters, and with the
> same sender and receiver. For example, doing a 2048 packet test (the
> default) from server1 to server2, I have seen results from 7 Mbytes/sec
> to 114 Mbytes/sec.

> I have the feeling that this may be because I'm not waiting long enough
> between tests, but I'm not able to confirm this with my testing, so I
> was wondering if anyone here has experience with TTCP, and might be able
> to tell what the problem is?

> I'd also be interested in any other network performance tool that might
> possibly do a better job?

You could also use another OS that might possibly do a better job.

NetBSD currently have the speed record for long distance gigabit network.

> Thanks in advance,
> Jim

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
>> The problem that I'm having is that I am getting very inconsistent
>> results when running TTCP, even with the same parameters, and with the
>> same sender and receiver. For example, doing a 2048 packet test (the
>> default) from server1 to server2, I have seen results from 7 Mbytes/sec
>> to 114 Mbytes/sec.

Try a larger number of packets -n 2048 is the default and
only transmits 16 MB. This only takes a second or three and
may be within the timing accuracy of the calls ttcpw uses

-- Robert

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Robert Redelmeier wrote:
>
> ohaya <ohaya@cox.net> wrote:
> >> The problem that I'm having is that I am getting very inconsistent
> >> results when running TTCP, even with the same parameters, and with the
> >> same sender and receiver. For example, doing a 2048 packet test (the
> >> default) from server1 to server2, I have seen results from 7 Mbytes/sec
> >> to 114 Mbytes/sec.
>
> Try a larger number of packets -n 2048 is the default and
> only transmits 16 MB. This only takes a second or three and
> may be within the timing accuracy of the calls ttcpw uses


Robert,

Thanks! Using a larger number of packets (100,000), I get much more
consistent results, but at the same time, the results seem relatively
low.

Between two servers, both with GigE interfaces, and connected using a
fiber cross-over cable, I am seeing only about 38-40 Mbytes/sec. Both
servers are identical, with 4 x Xeon 3.0 Ghz CPUs.

I have a much smaller test configuration at home, with two PCs (AMD
Athlon 1.2GHz) with 10/100 NICs, going through an SMC 10/100 switch, and
on this setup, I am seeing ~11 Mbytes/sec consistently

Any ideas as to why I'm not seeing much higher Mbytes/sec than the
40Mbytes/sec with the GigE setup?

Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> Between two servers, both with GigE interfaces, and connected
> using a fiber cross-over cable, I am seeing only about 38-40
> Mbytes/sec. Both servers are identical, with 4 x Xeon 3.0
> Ghz CPUs.

Sounds just about right for PCI cards.

You can easily get wirespeed (11+MByte/s) on 10/100,
but it's very hard to get more than 30 across a plain
32bit/33 MHz PCI bus. Yes, the theoretical bandwidth
_is_ 133 but the burst are short and setup is long.

To get closer to wirespeed, you need 64 bit or 66 MHz
cards and slots. I would think a server board should
have one or two.

-- Robert

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Robert Redelmeier wrote:
>
> ohaya <ohaya@cox.net> wrote:
> > Between two servers, both with GigE interfaces, and connected
> > using a fiber cross-over cable, I am seeing only about 38-40
> > Mbytes/sec. Both servers are identical, with 4 x Xeon 3.0
> > Ghz CPUs.
>
> Sounds just about right for PCI cards.
>
> You can easily get wirespeed (11+MByte/s) on 10/100,
> but it's very hard to get more than 30 across a plain
> 32bit/33 MHz PCI bus. Yes, the theoretical bandwidth
> _is_ 133 but the burst are short and setup is long.
>
> To get closer to wirespeed, you need 64 bit or 66 MHz
> cards and slots. I would think a server board should
> have one or two.


Robert,

I'm pretty sure that the GigE interfaces on these servers are
effectively either 64bit/66MHz PCI or PCI-X. Their GigE interfaces are
"embedded", i.e., they're part of the server system board. Plus, when I
was using the TTCP default of 2048 packets, I was sometimes seeing
~114MBytes/sec results.

Any ideas on this?

Thanks,
Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya wrote:
>
> Robert Redelmeier wrote:
> >
> > ohaya <ohaya@cox.net> wrote:
> > > Between two servers, both with GigE interfaces, and connected
> > > using a fiber cross-over cable, I am seeing only about 38-40
> > > Mbytes/sec. Both servers are identical, with 4 x Xeon 3.0
> > > Ghz CPUs.
> >
> > Sounds just about right for PCI cards.
> >
> > You can easily get wirespeed (11+MByte/s) on 10/100,
> > but it's very hard to get more than 30 across a plain
> > 32bit/33 MHz PCI bus. Yes, the theoretical bandwidth
> > _is_ 133 but the burst are short and setup is long.
> >
> > To get closer to wirespeed, you need 64 bit or 66 MHz
> > cards and slots. I would think a server board should
> > have one or two.
>
> Robert,
>
> I'm pretty sure that the GigE interfaces on these servers are
> effectively either 64bit/66MHz PCI or PCI-X. Their GigE interfaces are
> "embedded", i.e., they're part of the server system board. Plus, when I
> was using the TTCP default of 2048 packets, I was sometimes seeing
> ~114MBytes/sec results.
>
> Any ideas on this?
>
> Thanks,
> Jim


Hi,

I've been continuing to do searching, and I found another tool called
"iperf":

http://dast.nlanr.net/Projects/Iperf/

I've tried it on my 10/100 network setup, and it works ok, giving me
consistent results in the 11 Mbytes/sec or 93 Mbits/sec range. I'll
have to try this on the GigE network and see how it does...

Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> I'm pretty sure that the GigE interfaces on these servers
> are effectively either 64bit/66MHz PCI or PCI-X. Their GigE
> interfaces are "embedded", i.e., they're part of the server
> system board.

Well, solder-on isn't magic. They have to be soldered onto
the right bus! What is the Northbridge on those boards?
Does the board have any higher speed PCI slots?

> Plus, when I was using the TTCP default of 2048
> packets, I was sometimes seeing ~114MBytes/sec results.

Probably timing granularity. Try UDP in case MS-Windows is
still infected with defective TCP recv windows. See if there
is something like jumbo frames you can enable. Or as other
poster suggested, try another OS that might be able to handle
the hardware better.

I presume you're not going through any switch.

-- Robert

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Robert,

My comments/responses interspersed below...

Robert Redelmeier wrote:
>
> ohaya <ohaya@cox.net> wrote:
> > I'm pretty sure that the GigE interfaces on these servers
> > are effectively either 64bit/66MHz PCI or PCI-X. Their GigE
> > interfaces are "embedded", i.e., they're part of the server
> > system board.
>
> Well, solder-on isn't magic. They have to be soldered onto
> the right bus! What is the Northbridge on those boards?
> Does the board have any higher speed PCI slots?

These servers are IBM server, so I think they use the "Champion" (or
something like that) chipset.

The board has 2 PCI-X expansion slots.




> > Plus, when I was using the TTCP default of 2048
> > packets, I was sometimes seeing ~114MBytes/sec results.
>
> Probably timing granularity. Try UDP in case MS-Windows is
> still infected with defective TCP recv windows. See if there
> is something like jumbo frames you can enable. Or as other
> poster suggested, try another OS that might be able to handle
> the hardware better.


Ok, thanks re. the UDP comment. I'll give that a shot. As I also
mentioned in my previous post, I'm going to try 'iperf'.

Re. "jumbo frames": The embedded GigE interfaces are Intel 1000
interfaces. I think, but am not sure, that I can enable jumbo frames
under Network->Properties. For Windows 2003, is that all I need to do
to enable jumble frames, or do I also have to do any registry editting
for MTU, etc.?

Re. the OS, unfortunately a different OS is not an option, as part of
what I'm trying to determine is what kind of performance we can get over
the GigE under Windows 2003.



> I presume you're not going through any switch.


That (switch vs. no-switch performance) is one of the other "parameters"
that I'm testing, i.e., I will eventually want to get two sets of
performance numbers:

- Going through a GigE switch that we have, and
- Going point-to-point over a fiber cross-over cable.

Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> The board has 2 PCI-X expansion slots.

Then it should have the GigE wired in for full capability.

> Re. "jumbo frames": The embedded GigE interfaces are Intel
> 1000 interfaces. I think, but am not sure, that I can enable
> jumbo frames under Network->Properties. For Windows 2003,
> is that all I need to do to enable jumble frames, or do I
> also have to do any registry editting for MTU, etc.?

I try not to know anything about MS-Windows.
I would hope that seeting jumbo frames would
set the MTU, but who knows. UDP '-u' gets
around the TCP issues.

> Re. the OS, unfortunately a different OS is not an option,
> as part of what I'm trying to determine is what kind of
> performance we can get over the GigE under Windows 2003.

Then perhaps you should look for specific tuning
tips for MS-Win2003.

> That (switch vs. no-switch performance) is one of the other
> "parameters" that I'm testing, i.e., I will eventually want
> to get two sets of performance numbers:

> - Going through a GigE switch that we have, and - Going
> point-to-point over a fiber cross-over cable.

It will be very odd if the switch beats crossover.

-- Robert

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

> > That (switch vs. no-switch performance) is one of the other
> > "parameters" that I'm testing, i.e., I will eventually want
> > to get two sets of performance numbers:
>
> > - Going through a GigE switch that we have, and - Going
> > point-to-point over a fiber cross-over cable.
>
> It will be very odd if the switch beats crossover.


Robert,

Yes, I'm assuming that will be the case, but with my testing, I am
hoping to get some idea how much slowere the switch will be vs.
crossover.

Thanks,
Jim

P.S. If anyone else knows if I have to do anything besides set the
"Jumbo Frame" setting in the Network properties under Windows 2003,
please post! Thanks...

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> Robert,

> My comments/responses interspersed below...

> Robert Redelmeier wrote:
>>
>> ohaya <ohaya@cox.net> wrote:
>> > I'm pretty sure that the GigE interfaces on these servers
>> > are effectively either 64bit/66MHz PCI or PCI-X. Their GigE
>> > interfaces are "embedded", i.e., they're part of the server
>> > system board.
>>
>> Well, solder-on isn't magic. They have to be soldered onto
>> the right bus! What is the Northbridge on those boards?
>> Does the board have any higher speed PCI slots?

> These servers are IBM server, so I think they use the "Champion" (or
> something like that) chipset.

> The board has 2 PCI-X expansion slots.



>
>> > Plus, when I was using the TTCP default of 2048
>> > packets, I was sometimes seeing ~114MBytes/sec results.
>>
>> Probably timing granularity. Try UDP in case MS-Windows is
>> still infected with defective TCP recv windows. See if there
>> is something like jumbo frames you can enable. Or as other
>> poster suggested, try another OS that might be able to handle
>> the hardware better.


> Ok, thanks re. the UDP comment. I'll give that a shot. As I also
> mentioned in my previous post, I'm going to try 'iperf'.

> Re. "jumbo frames": The embedded GigE interfaces are Intel 1000
> interfaces. I think, but am not sure, that I can enable jumbo frames
> under Network->Properties. For Windows 2003, is that all I need to do
> to enable jumble frames, or do I also have to do any registry editting
> for MTU, etc.?

> Re. the OS, unfortunately a different OS is not an option, as part of
> what I'm trying to determine is what kind of performance we can get over
> the GigE under Windows 2003.

Well, even if you cannot use another OS "in vitrio" you should exmine
where the limits are, in order to put some pressure on your lucky vendor.

>> I presume you're not going through any switch.


> That (switch vs. no-switch performance) is one of the other "parameters"
> that I'm testing, i.e., I will eventually want to get two sets of
> performance numbers:

> - Going through a GigE switch that we have, and

Brand of swith _is_ importent, so is some tunebals in them.
Or is switch brand also predefined ?

> - Going point-to-point over a fiber cross-over cable.

> Jim

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

> > - Going through a GigE switch that we have, and
>
> Brand of swith _is_ importent, so is some tunebals in them.
> Or is switch brand also predefined ?


Peter,

I'm not sure who the actual manufacturer of the switch is, it's an "IBM"
switch, but I believe the for their servers, it's re-branded by IBM
(i.e., not made by them). Regardless, that (the IBM) switch will be the
only one we can use in this case (as you surmised).

Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:

>> > - Going through a GigE switch that we have, and
>>
>> Brand of swith _is_ importent, so is some tunebals in them.
>> Or is switch brand also predefined ?


> Peter,

> I'm not sure who the actual manufacturer of the switch is, it's an "IBM"
> switch, but I believe the for their servers, it's re-branded by IBM
> (i.e., not made by them). Regardless, that (the IBM) switch will be the
> only one we can use in this case (as you surmised).

> Jim


Why do we even bother since everything already is decided ??

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> P.S. If anyone else knows if I have to do anything besides
> set the "Jumbo Frame" setting in the Network properties
> under Windows 2003, please post! Thanks...

Try the UDP. It will tell you if you need to fix
the TCP windows sizes.

-- Robert

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

> > Peter,
>
> > I'm not sure who the actual manufacturer of the switch is, it's an "IBM"
> > switch, but I believe the for their servers, it's re-branded by IBM
> > (i.e., not made by them). Regardless, that (the IBM) switch will be the
> > only one we can use in this case (as you surmised).
>
> > Jim
>
> Why do we even bother since everything already is decided ??



Peter,

What got me started on this (wanting to measure bandwidth on the GigE
connections) is that we are considering booting the servers via GigE vs.
some other options. As I'm looking into this, I found somewhat low
rates when doing the boot (~16Mbytes/sec), so I started looking into the
underlying performance of the GigE connection itself.

Jim

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

ohaya <ohaya@cox.net> wrote:
> What got me started on this (wanting to measure bandwidth on the GigE
> connections) is that we are considering booting the servers via GigE vs.
> some other options. As I'm looking into this, I found somewhat low
> rates when doing the boot (~16Mbytes/sec), so I started looking into the
> underlying performance of the GigE connection itself.

Boot protocols are (often) request/response - for example TFTP. As
such, they can be very latency sensitive and 16 MB/s on GigE may not
be all that bad. What protocol is used to load the boot image(s)?

If you want to "simulate" TFTP or a request/response sort of
situation, you might consider something like a netperf UDP_RR test
with "apropriate" request and response sizes. Or TCP_RR. It will
report transactions per second, but you can multiply that by the
request or response size to get a bandwidth.

rick jones
--
denial, anger, bargaining, depression, acceptance, rebirth...
where do you want to be today?
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Robert Redelmeier wrote:
>
> ohaya <ohaya@cox.net> wrote:
> > P.S. If anyone else knows if I have to do anything besides
> > set the "Jumbo Frame" setting in the Network properties
> > under Windows 2003, please post! Thanks...
>
> Try the UDP. It will tell you if you need to fix
> the TCP windows sizes.
>
> -- Robert


Hi Robert,

I just tried UDP instead of TCP/IP, with both Iperf and Netperf, and
still got ~300 Mbits/sec. Seems like that (the 300 Mbits/sec) must be
about right, at least without any tweaking (which is my next step).

Jim