concerns about jumbo frames

[Guest]

Archived from groups: comp.dcom.lans.ethernet,comp.os.linux.networking (More info?)

I'm considering a switch of our RHEL3 fileserver to use jumbo frames
in an attempt to improve Samba and NFS performance. Our entire LAN
is gigabit, and connected via switches that support jumbo frames. So
I expect that switching shouldn't be a problem. But I have a few
concerns:

Since our server will also need to communicate with the outside world,
I need to make sure that PMTU discovery can't break. I understand
that it should get back an ICMP fragmentation-needed packet if a
packet encounters a link with a smaller MTU. But how reliable is
this? Are there routers that don't send those back? (I'd expect it
to break even the standard 1500-byte MTU if there were.) Is this
something to worry about?

Also, I'm curious if anything needs to be done to get NFS to take
advantage of the larger MTU. According to the RFCs, you shouldn't
send an packet larger than 576 bytes without knowledge that the
receiving end can handle it. In TCP, the exchange of MTU information
is handled during the initial handshake. But if NFS is being sent
over UDP, how would it know it can use larger packets?

Finally, I've seen the arguments against jumbo frames (the percentage
increase isn't worth it) and suggestions to try other things (like
change the window size or acking procedures). Any specific advice on
things like that would be appreciated. Given that this is a
production server, I won't want to try anything too unusual, but if
there are fairly standard/simple changes then I wouldn't mind giving
them a go.

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Damian Menscher <menscher+usenet@uiuc.edu> wrote:
> Since our server will also need to communicate with the outside world,
> I need to make sure that PMTU discovery can't break. I understand
> that it should get back an ICMP fragmentation-needed packet if a
> packet encounters a link with a smaller MTU. But how reliable is
> this? Are there routers that don't send those back? (I'd expect it
> to break even the standard 1500-byte MTU if there were.) Is this
> something to worry about?

All you need to do is make sure that your gateway router a) enforces a 1500
mtu on outgoing traffic and b) supports PMTU properly. Then the jumbo
frames will stop at the door.

--
Bruce

A problem shared brings the consolation that someone else is now
feeling as miserable as you.

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

In article <cnr1cv$54e$1@news.ks.uiuc.edu>,
Damian Menscher <menscher+usenet@uiuc.edu> wrote:
:Since our server will also need to communicate with the outside world,
:I need to make sure that PMTU discovery can't break. I understand
:that it should get back an ICMP fragmentation-needed packet if a
acket encounters a link with a smaller MTU. But how reliable is
:this? Are there routers that don't send those back? (I'd expect it
>to break even the standard 1500-byte MTU if there were.)

There are, and they do.

There few routers that -cannot- send back the appropriate ICMPs, but
there is an unfortunate trend to turn the ICMPs off or to filter
them out.

The reasons vary, I'm sure, but one of the reasons is that
people are tending to believe that their firewalls or routers should be
"invisible to the outside world", so they block outbound icmp. This is
of course nothing more than "security through obscurity", but over
the course of several years of working in the Cisco newsgroups, I
can only recall one time that someone said "Oops, I didn't think about
that" and agreed to abandon the idea of "invisibility".

Sometimes the problem is just that people haven't heard about PMTU
Discovery, or that it has slipped their mind. If it's just slipped
their mind then they usually turn on the icmp as soon as they are
reminded, but if they haven't heard about PMTU Discovery then it can be
a Problem to convince them that it's a real problem -- *they* have
never experienced the problem and so to them the problem must not exist
or else must academic (sort of like being told that there is still a
law on the books saying that all "horseless carriages" must be
proceeded by a person on foot waving a red flag.")


:Is this
:something to worry about?

In your situation, I'm not sure it's worth worrying about. I would
expect that your border routers are likely to reject the jumbo
packets before they get onto the Internet [or Internet II].
In the more general case, though, it continues to be a problem that
plagues many people: ADSL providers who use PPPoE are (historically)
particularily bad at ensuring that the proper icmp makes it back.
It isn't uncommon for ADSL providers to simply tell all their customers
that they must lower their MTU, usually to between 1360 and 1492.
For example, my residential provider used to use a higher encapsulation
overhead than they do now, and at that time if one didn't lower one's
MTU drastically, then one could not even successfully load the provider's
home page.
--
Most Windows users will run any old attachment you send them, so if
you want to implicate someone you can just send them a Trojan
-- Adam Langley

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Damian Menscher <menscher+usenet@uiuc.edu> writes:

>I'm considering a switch of our RHEL3 fileserver to use jumbo frames
>in an attempt to improve Samba and NFS performance. Our entire LAN
>is gigabit, and connected via switches that support jumbo frames. So
>I expect that switching shouldn't be a problem.

You must be aware that _all_ nodes participating in this LAN will have
to be configured for (the same) jumbo frame MTU.

>Since our server will also need to communicate with the outside world,
>I need to make sure that PMTU discovery can't break. I understand
>that it should get back an ICMP fragmentation-needed packet if a
>packet encounters a link with a smaller MTU. But how reliable is
>this? Are there routers that don't send those back? Is this
>something to worry about?

Yes, it is something to worry about. And the worries are not under
your control - it's routers, loadbalancers, and firewalls in remote
places, e.g. in front of www.ebay.com.

The "normal" 1500 byte ethernet MTU almost always works in those
icmp-broken cases. Thus, the easiest way to circumvent problems,
is to have a second ethernet link on all machines where you need
to communicate with the outside world, running that link with
the ethernet standard MTU. If you can't easily put a second
interface on the machines, maybe VLAN trunking would work,
with one (internal) VLAN configured for jumbo frames, and
the other (external) VLAN configured with the standard MTU.

best regards
Patrick

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

In comp.dcom.lans.ethernet Damian Menscher <menscher+usenet@uiuc.edu> wrote:
> I'm considering a switch of our RHEL3 fileserver to use jumbo frames
> in an attempt to improve Samba and NFS performance. Our entire LAN
> is gigabit, and connected via switches that support jumbo frames. So
> I expect that switching shouldn't be a problem. But I have a few
> concerns:

> Since our server will also need to communicate with the outside world,
> I need to make sure that PMTU discovery can't break. I understand
> that it should get back an ICMP fragmentation-needed packet if a
> packet encounters a link with a smaller MTU. But how reliable is
> this? Are there routers that don't send those back? (I'd expect it
> to break even the standard 1500-byte MTU if there were.) Is this
> something to worry about?

> Also, I'm curious if anything needs to be done to get NFS to take
> advantage of the larger MTU. According to the RFCs, you shouldn't
> send an packet larger than 576 bytes without knowledge that the
> receiving end can handle it. In TCP, the exchange of MTU information
> is handled during the initial handshake. But if NFS is being sent
> over UDP, how would it know it can use larger packets?

> Finally, I've seen the arguments against jumbo frames (the percentage
> increase isn't worth it) and suggestions to try other things (like
> change the window size or acking procedures). Any specific advice on
> things like that would be appreciated. Given that this is a
> production server, I won't want to try anything too unusual, but if
> there are fairly standard/simple changes then I wouldn't mind giving
> them a go.

I would start some measurment to assure that the current status and
sort out any packet drops and other misbehavior. Also arrange a
"measurment client" where scripted "typical transactions" are performed.
With this as a base you might start testing if jumboframes
gives anything.
You should also dump the networking gears counters and monitor
them during traffic. Any packet drops, buffer shortages or simular
will affect your performace. Aslo look out for networking-layer
wierdness ( wrong netmasks etc)

Yes, the world outside your LAN will be affected as jumboframes
is a "layer-2" thingy, and not standardized in any way.

NFS will be using your jumboframes automatically. This both for
nfs/UDP and nfs/TCP.

But as i read that you use linux, you might find the sources of
your performace trouble here, depending on kernel version.

> Damian Menscher
> --
> -=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
> -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc217)333-0038 |#=-
> -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group217)244-3074 |#=-
> -=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax217)333-9819 |#=-
> -=#| The above opinions are not necessarily those of my employers. |#=-

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

 

[Guest]

Archived from groups: comp.dcom.lans.ethernet (More info?)

Bruce Richardson <itsbruce@uklinux.net> wrote:
> Damian Menscher <menscher+usenet@uiuc.edu> wrote:
>> Since our server will also need to communicate with the outside world,
>> I need to make sure that PMTU discovery can't break. I understand
>> that it should get back an ICMP fragmentation-needed packet if a
>> packet encounters a link with a smaller MTU. But how reliable is
>> this? Are there routers that don't send those back? (I'd expect it
>> to break even the standard 1500-byte MTU if there were.) Is this
>> something to worry about?

> All you need to do is make sure that your gateway router a) enforces a 1500
> mtu on outgoing traffic and b) supports PMTU properly. Then the jumbo
> frames will stop at the door.

and beyond that, the MSS exchange at the beginning of a TCP connection
will "cover" 90+% of the situations to begin with - all those remote
sites will _probably_ still be on standard MTUs and so will be sending
MSS options of ~1460 bytes, which your TCP will be honoring anyway.
The time you really need PMTU to work is when you connect to a remote
site that is also useing jumbo frame. Then you have what has been
called a "dumbell" network - larger on the ends than in the middle -
precisely the situation for which PMTU Discovery was invented in the
first place.

As others point-out, some folks filter the ICMPs - probably concerned
about them being spoofed or something I guess. That tends to break
PMTU Discovery - or at least slow things down by forcing extra
retransmissions while TCP's walk their PTMU tables.

rick jones
--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
- Rick Jones
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...