Archived from groups: comp.sys.ibm.pc.hardware.storage (
More info?)
Previously Michael <michaeln@twentyten.org> wrote:
> Hey guys,
> I work for a small company that's growing very, very rapidly. We provide a
> mail service to our customer base but have found that we're running into
> performance issues as our user base grows. The performance bottleneck that
> we're seeing appears to be seek time, that is, a user accesses their
> mailbox, grabs a few hundred KB of data, and goes away.
> What is the right way to deal with this situation? We've broken up our SAN
> to have many smaller arrays and are moving users to those smaller drive
> sets, but I was interested in hearing how the users of this group would do
> it.
> Here's our basic setup:
> - Hewlett-Packard SAN array
> - 2Gb Fibre connections
> - Drives consist of 73GB, 15K U320 drives
> From our metrics, we're way under capacity for Disk IO bandwidth, but our
> CPU time during which I/O requests were issued to the device is way
> high.
> Just wondering what the pros would suggest...
I think that the problem is probably not your hardware.
On the OS level you need something that does good caching and needs
to be as memory efficient as possible. You also need a filesystem
that works well with your access patterns. On the software side
you need software that is as efficient as possible, does few seeks
loads fast or better, resides in memory completely and permanently
and generally sees the most common access pattern not as a major
undertaking, but as something to be done fast and efficient.
The high CPU load basically means that the SAN is not the
bottleneck.
One important factor is that flat files are far more efficient
than any kind of DB or DB like structure. If each mail is stored
in an individual file (e.g., just to illustrate the point), you
get maybe 2-3 seeks per mail. If the customer has a single mail
file per account, it can be read completely once and then rewritten
once. Taking into acount that you get 100KB...500KB of linear reading
for the time of a seek on a good filesystem a single maibox
file is probably several orders of magitude faster than individual
files, if the machine this is running on has enough RAM.
One other problem is that your SAN is not so much more efficient than
an bunch of IDE disks with software RAID. Maybe half an order of
magnitude, if that. On the software, filesystem and OS side, there
is usually far more room for optimisation. Another effect is that
a SAN does not perform better in seeks than a single disk if it
is RAID5. It does so with regard to throughput, but not with regard
to access time. I think you already know this.
The other thing you need to think about is distributing. The
task is very well suited to it. Maybe put RAID-1 pairs of these
disks in high-quality PCs with FreeBSD or Linux and forget about
SANs completely, except for additional backup which the customer
never accesses. A SAN is ideal for large amounts of data with
little requests. You seem to need smaller amounts of data with
a lot of requests and possibly some CPU load. So assign as
few disks as possibly to each CPU and get more CPUs.
Then you can add a new machine for every 1000 customers or the like
(no idea how high this number would be) and your bottleneck becomes
the mail handling on the SMTP side (can be possibly done with a
forward server that does no disk access for the individual forward,
keeping the cutomer DB completely in memory, e.g.) and log-in (can be
done with one or a number of log-in servers that forward the
individual log-ins or requests to the storage servers and also
does practically no disk accesses. This way you be able to scale
pretty fast.
Just some thoughts on the problem. What you urgently need
is a strategic plan that allows scaling for a longer time
and keeps cost per customer as constant as possible.
Arno
--
For email address: lastname AT tik DOT ee DOT ethz DOT ch
GnuPG: ID:1E25338F FP:0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
"The more corrupt the state, the more numerous the laws" - Tacitus
Facebook
Twitter
Instagram