[Linux-HA] High availability, fast, fileservers
david.lang at digitalinsight.com
Wed Mar 17 21:31:25 MST 2004
On Wed, 17 Mar 2004, Jeff Tucker wrote:
> I'm researching putting together a fileserver for an ISP for a moderately
> busy mail server. Storage requirements are modest, 500 GB to 1 TB will
> serve for now. A single disk failure can't bring the system down, as those
> happen all too often. Several of the solutions I'm researching would use
> heartbeat and DRDB to give me failover capability for other hardware
> failures as well. I hope this email isn't too far off-topic. I think this
> is a topic that is of wide interest to others.
> I mention that this is an email server because that establishes several
> important data points. The average file size is small, around 5 KB. File
> access is random, all over the disks, as different users access their
> email. And, write speed is important. Writes are probably 1/3 of all
> operations on the disks. All access is from a cluster of Linux servers
> using NFS to get to the fileserver. The current system sustains around 20
> Mbps of data being read continuously from the disks. The current system
> occasionally gets bogged down, probably with individual file accesses as
> opposed to raw data reading speeds.
I don' know what mailserver software you are looking at, but take a look
at project cyrus, it's been around for a while and has all sorts of things
to help you scale to huge numbers of users (it also has supurb IMAP
support as well as POP support)
also look at XFS, I haven't had a chance to work with it much, but I've
heard a lot of good things about it with lots of drives.
with mail you end up doing a lot of syncing where the system waits until
the data is safe on disk, in these cases you really need the fastest disks
that you can buy, the difference between a 7200 rpm IDE/SATA and a 15Krpm
SCSI drive will amaze you. along the same line of thought NFS doesn't have
very good locking performance so you will find that many e-mail systems
reccomend strongly against useing NFS for the datastore.
one thing to keep in mind when looking at the SW raid option is that if
your system starts to get loaded the overhead of the SW raid can topple
you over. if you have dedicated hardware for the fileserver this may not
be an issue (you can buy so much CPU nowdays it's silly), but if you have
good raid hardware you may be able to eliminate the seperate fileserver
another thing to think of when trying to do this is that anytime you have
wires joining a system you have the potential for a major problem in one
system to fry components in another system.
given a (relativly) unlimited budget the approach I would take is the
a pair of dual opteron systems with a few gig of ram hooked via fibre to a
external array with redundant controllers, put a couple gig-fiber
ethernets in each box and do channel bonding so that you can survive a
interface outage and make sure that the boxes are plugged into seperate
UPS systems and you have a STONITH system in place (possibly through the
UPS) and you should be in pretty good shape.
you can buy external drive cages that use SCSI, SATA, or IDE drives
nowdays, but look to your system like a single SCSI drive.
if your data wasn't as dynamic I would go with internal drives on the
servers and have a periodic process (rsync, DRDB, etc) to keep them in
sync, with today's IDE drives a 1T array is only 4 drives, but when you
really need to speed you need the higher RPM of the high-end SCSI drives
I've been useing netapps for about five years now and while they are
fabulous for some things (the snapshot feature can be a lifesaver in many
cases) I have managed to overwelm several generations of them with writes
so I would only use them in places where the access is mostly read, their
drive pricing is also extremely high (a couple years ago when I was
directly involved with them it was something over $1k/drive, and while
capacity per drive is going up the cost per drive is not dropping) I would
not try to use a netapp for mail server storage
if you continue up to the high end beyond the netapps you will find that
some of the SAN disk arrays will implment their own replication for you so
you can buy two of them and have the array itself sync the data. this is
pretty reliable, but it is just a snapshot every x min so you do loose
some data when you failover, the upside is that these will let you
replicate your data across the country if you need to and to your system
it looks like just a single drive.
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
More information about the Linux-HA