[Linux-ha-dev] Any interest for a hardened version of MySQL OCF RA ?

[Joe Bill]

While setting up a Linux-HA/DRBD cluster last november
I stumbled on the MySQL RA which eventually I got to
work, but had a few problems handling my HA operations
requirements.

My application architecture is based on grouping
different services into distinct application
instances, and the ability to run, not only all
services that constitute an instance on the same host,
but also all application instances of the application
set on the same host, which may include the necessity
to run multiple instances of a same program on the
same host, but with totally different configuration
and operating environments, even though the executable
may very well be in many cases exactly the same for
all instances. Basically, I dedicate a DRBD volume to
each application instance. Each application's instance
dedicated volume contains an instance specific file
hierarchy structure, with a /bin, /etc, /var, to hold
all service configuration files, instance specific
scripts, jobs, data, logs, sockets, pid-files and so
on... That way, when an application instance is moved
to another host, mounting the instance dedicated
volume guarantees to the new host exclusive access to
it.

As an example of architecture, here is subset of a
high level network management and control platform I
am working on. It's  based on three application
instances:
app-db contains:
    - an instance of mysql with data
    - an instance of apache to provide db management
through the web (phpmyadmin)
   - an instance of cronjobs
app-www contains:
   - an instance of frontend applications to the data
stored in the mysql databases offered by the app-db
instance
   - an instance of mysql with data regarding the
usage of the frontend www applications
   - an instance of cronjobs
app-netsvc contains:
   - an instance of a dhcp server
   - an instance of a bind server
   - an instance of a few utilities used to collect
data
   - an instance of apache with www accessible tools
to manage the online operations of the dhcp server and
the bind server
   - an instance of cronjobs

This application architecture also offers maximum
flexibility when comes the time to update software,
provided your software allows you to install multiple
versions of itself on the same node. All you need is
pull out one node out of the cluster and update
software there, test it, and if you're happy bump all
your instances on that node.

In fact, the way to do it would be to copy an
instance, say app-db, to appupg-db. Two different
application instances, each served with it's own
instance IP address.

The first problem I encountered has it's origin in
MySQL and it's half dozen ways of specifying options,
including overwriting options previously defined in
configuration files by including those in other
configuration files or command line options which are
read and assigned later in the startup process.
Various factors such as specifying the executable
itself, with it's set of built-in defaults defined at
compile time, or specifying 'basedir', the root
installation path, have the dramatic effect of
starting a mysqld server daemon with very different
parameter values, depending on the node on which the
server is started, regardless of identical
configuration files. I must admit, I wouldn't be so
careful with say apache, simply because, launched as a
root process MySQL, has the potential to write data
and cause damage by overwriting existing files.

The second issue is the limitation of DRBD which
prevents read access to any foreign mounted DRBD
volume.

Additionally, the difficulty of thoroughly verifying
configuration option values on each node, caused by
the unability for all nodes of a cluster to read a
DRBD volume mounted by one node, the mount giving
exclusive access to that node, the DRBD volume
containing the whole configuration and data specific
to an instance. In other words, you can only
thoroughly verify when you have access to the
instance's DRBD volume, in other words at the
instance's startup on that node.

For these reasons, I undertook a rewrite of the MySQL
OCF RA to greatly improve configuration data
validation, eliminate configuration option value and
defaults ambiguities, forcing in some cases the
mandatory assignment of options as to avoid multiple
instances of mysql trying to access the same data, or
some other instance's data, if, for some reason, all
application instance end up all running on the same
node, provided of course the node has the horsepower
needed for this.

I believe I'll have the RA script available in roughly
a week. If there is anybody interested in giving it a
whirl, just leave your email address in this thread.



 
____________________________________________________________________________________
Be a PS3 game guru.
Get your game face on with the latest PS3 news and previews at Yahoo! Games.
http://videogames.yahoo.com/platform?platform=120121