[Linux-ha-dev] Any interest for a hardened version of MySQL OCF RA ?

[Eric Defferard]

While setting up a Linux-HA/DRBD cluster last november I stumbled on the 
MySQL RA which eventually I got to work, but had a few problems handling 
my HA operations requirements.

My application architecture is based on grouping different services into 
distinct application instances, and the ability to run, not only all 
services that constitute an instance on the same host, but also all 
application instances of the application set on the same host, which may 
include the necessity to run multiple instances of a same program on the 
same host, but with totally different configuration and operating 
environments, even though the executable may very well be in many cases 
exactly the same for all instances. Basically, I dedicate a DRBD volume 
to each application instance. Each applications instance dedicated 
volume contains an instance specific file hierarchy structure, with a 
/bin, /etc, /var, to hold all service configuration files, instance 
specific scripts, jobs, data and logs. That way, when an application 
instance is moved to another host, mounting the instance dedicated 
volume guarantees to the new host exclusive access to it.

As an example of architecture, here is subset of a high level network 
management and control platform I am working on. It's  based on three 
application instances:
app-db contains:
     - an instance of mysql with data
     - an instance of apache to provide db management through the web 
(phpmyadmin)
    - an instance of cronjobs
app-www contains:
    - an instance of frontend applications to the data stored in the 
mysql databases offered by the app-db instance
    - an instance of mysql with data regarding the usage of the frontend 
www applications
    - an instance of cronjobs
app-netsvc contains:
    - an instance of a dhcp server
    - an instance of a bind server
    - an instance of a few utilities used to collect data
    - an instance of apache with www accessible tools to manage the 
online operations of the dhcp server and the bind server
    - an instance of cronjobs

This application architecture also offers maximum flexibility when comes 
the time to update software, provided your software allows you to 
install multiple versions of itself on the same node. All you need is 
pull out one node out of the cluster and update software there, test it, 
and if you're happy bump all your instances on that node.

In fact, the way to do it would be to copy an instance, say app-db, to 
appupg-db. Two different application instances, each served with it's 
own instance IP address.

The first problem I encountered has it's origin in MySQL and it's half 
dozen ways of specifying options, including overwriting options 
previously defined in configuration files by including those in other 
configuration files or command line options which are read and assigned 
later in the startup process. Various factors such as specifying the 
executable itself, with it's set of built-in defaults defined at compile 
time, or specifying 'basedir', the root installation path, have the 
dramatic effect of starting a mysqld server daemon with very different 
parameter values, depending on the node on which the server is started, 
regardless of identical configuration files. I must admit, I wouldn't be 
so careful with say apache, simply because, MySQL has the potential to 
write data and cause damage by overwriting existing files.

The second issue is the limitation of DRBD which prevents read access to 
any foreign mounted DRBD volume.

Additionally, the difficulty of thoroughly verifying configuration 
option values on each node, caused by the unability for all nodes of a 
cluster to read a DRBD volume mounted by one node, the mount giving 
exclusive access to that node, the DRBD volume containing the whole 
configuration and data specific to an instance. In other words, you can 
only thoroughly verify when you have access to the instance's DRBD 
volume, in other words at the instance startup on that node.

For these reasons, I undertook a rewrite of the MySQL OCF RA to greatly 
improve configuration data validation, eliminate configuration option 
value and defaults ambiguities, forcing in some cases the mandatory 
assignment of options as to avoid multiple instances of mysql of running 
after the same data, if, for some reason, all application instance end 
up all running on the same node, provided of course the node has the 
horsepower needed for this.

I believe I'll the RA script available in roughly a week. If there is 
anybody interested in giving it a whirl, just leave your email address 
in this thread.