Skip to content
{ 2007 06 07 }

A Journey On DRBD For MySQL

Alhamdulillah, I’m successfully install and deploy DRBD for MySQL, yet another Linux Cluster.

Prologue

The objective is to deploy a high-availability MySQL database cluster to use with Desknow. After getting frustrated (I mean really frustrated!) with MySQL Cluster and Continuent uni/cluster solution, I decided to try DRBD for MySQL. Thanks to MySQL Newsletter May 2007, which is mentioning about DRBD for MySQL.

Scenario

drbd_4_mysql.png

Server Function IP Address Operating System MySQL version
server1 MySQL server (primary) x.x.x.x RHEL 5 5.0.22-2.1
server2 MySQL server (secondary) y.y.y.y RHEL 5 5.0.22-2.1
desknow Desknow Application z.z.z.z RHEL 4 U5 -

Legend:
server1.tld = FQDN for server1
server2.tld = FQDN for server2
a.a.a.a = virtual IP for the heartbeat service
b.b.b.b = Gateway address for server1 and server2
x.x.x.x = IP address for server1
y.y.y.y = IP address for server2
z.z.z.z = IP address for desknow
?.?.?.? = broadcast address for a.a.a.a

Pre-Requisites

DRBD Configuration

1. Download and install latest DRBD on server1,
[root@server1 ~]# wget http://oss.linbit.com/drbd/8.0/drbd-8.0.3.tar.gz
[root@server1 ~]# cd /tmp/
[root@server1 tmp]# tar czvf /root/drbd-8.0.3.tar.gz
[root@server1 tmp]# cd drbd-8.0.3
[root@server1 drbd-8.0.3]# make tools
[root@server1 drbd-8.0.3]# make install

2. Test it,
[root@server1 drbd-8.0.3]# modprobe drbd
[root@server1 drbd-8.0.3]# cat /proc/drbd
version: 8.0.3 (api:86/proto:86)
SVN Revision: 2881 build by root@server1.tld, 2007-06-04 10:11:24

3. Repeat Step 1 & Step 2 for server2.

4. Configure
Edit /etc/drbd.conf on both server1 and server2,

code
< view plain text >
  1. resource mysql {
  2.   protocol      C;
  3.  
  4.   startup { wfc-timeout 0; degr-wfc-timeout     120; }
  5.   disk { on-io-error detach; }
  6.   syncer {
  7.   }
  8.   on server1.tld {
  9.     device      /dev/drbd0;
  10.     disk        /dev/sdb1;
  11.     address     x.x.x.x:8000;
  12.     meta-disk   internal;
  13.   }
  14.   on server2.tld {
  15.     device      /dev/drbd0;
  16.     disk        /dev/sdb1;
  17.     address     y.y.y.y:8000;
  18.     meta-disk   internal;
  19.   }
  20. }

On server1,
[root@server1 etc]# drbdadm create-md mysql
[root@server1 etc]# drbdadm adjust mysql
[root@server1 ~]# drbdsetup /dev/drbd0 primary -o
[root@server1 ~]# service drbd start
[root@server1 ~]# service drbd status

On server2,
[root@server1 ~]# service drbd start
[root@server1 ~]# service drbd status

MySQL Configuration

Nothing much to talk about MySQL, I’m just using default packages (and default configurations) from RHEL 5.

Heartbeat Configuration

1. Installation
Ok, RHEL does not include Heartbeat. It sucks, isn’t it?
It’s not a big deal, though. Just get it from Linux-HA website.

Pre-compiled RPM is available from Linux-HA Download Site. Nevertheless, I’ve got problem to use it. Hence, I chose to build my own RPM package.

I need these 2 packages in order to compile the heartbeat source,
[root@server1 x86_64]# yum install pam-devel swig

Download the heartbeat RPM source package and install it,
[root@server1 ~]# wget http://linux-ha.org/download/heartbeat-2.0.8-1.src.rpm
[root@server1 ~]# rpm -ivh heartbeat-2.0.8-1.src.rpm

Build RPM package,
[root@server1 ~]# cd /usr/src/redhat/SPECS/
[root@server1 SPECS]# rpmbuild -bb heartbeat.spec

Install particular packages,
[root@server1 SPECS]# cd /usr/src/redhat/RPMS/x86_64/
[root@server1 x86_64]# rpm -ivh heartbeat-pils-2.0.8-1.x86_64.rpm
[root@server1 x86_64]# rpm -ivh heartbeat-stonith-2.0.8-1.x86_64.rpm

At first, the installation of heartbeat failed.

code
< view plain text >
  1. [root@server1 x86_64]# rpm -ivh heartbeat-2.0.8-1.x86_64.rpm
  2. Preparing...                ########################################### [100%]
  3. useradd: user hacluster exists
  4. error: %pre(heartbeat-2.0.8-1.x86_64) scriptlet failed, exit status 9
  5. error:   install: %pre scriptlet failed (2), skipping heartbeat-2.0.8-1

But it is ok for the second time.

code
< view plain text >
  1. [root@server1 x86_64]# rpm -ivh heartbeat-2.0.8-1.x86_64.rpm
  2. Preparing...                ########################################### [100%]
  3. 1:heartbeat              ########################################### [100%]

2. Configuration
Create ha.cf, haresources and authekeys files by copying from /usr/share/doc/heartbeat-2.0.8/
[root@server1 ~]# cd /etc/ha.d/
[[root@server1 ha.d]# cp /usr/share/doc/heartbeat-2.0.8/ha.cf .
[root@server1 ha.d]# cp /usr/share/doc/heartbeat-2.0.8/haresources .
[root@server1 ha.d]# cp /usr/share/doc/heartbeat-2.0.8/authkeys .

Edit haresources,

code
< view plain text >
  1. server1.tld z.z.z.z drbddisk::mysql Filesystem::/dev/drbd0::/var/lib/mysql::ext3 mysqld

Edit ha.cf,

code
< view plain text >
  1. debugfile /var/log/ha-debug
  2. logfile /var/log/ha-log
  3. logfacility     local0
  4. keepalive 2
  5. deadtime 30
  6. warntime 10
  7. initdead 120
  8. udpport 694
  9. bcast   eth0            # Linux
  10. auto_failback on
  11. node    server1.tld
  12. node    server2.tld
  13. ping b.b.b.b
  14. apiauth ipfail gid=haclient uid=hacluster

Edit authkeys,

code
< view plain text >
  1. auth 2
  2. 2 sha1 desknow

Edit /etc/ha.d/resource.d/drbddisk

code
< view plain text >
  1. DEFAULTFILE="/etc/drbd.conf"

Copy MySQL startup script,
[root@server1 ~]# cp /etc/rc.d/init.d/mysqld /etc/ha.d/resource.d/

Copy all relevant files from server1 to server2
[root@server1 x86_64]# scp -v heartbeat-* root@server2.tld:~
[root@server1 ha.d]# scp -v authkeys ha.cf haresources root@server2.tld:/etc/ha.d/

Integration & Testing

1. Stop all services,
[root@server1 ~]# service drbd stop
[root@server1 ~]# service mysqld stop
[root@server1 ~]# service heartbeat stop

[root@server2 ~]# service drbd stop
[root@server2 ~]# service mysqld stop
[root@server2 ~]# service heartbeat stop

2. Configure particular services to enable/disable automatic startup,
Please take note that mysqld should be off during startup, while drbd and heartbeat should be on.
[root@server1 ~]# chkconfig drbd on
[root@server2 ~]# chkconfig drbd on
[root@server1 ~]# chkconfig mysqld off
[root@server2 ~]# chkconfig mysqld off
[root@server1 ~]# chkconfig heartbeat on
[root@server1 ~]# chkconfig heartbeat on

3. Start drbd on both machines
[root@server1 ~]# service drbd start
[root@server2 ~]# service drbd start

4. Start heartbeat on both machines
[root@server1 ~]# service heartbeat start
[root@server2 ~]# service heartbeat start

server1 should get a virtual IP and become primary,

code
< view plain text >
  1. eth0:0    Link encap:Ethernet  HWaddr ??:??:??:??:??:??
  2. inet addr:a.a.a.a  Bcast:?.?.?.?  Mask:255.255.255.0
  3. UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  4. Interrupt:169 Memory:da000000-da011100

5. Desknow Installation
Install Desknow on desknow machine, the MySQL database should be pointed to virtual IP, which is a.a.a.a.

Fail-over Testing

1. Stop heartbeat service on server1
[root@server1 ~]# service heartbeat stop

At server1, MySQL service has stopped while DRBD becomes secondary,

code
< view plain text >
  1. root@server1 ~]# service mysqld status
  2. mysqld is stopped
  3. [root@server1 ~]# service drbd status
  4. drbd driver loaded OK; device status:
  5. version: 8.0.3 (api:86/proto:86)
  6. SVN Revision: 2881 build by root@server1.tld, 2007-06-04 10:11:24
  7. 0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
  8. ns:55564 nr:984 dw:56548 dr:14845 al:5 bm:529 lo:0 pe:0 ua:0 ap:0
  9. resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0
  10. act_log: used:0/127 hits:13886 misses:5 starving:0 dirty:0 changed:5

At server2, you can verify that MySQL service is running while DRBD becomes primary.

code
< view plain text >
  1. [root@server2 ~]# service mysqld status
  2. mysqld (pid 9318) is running...
  3. [root@server2 ~]# service drbd status
  4. drbd driver loaded OK; device status:
  5. version: 8.0.3 (api:86/proto:86)
  6. SVN Revision: 2881 build by root@server2.tld, 2007-06-04 10:49:29
  7. 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
  8. ns:960 nr:55564 dw:56524 dr:8638 al:1 bm:12 lo:0 pe:0 ua:0 ap:0
  9. resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0
  10. act_log: used:0/127 hits:239 misses:1 starving:0 dirty:0 changed:1

Output from /var/log/ha-log (server1),

code
< view plain text >
  1. heartbeat[18493]: 2007/06/06_18:31:25 info: Heartbeat shutdown in progress. (18493)
  2. heartbeat[19310]: 2007/06/06_18:31:25 info: Giving up all HA resources.
  3. ResourceManager[19320]: 2007/06/06_18:31:25 info: Releasing resource group: server1.tld 161.139.18.5 drbddisk::mysql Filesystem::/dev/drbd0::/var/lib/mysql::ext3 mysqld
  4. ResourceManager[19320]: 2007/06/06_18:31:25 info: Running /etc/ha.d/resource.d/mysqld  stop
  5. ResourceManager[19320]: 2007/06/06_18:31:31 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /var/lib/mysql ext3 stopFilesystem[19455]:      2007/06/06_18:31:31 INFO: Running stop for /dev/drbd0 on /var/lib/mysql
  6. Filesystem[19455]:      2007/06/06_18:31:31 INFO: Trying to unmount /var/lib/mysql
  7. Filesystem[19455]:      2007/06/06_18:31:31 INFO: unmounted /var/lib/mysql successfully
  8. Filesystem[19452]:      2007/06/06_18:31:31 INFO:  Success
  9. ResourceManager[19320]: 2007/06/06_18:31:31 info: Running /etc/ha.d/resource.d/drbddisk mysql stop
  10. ResourceManager[19320]: 2007/06/06_18:31:31 info: Running /etc/ha.d/resource.d/IPaddr 161.139.18.5 stop
  11. IPaddr[19561]:  2007/06/06_18:31:31 INFO: /sbin/ifconfig eth0:0 a.a.a.a down
  12. IPaddr[19552]:  2007/06/06_18:31:31 INFO:  Success
  13. heartbeat[19310]: 2007/06/06_18:31:31 info: All HA resources relinquished.
  14. heartbeat[18493]: 2007/06/06_18:31:32 WARN: 1 lost packet(s) for [server2.tld] [5687:5689]
  15. heartbeat[18493]: 2007/06/06_18:31:32 info: No pkts missing from server2.tld!
  16. heartbeat[18493]: 2007/06/06_18:31:33 info: killing HBFIFO process 18496 with signal 15
  17. heartbeat[18493]: 2007/06/06_18:31:33 info: killing HBWRITE process 18497 with signal 15
  18. heartbeat[18493]: 2007/06/06_18:31:33 info: killing HBREAD process 18498 with signal 15
  19. heartbeat[18493]: 2007/06/06_18:31:33 info: killing HBWRITE process 18499 with signal 15
  20. heartbeat[18493]: 2007/06/06_18:31:33 info: killing HBREAD process 18500 with signal 15
  21. heartbeat[18493]: 2007/06/06_18:31:33 info: Core process 18498 exited. 5 remaining
  22. heartbeat[18493]: 2007/06/06_18:31:33 info: Core process 18499 exited. 4 remaining
  23. heartbeat[18493]: 2007/06/06_18:31:33 info: Core process 18500 exited. 3 remaining
  24. heartbeat[18493]: 2007/06/06_18:31:33 info: Core process 18497 exited. 2 remaining
  25. heartbeat[18493]: 2007/06/06_18:31:33 info: Core process 18496 exited. 1 remaining
  26. heartbeat[18493]: 2007/06/06_18:31:33 info: server1.tld Heartbeat shutdown complete.

Output from /var/log/ha-log (server2),

code
< view plain text >
  1. heartbeat[7417]: 2007/06/06_18:31:31 info: Received shutdown notice from 'server1.tld'.
  2. heartbeat[7417]: 2007/06/06_18:31:31 info: Resources being acquired from server1.tld.
  3. heartbeat[8799]: 2007/06/06_18:31:31 info: acquire local HA resources (standby).
  4. heartbeat[8799]: 2007/06/06_18:31:31 info: local HA resource acquisition completed (standby).
  5. heartbeat[7417]: 2007/06/06_18:31:31 info: Standby resource acquisition done [foreign].
  6. heartbeat[8800]: 2007/06/06_18:31:31 info: No local resources [/usr/lib64/heartbeat/ResourceManager listkeys server2.tld]
  7. to acquire.
  8. harc[8819]:     2007/06/06_18:31:31 info: Running /etc/ha.d/rc.d/status status
  9. mach_down[8829]:        2007/06/06_18:31:32 info: Taking over resource group 161.139.18.5
  10. ResourceManager[8849]:  2007/06/06_18:31:32 info: Acquiring resource group: server1.tld 161.139.18.5 drbddisk::mysql Filesystem::/dev/drbd0::/var/lib/mysql::ext3 mysqld
  11. IPaddr[8873]:   2007/06/06_18:31:32 INFO:  Resource is stopped
  12. ResourceManager[8849]:  2007/06/06_18:31:32 info: Running /etc/ha.d/resource.d/IPaddr a.a.a.a start
  13. IPaddr[8927]:   2007/06/06_18:31:32 INFO: Using calculated nic for a.a.a.a: eth0
  14. IPaddr[8927]:   2007/06/06_18:31:32 DEBUG: Using calculated netmask for a.a.a.a: 255.255.255.0
  15. IPaddr[8927]:   2007/06/06_18:31:32 DEBUG: Using calculated broadcast for a.a.a.a: ?.?.?.?
  16. IPaddr[8927]:   2007/06/06_18:31:32 INFO: eval /sbin/ifconfig eth0:0 a.a.a.a netmask 255.255.255.0 broadcast ?.?.?.?
  17. IPaddr[8927]:   2007/06/06_18:31:32 DEBUG: Sending Gratuitous Arp for a.a.a.a on eth0:0 [eth0]
  18. IPaddr[8918]:   2007/06/06_18:31:32 INFO:  Success
  19. ResourceManager[8849]:  2007/06/06_18:31:32 info: Running /etc/ha.d/resource.d/drbddisk mysql start
  20. Filesystem[9075]:       2007/06/06_18:31:32 INFO:  Resource is stopped
  21. ResourceManager[8849]:  2007/06/06_18:31:32 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /var/lib/mysql ext3 start
  22. Filesystem[9116]:       2007/06/06_18:31:32 INFO: Running start for /dev/drbd0 on /var/lib/mysql
  23. Filesystem[9113]:       2007/06/06_18:31:32 INFO:  Success
  24. ResourceManager[8849]:  2007/06/06_18:31:32 info: Running /etc/ha.d/resource.d/mysqld  start
  25. mach_down[8829]:        2007/06/06_18:31:33 info: /usr/lib64/heartbeat/mach_down: nice_failback: foreign resources acquired
  26. mach_down[8829]:        2007/06/06_18:31:33 info: mach_down takeover complete for node server1.tld.
  27. heartbeat[7417]: 2007/06/06_18:31:33 info: mach_down takeover complete.
  28. heartbeat[7417]: 2007/06/06_18:32:02 WARN: node server1.tld: is dead
  29. heartbeat[7417]: 2007/06/06_18:32:02 info: Dead node server1.tld gave up resources.
  30. heartbeat[7417]: 2007/06/06_18:32:02 info: Link server1.tld:eth0 dead.

2. Desknow state
Test whether Desknow application is still running properly or not. In my case, it still good as before. Hence, the fail-over test is succeed.

3. Restart heartbeat service on server1

mysqld service should be running on server1 (and should be stopped on server2).

code
< view plain text >
  1. [root@server1 ~]# service mysqld status
  2. mysqld (pid 21385) is running...
  3. [root@server2 ~]# service mysqld status
  4. mysqld is stopped

Output from /var/log/ha-log (server1),

code
< view plain text >
  1. heartbeat[20884]: 2007/06/07_12:32:32 WARN: Logging daemon is disabled --enabling logging daemon is recommended
  2. heartbeat[20884]: 2007/06/07_12:32:32 info: **************************
  3. heartbeat[20884]: 2007/06/07_12:32:32 info: Configuration validated. Starting heartbeat 2.0.8
  4. heartbeat[20885]: 2007/06/07_12:32:32 info: heartbeat: version 2.0.8
  5. heartbeat[20885]: 2007/06/07_12:32:32 info: Heartbeat generation: 32
  6. heartbeat[20885]: 2007/06/07_12:32:32 info: G_main_add_TriggerHandler: Added signal manual handler
  7. heartbeat[20885]: 2007/06/07_12:32:32 info: G_main_add_TriggerHandler: Added signal manual handler
  8. heartbeat[20885]: 2007/06/07_12:32:32 info: Removing /var/run/heartbeat/rsctmp failed, recreating.
  9. heartbeat[20885]: 2007/06/07_12:32:32 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
  10. heartbeat[20885]: 2007/06/07_12:32:32 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
  11. heartbeat[20885]: 2007/06/07_12:32:32 info: glib: ping heartbeat started.
  12. heartbeat[20885]: 2007/06/07_12:32:32 info: G_main_add_SignalHandler: Added signal handler for signal 17
  13. heartbeat[20885]: 2007/06/07_12:32:32 info: Local status now set to: 'up'
  14. heartbeat[20885]: 2007/06/07_12:32:33 info: Link b.b.b.b:b.b.b.b up.
  15. heartbeat[20885]: 2007/06/07_12:32:33 info: Status update for node b.b.b.b: status ping
  16. heartbeat[20885]: 2007/06/07_12:32:33 info: Link server1.tld:eth0 up.
  17. heartbeat[20885]: 2007/06/07_12:32:33 info: Link server2.tld:eth0 up.
  18. heartbeat[20885]: 2007/06/07_12:32:33 info: Status update for node server2.tld: status active
  19. harc[20894]:    2007/06/07_12:32:33 info: Running /etc/ha.d/rc.d/status status
  20. heartbeat[20885]: 2007/06/07_12:32:34 info: Comm_now_up(): updating status to active
  21. heartbeat[20885]: 2007/06/07_12:32:34 info: Local status now set to: 'active'
  22. heartbeat[20885]: 2007/06/07_12:32:34 info: remote resource transition completed.
  23. heartbeat[20885]: 2007/06/07_12:32:34 info: remote resource transition completed.
  24. heartbeat[20885]: 2007/06/07_12:32:34 info: Local Resource acquisition completed. (none)
  25. heartbeat[20885]: 2007/06/07_12:32:35 info: server2.tld wants to go standby [foreign]
  26. heartbeat[20885]: 2007/06/07_12:32:40 info: standby: acquire [foreign] resources from server2.tld
  27. heartbeat[20905]: 2007/06/07_12:32:40 info: acquire local HA resources (standby).
  28. ResourceManager[20915]: 2007/06/07_12:32:40 info: Acquiring resource group: server1.tld 161.139.18.5 drbddisk::mysql Filesystem::/dev/drbd0::/var/lib/mysql::ext3 mysqld
  29. IPaddr[20939]:  2007/06/07_12:32:40 INFO:  Resource is stopped
  30. ResourceManager[20915]: 2007/06/07_12:32:40 info: Running /etc/ha.d/resource.d/IPaddr a.a.a.a start
  31. IPaddr[20993]:  2007/06/07_12:32:40 INFO: Using calculated nic for a.a.a.a: eth0
  32. IPaddr[20993]:  2007/06/07_12:32:40 DEBUG: Using calculated netmask for a.a.a.a: 255.255.255.0
  33. IPaddr[20993]:  2007/06/07_12:32:40 DEBUG: Using calculated broadcast for a.a.a.a: ?.?.?.?
  34. IPaddr[20993]:  2007/06/07_12:32:40 INFO: eval /sbin/ifconfig eth0:0 a.a.a.a netmask 255.255.255.0 broadcast ?.?.?.?
  35. IPaddr[20993]:  2007/06/07_12:32:40 DEBUG: Sending Gratuitous Arp for a.a.a.a on eth0:0 [eth0]
  36. IPaddr[20984]:  2007/06/07_12:32:40 INFO:  Success
  37. ResourceManager[20915]: 2007/06/07_12:32:40 info: Running /etc/ha.d/resource.d/drbddisk mysql start
  38. Filesystem[21142]:      2007/06/07_12:32:40 INFO:  Resource is stopped
  39. ResourceManager[20915]: 2007/06/07_12:32:40 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /var/lib/mysql ext3 start
  40. Filesystem[21183]:      2007/06/07_12:32:41 INFO: Running start for /dev/drbd0 on /var/lib/mysql
  41. Filesystem[21180]:      2007/06/07_12:32:41 INFO:  Success
  42. ResourceManager[20915]: 2007/06/07_12:32:41 info: Running /etc/ha.d/resource.d/mysqld  start
  43. heartbeat[20905]: 2007/06/07_12:32:42 info: local HA resource acquisition completed (standby).
  44. heartbeat[20885]: 2007/06/07_12:32:42 info: Standby resource acquisition done [foreign].
  45. heartbeat[20885]: 2007/06/07_12:32:42 info: Initial resource acquisition complete (auto_failback)
  46. heartbeat[20885]: 2007/06/07_12:32:42 info: remote resource transition completed.

Output from /var/log/ha-log (server2),

code
< view plain text >
  1. heartbeat[7417]: 2007/06/07_12:32:33 info: Heartbeat restart on node server1.tld
  2. heartbeat[7417]: 2007/06/07_12:32:33 info: Link server1.tld:eth0 up.
  3. heartbeat[7417]: 2007/06/07_12:32:33 info: Status update for node server1.tld: status init
  4. heartbeat[7417]: 2007/06/07_12:32:33 info: Status update for node server1.tld: status up
  5. harc[12421]:    2007/06/07_12:32:33 info: Running /etc/ha.d/rc.d/status status
  6. harc[12431]:    2007/06/07_12:32:33 info: Running /etc/ha.d/rc.d/status status
  7. heartbeat[7417]: 2007/06/07_12:32:33 info: all clients are now paused
  8. heartbeat[7417]: 2007/06/07_12:32:34 info: Status update for node server1.tld: status active
  9. harc[12441]:    2007/06/07_12:32:34 info: Running /etc/ha.d/rc.d/status status
  10. heartbeat[7417]: 2007/06/07_12:32:34 info: remote resource transition completed.
  11. heartbeat[7417]: 2007/06/07_12:32:34 info: server2.tld wants to go standby [foreign]
  12. heartbeat[7417]: 2007/06/07_12:32:35 info: all clients are now resumed
  13. heartbeat[7417]: 2007/06/07_12:32:35 info: standby: server1.tld can take our foreign resources
  14. heartbeat[12451]: 2007/06/07_12:32:35 info: give up foreign HA resources (standby).
  15. ResourceManager[12461]: 2007/06/07_12:32:35 info: Releasing resource group: server1.tld 161.139.18.5 drbddisk::mysql Filesystem::/dev/drbd0::/var/lib/mysql::ext3 mysqld
  16. ResourceManager[12461]: 2007/06/07_12:32:35 info: Running /etc/ha.d/resource.d/mysqld  stop
  17. ResourceManager[12461]: 2007/06/07_12:32:40 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /var/lib/mysql ext3 stopFilesystem[12595]:      2007/06/07_12:32:40 INFO: Running stop for /dev/drbd0 on /var/lib/mysql
  18. Filesystem[12595]:      2007/06/07_12:32:40 INFO: Trying to unmount /var/lib/mysql
  19. Filesystem[12595]:      2007/06/07_12:32:40 INFO: unmounted /var/lib/mysql successfully
  20. Filesystem[12592]:      2007/06/07_12:32:40 INFO:  Success
  21. ResourceManager[12461]: 2007/06/07_12:32:40 info: Running /etc/ha.d/resource.d/drbddisk mysql stop
  22. ResourceManager[12461]: 2007/06/07_12:32:40 info: Running /etc/ha.d/resource.d/IPaddr a.a.a.a stop
  23. IPaddr[12701]:  2007/06/07_12:32:40 INFO: /sbin/ifconfig eth0:0 a.a.a.a down
  24. IPaddr[12692]:  2007/06/07_12:32:40 INFO:  Success
  25. heartbeat[12451]: 2007/06/07_12:32:40 info: foreign HA resource release completed (standby).
  26. heartbeat[7417]: 2007/06/07_12:32:40 info: Local standby process completed [foreign].
  27. heartbeat[7417]: 2007/06/07_12:32:42 WARN: 1 lost packet(s) for [server1.tld] [15:17]
  28. heartbeat[7417]: 2007/06/07_12:32:42 info: remote resource transition completed.
  29. heartbeat[7417]: 2007/06/07_12:32:42 info: No pkts missing from server1.tld!
  30. heartbeat[7417]: 2007/06/07_12:32:42 info: Other node completed standby takeover of foreign resources.

Conclusion

  • I can achieve a penny-less MySQL Database Cluster by using DRBD for MySQL solution. If the price is reasonable, might be subscribed to MySQL professional service.
  • x86_64 architecture proves to be well-integrated with DRBD for MySQL

Disclaimer:

  • I’m not a professional technical writer, apologize if there is anything that you hard to understand
  • I’m not responsible for any damage, disaster or calamity by following this so-called howto
  • I do welcome any comments/critics but I don’t give any technical support here ;)

Comment(s) via Facebook

Posted by irwan on Thursday, June 7, 2007, at 4:29 pm. Filed under Free/OpenSource Software, RHEL, Tips/Trick/Howto. Tagged , . Follow any responses to this post with its comments RSS feed. You can post a comment or trackback from your blog.

{ 25 } Comments

  1. Kevin Francis | June 8, 2007 at 12:30 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.3 on Ubuntu Linux Ubuntu Linux

    Thanks for this article man. I haven’t had to the time to read it yet, but the detailed view of how you did things is greatly appreciated.

    Using Mozilla Firefox Mozilla Firefox 2.0.0.3 on Ubuntu Linux Ubuntu Linux
  2. psteve | June 8, 2007 at 2:46 am | Permalink
    Using Safari Safari 419.3 on Mac OS Mac OS

    Yes, thanks for the details!

    I’d also like to know more about your frustrations with the Continuent clustering software; it’s something I’ve been considering.

    Using Safari Safari 419.3 on Mac OS Mac OS
  3. sicksand | June 8, 2007 at 9:27 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.4 on Windows Windows XP

    bro, nice article. now we can dump emic cluster. woo hooo….

    Using Mozilla Firefox Mozilla Firefox 2.0.0.4 on Windows Windows XP
  4. irwan | June 8, 2007 at 3:03 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on RedHat Linux RedHat Linux

    psteve,
    My problem with Continuent is quite personal, it’s a good software actually. Nevertheless, one of the biggest problem is price. IMHO, it’s quite expensive.

    Kevin Francis & sicksand,
    Thanks for the compliment :)

    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on RedHat Linux RedHat Linux
  5. Praveen | June 9, 2007 at 12:54 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on Linux Linux

    Thanks. I saw your article. Its looks nice and more detailed.

    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on Linux Linux
  6. beck | June 20, 2007 at 7:21 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.4 on Windows Windows XP

    cool article :)

    but what if just mysqld fail ??

    I mean heartbeat is still working well, but only mysql daemon down.

    Using Mozilla Firefox Mozilla Firefox 2.0.0.4 on Windows Windows XP
  7. irwan | June 22, 2007 at 4:13 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on RedHat Linux RedHat Linux

    beck,
    You’ve got the killing point. From my observation; if MySQL service is down (heartbeat still ok), then the secondary machine will not take over the service. Hence, there will be an interruption and downtime.

    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on RedHat Linux RedHat Linux
  8. daikinee | July 16, 2007 at 11:35 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.4 on Windows Windows XP

    Two solutions for the case of MySQL being down:

    1) You can install “mon” on the the primary machine (or another tool) to check wether MySQL is running. If not, configure “mon” to stop Heartbeat.

    2) Use the Heartbeat V2 mode (CRM) and the mysql OCS resource which include a monitor part.

    Using Mozilla Firefox Mozilla Firefox 2.0.0.4 on Windows Windows XP
  9. Brajesh | August 15, 2007 at 2:27 pm | Permalink
    Using Internet Explorer Internet Explorer 6.0 on Windows Windows XP

    I just went through your DRBD + Heartbeat MySQL cluster article published in your blog. That’s really a fantastic one. But there are some questions which are killing my head; hope you will help me in this regard… Your proposed design is a nice low cost solution which can be implemented in a little time… & I built my own lab with your article..

    Just tell me how we will administer these servers, I mean if I give any permission in primary server I have to repeat it in secondary??

    My view is MySQL configuration must be same, right?? i.e. 1st give permission in the 1st one then do a fail over and do the changes on the 2nd server

    Is this the way to administer this cluster or there are some other way too…

    Even I don’t know much about MySQL administration, maybe I am wrong.. please guide me to right point.

    Using Internet Explorer Internet Explorer 6.0 on Windows Windows XP
  10. Irwan | August 16, 2007 at 11:17 am | Permalink
    Using Netfront Netfront 3.3 on SonyEricsson SonyEricsson W850i

    Brajesh, Yes, you need to set particular permission on both servers. MySQL configurations should be same. To be frank, you don’t have to bother much on the MySQL parts.
    (I’m not an expert either)

    Using Netfront Netfront 3.3 on SonyEricsson SonyEricsson W850i
  11. tlin | September 11, 2007 at 6:25 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.6 on Windows Windows XP

    Got stock on the RPMBUILD as below on RHEL 5 trying to install Heartbeat, as following:

    [root@lb-s02 SPECS]# rpmbuild -bb heartbeat.spec
    error: Failed build dependencies:
    libnet is needed by heartbeat-2.0.8-1.i386
    [root@lb-s02 SPECS]# yum install libnet
    Loading “installonlyn” plugin
    Loading “rhnplugin” plugin
    Setting up Install Process
    Setting up repositories
    rhel-i386-server-5 100% |=========================| 1.4 kB 00:00
    rhel-i386-server-vt-5 100% |=========================| 1.4 kB 00:00
    rhn-tools-rhel-i386-serve 100% |=========================| 1.2 kB 00:00
    Reading repository metadata in from local files
    Parsing package install arguments
    No Match for argument: libnet
    Nothing to do
    [root@lb-s02 SPECS]#

    Please help.
    Thanks,
    Terry

    Using Mozilla Firefox Mozilla Firefox 2.0.0.6 on Windows Windows XP
  12. irwan | September 13, 2007 at 9:26 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on RedHat Linux RedHat Linux

    tlin,
    Get libnet from http://www.packetfactory.net/libnet/

    Using Mozilla Firefox Mozilla Firefox 1.5.0.12 on RedHat Linux RedHat Linux
  13. Tamas Tompa | January 29, 2008 at 5:13 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.11 on Windows Windows XP

    Is there anybody who tried to install / test MySQL DRBD on virtual machines (ex: vmware)? Is there any sense?

    Thanks,
    Tamas

    Using Mozilla Firefox Mozilla Firefox 2.0.0.11 on Windows Windows XP
  14. Pablo Godel | January 30, 2008 at 2:23 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.11 on Windows Windows XP

    Can you comment on performance? do you notice any degradation ?

    thanks!

    Using Mozilla Firefox Mozilla Firefox 2.0.0.11 on Windows Windows XP
  15. Irwan | February 2, 2008 at 6:04 am | Permalink
    Using Netscape Navigator Netscape Navigator 9.0.0.5 on Debian GNU/Linux Debian GNU/Linux

    Tamas Tompa,
    I never tried it. But theoretically it should work, I guess.

    Pablo Godel,
    Sorry, I never did any **real** performance test. So far, it’s just working without big problem

    Using Netscape Navigator Netscape Navigator 9.0.0.5 on Debian GNU/Linux Debian GNU/Linux
  16. Terry Lin | February 7, 2008 at 11:33 am | Permalink
    Using Internet Explorer Internet Explorer 7.0 on Windows Windows XP

    I followed almost every DRBD configuration but I failed the following steps:
    # drbdadm adjust mysql
    Failure: (114) Lower device is already claimed. This usually means it is mounted.
    Command ‘drbdsetup /dev/drbd0 disk /dev/sdb1 /dev/sdb1 internal –set-defaults –create-device –on-io-error=detach’ terminated with exit code 10
    # drbdsetup /dev/drbd0 primary -o
    State change failed: (-2) Refusing to be Primary without at least one UpToDate disk

    Pleeease Heeelp!!!
    Terry Lin
    IT SysAdmin

    Using Internet Explorer Internet Explorer 7.0 on Windows Windows XP
  17. irwan | February 8, 2008 at 3:30 am | Permalink
    Using Debian IceWeasel Debian IceWeasel 2.0.0.11 on Debian GNU/Linux Debian GNU/Linux

    Terry Lin,
    From the error message “Failure: (114) Lower device is already claimed. This usually means it is mounted.”; maybe that filesystem has already been used.

    Using Debian IceWeasel Debian IceWeasel 2.0.0.11 on Debian GNU/Linux Debian GNU/Linux
  18. Thorolf | February 11, 2008 at 8:55 pm | Permalink
    Using Mozilla Mozilla 1.8.1.11

    Hi,

    @Tamas Tompa:
    There is an other good article for heartbeat/drbd/mysql, and for testing-purpose he is using vmware:
    http://marksitblog.blogspot.com/2007/07/mysql-5-high-availability-with-drbd-8.html

    @All:
    My configuration is running quite good on CentOS 5.1, but there are two problems that make the configuration quite problematic:
    - If starting only ONE of the servers, DRBD does not come up automaticly. I set “wfc-timeout 300″ that the server at least starts without any user input, but it does not bring up DRBD in “Primary/unknown” status.
    - After a failover to the backup server there should NOT be an autofailback to the primary one. But the DRBD disk should reconnect automaticly as secondary disk to the primary one located on the backup server if the primary server was restarted. But this does not work, the (restarted) primary still shows “Secondary/Unknown” while the backup is in “Primary/Unknown” state, until I solve the problem manually with drbdadm.

    Any idea how to solve these issues?

    Thanks a lot,

    Thorolf

    Using Mozilla Mozilla 1.8.1.11
  19. Terry Lin | February 12, 2008 at 4:23 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.11 on Windows Windows XP

    The default drive mapping has mapped both /dev/sda2 and /dev/sdb1 to the root / at the kickstart installation so the df -h looks like the following:
    Filesystem Type Size Used Avail Use% Mounted on
    /dev/mapper/VolGroup00-LogVol00
    ext3 265G 3.0G 249G 2% /
    /dev/sda1 ext3 99M 15M 80M 16% /boot
    none tmpfs 7.9G 0 7.9G 0% /dev/shm

    How should separte it out without reinstall the OS?
    –terry

    Using Mozilla Firefox Mozilla Firefox 2.0.0.11 on Windows Windows XP
  20. gobok | March 10, 2008 at 12:13 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.12 on Windows Windows XP

    irwan
    thanks for sharing this knowledge journey of yours with us .. very apreciated

    Terry Lin
    i believed u doesn’t have any free space to install drbd .. drbd needs free space or free disk to map drbd drive

    your options is
    1) resize your partition .. usr fdisk or which you comfortable with
    2) add another disk
    3) other sysadmin might have the point of views .. seek their help :)

    Using Mozilla Firefox Mozilla Firefox 2.0.0.12 on Windows Windows XP
  21. Arjan | April 26, 2008 at 5:30 am | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.14 on Windows Windows XP

    In response to the person above with the VmWare question…

    We are running two Vmware Server instances (2 physical) servers) with Centos 4.5 and Mysql…

    Heartbeat + drbd

    Running very wel!!

    Using Mozilla Firefox Mozilla Firefox 2.0.0.14 on Windows Windows XP
  22. Flash | June 6, 2008 at 12:22 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 2.0.0.14 on Windows Windows XP

    Great guide!.

    Are there any sample configuration file (my.cnf) for the above DRBD MySQL setup which we can reference?

    Using Mozilla Firefox Mozilla Firefox 2.0.0.14 on Windows Windows XP
  23. irwan | June 14, 2008 at 10:24 am | Permalink
    Using Debian IceWeasel Debian IceWeasel 2.0.0.14 on Debian GNU/Linux Debian GNU/Linux

    Flash,
    Sorry for the late reply. But as I mentioned in my post, I just use default MySQL configuration, there’s no fancy/special configuration needed :)

    Using Debian IceWeasel Debian IceWeasel 2.0.0.14 on Debian GNU/Linux Debian GNU/Linux
  24. dark | October 28, 2008 at 6:15 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 3.0.3 on Windows Windows XP

    hi irwan ..

    gr8 guided there .. i’ve manage to run this 4 few month already .. but now a prob arise .. my floating IP (active) connection seem very slow .. very damn slow .. and i have to switch it to the passive servers 4 temporary ,but a while i tried to switch it back to original state, it connection seem like forever to connect to the floating ip but sometimes i doesn’t at all … mind sharing with me what i need to do .. :| diagnose .. or i have to reconfigure one server back .. but i have no idea howto :)

    Using Mozilla Firefox Mozilla Firefox 3.0.3 on Windows Windows XP
  25. Irwan | November 7, 2008 at 3:41 pm | Permalink
    Using Mozilla Firefox Mozilla Firefox 3.0.3 on Windows Windows XP

    dark,
    I am sorry to hear that. For your info, it has been 1+ year since my last involvement in DRBD for MySQL. Sorry, I can’t help you, maybe you should try to get assistance in the MySQL mailing-list and forum.

    Using Mozilla Firefox Mozilla Firefox 3.0.3 on Windows Windows XP

{ 2 } Trackbacks

  1. Colin Charles Agenda » Blog Archive » MySQL Resources across the Interwebs | June 8, 2007 at 12:38 pm | Permalink
    Using WordPress WordPress 2.1.3

    [...] adminstrator at one of the only Linux support companies in Malaysia, has recently blogged about his journey with DRBD and MySQL. The operating system base is RHEL 5, and there is a step-by-step guide on how he got it working. [...]

  2. MySQL High-Availability Solutions: Replication, Cluster, DRBD, LVS, MPP « Kunawut's knowledge blog | July 20, 2012 at 10:26 pm | Permalink
    Using WordPress WordPress 3.4.1

    [...] A Journey On DRBD For MySQL http://blog.irwan.name/?p=118 [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *