So, what you need to do after upgrading Linux kernel and/or applying GPFS fix pack?
Situation 1: Upgrading Linux kernel only
Previous Linux kernel: 2.6.18-274.3.1.el5
Current Linux kernel: 2.6.18-274.12.1.el5
After rebooted to new kernel, GPFS won’t be able to start due missing of GPFS portability layer
[root@kpgmedprod02 ~]# mmstartup
Tue Dec 13 15:26:41 MYT 2011: mmstartup: Starting GPFS ...
[root@kpgmedprod02 ~]# tail -f /var/adm/ras/mmfs.log.latest
Tue Dec 13 15:26:41 MYT 2011: runmmfs starting
Removing old /var/adm/ras/mmfs.log.* files:
Unloading modules from /lib/modules/2.6.18-274.12.1.el5/extra
runmmfs: The /lib/modules/2.6.18-274.12.1.el5/extra/mmfslinux.ko kernel extension does not exist.
runmmfs: Unable to verify kernel/module configuration.
Loading modules from /lib/modules/2.6.18-274.12.1.el5/extra
runmmfs: The /lib/modules/2.6.18-274.12.1.el5/extra/mmfslinux.ko kernel extension does not exist.
runmmfs: Unable to verify kernel/module configuration.
Tue Dec 13 15:26:41 MYT 2011 runmmfs: error in loading or unloading the mmfs kernel extension
Tue Dec 13 15:26:41 MYT 2011 runmmfs: stopping GPFS
[root@kpgmedprod02 ~]# mmgetstate
Node number Node name GPFS state
------------------------------------------
2 kpgmedprod02 down
What we need to do is these 4 simple steps: cd /usr/lpp/mmfs/src && make Autoconfig && make World && make InstallImages
Make sure kernel-devel package for running kernel is installed, otherwise we’ll hit this error:
[root@kpgmedprod02 ~]# cd /usr/lpp/mmfs/src && make Autoconfig && make World && make InstallImages
cd /usr/lpp/mmfs/src/config; ./configure --genenvonly; if [ $? -eq 0 ]; then /usr/bin/cpp -P def.mk.proto > ./def.mk; exit $? || exit 1; else exit $?; fi
Cannot find a valid kernel include dir
make: *** [Autoconfig] Error 1
If everything is good, we won’t see any error. And we can start GPFS daemon,
[root@kpgmedprod02 ~]# mmstartup
Tue Dec 13 15:31:35 MYT 2011: mmstartup: Starting GPFS ...
We can verify the result in GPFS log (/var/adm/ras/mmfs.log.latest),
Tue Dec 13 15:31:35 MYT 2011: runmmfs starting
Removing old /var/adm/ras/mmfs.log.* files:
Unloading modules from /lib/modules/2.6.18-274.12.1.el5/extra
Loading modules from /lib/modules/2.6.18-274.12.1.el5/extra
Module Size Used by
mmfs26 1384264 0
mmfslinux 316776 1 mmfs26
tracedev 67148 2 mmfs26,mmfslinux
Tue Dec 13 15:31:36.027 2011: mmfsd initializing. {Version: 3.3.0.15 Built: Jun 7 2011 15:09:10} ...
Tue Dec 13 15:31:37.106 2011: Connecting to 10.200.66.150 kpgmedprod01
Tue Dec 13 15:31:37.107 2011: Connected to 10.200.66.150 kpgmedprod01
Tue Dec 13 15:31:37.106 2011: Connecting to 10.200.66.154 kpgmedprod05
Tue Dec 13 15:31:37.107 2011: Connected to 10.200.66.154 kpgmedprod05
Tue Dec 13 15:31:37.118 2011: mmfsd ready
And yes, GPFS state is now active,
[root@kpgmedprod02 ~]# mmgetstate
Node number Node name GPFS state
------------------------------------------
2 kpgmedprod02 active
Situation 2: Applying GPFS Fix Pack only
Current GPFS version (3.3.0-15),
[root@kpgmedprod05 ~]# rpm -qa | grep -i gpfs
gpfs.gpl-3.3.0-15
gpfs.docs-3.3.0-15
gpfs.base-3.3.0-15
gpfs.gui-3.3.0-15
gpfs.msg.en_US-3.3.0-15
I’m going to install latest GPFS fix pack (version 3.3.0-18)
[root@kpgmedprod05 ~]# ls -lh gpfs.*
-rw-r--r-- 1 root root 6.1M Dec 13 12:45 gpfs.base-3.3.0-18.x86_64.update.rpm
-rw-r--r-- 1 root root 163K Dec 13 12:45 gpfs.docs-3.3.0-18.noarch.rpm
-rw-r--r-- 1 root root 430K Dec 13 12:45 gpfs.gpl-3.3.0-18.noarch.rpm
-rw-r--r-- 1 root root 55M Dec 13 12:45 gpfs.gui-3.3.0-18.x86_64.rpm
-rw-r--r-- 1 root root 77K Dec 13 12:45 gpfs.msg.en_US-3.3.0-18.noarch.rpm
[root@kpgmedprod05 ~]# rpm -Uvh gpfs.*
Preparing... ########################################### [100%]
1:gpfs.base ########################################### [ 20%]
2:gpfs.docs ########################################### [ 40%]
3:gpfs.gpl ########################################### [ 60%]
4:gpfs.gui ########################################### [ 80%]
You may start the GPFS GUI now by typing : /etc/init.d/gpfsgui start
Alternatively, the GPFS GUI will start on reboot.
5:gpfs.msg.en_US ########################################### [100%]
Like Situation 1 above, we need to rebuild GPFS portability layer,
[root@kpgmedprod05 ~]# cd /usr/lpp/mmfs/src/ && make Autoconfig && make World && make InstallImages
Once done, bring up GPFS service,
[root@kpgmedprod05 src]# mmstartup
Tue Dec 13 15:47:45 MYT 2011: mmstartup: Starting GPFS ...
Tadaaa, as expected everything is good,
[root@kpgmedprod05 src]# mmgetstate
Node number Node name GPFS state
------------------------------------------
5 kpgmedprod05 active
Situation 3: Upgrading Linux kernel and Applying GPFS Fix Pack
This actually a combination of Situation 1 & Situation 2, just rebuild GPFS portability layer again with this command:
cd /usr/lpp/mmfs/src && make Autoconfig && make World && make InstallImages
Conclusion
We need to rebuilt GPFS portability layer (command: cd /usr/lpp/mmfs/src && make Autoconfig && make World && make InstallImages) after upgrading Linux kernel and/or after applying GPFS fix pack.
Tagged ibm-gpfs