Exadata and ZDLRA, Disable HAIP
HAIP (High Availability IP) is not supported for the Exadata environment but can occur (if you did not create the cluster using OEDA) that HAIP became in use. And this particularity true for ZDLRA. So, during the upgrade from the previous version (12.2) to a higher version, it is needed to remove HAIP.
Usually, when we upgrading from 12.2 to 18c the HAIP is removed from Exadata. If the upgrade is from 12.1, and HAIP is there, it continues and is not removed by the upgrade process. If you are using HAIP and your GI is 12.1, this procedure as-is described here can’t be used (need some adaptation), because of some requirements from ASM+ACFS+DB. But since this is a preliminary step from a GI upgrade, the focus is to disable and remove it from GI.
The HAIP is not needed for Exadata because by architecture the InfiniBand network already defines (per server) two IP’s to avoid the single point of failure. So, it is not needed to create an additional layer (HAIP and virtual IP), that does the same that already exists by network design.
*Image removed from Oracle Presentation: Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour Oct 2019
This procedure can be used for a non-Exadata environment too. But before continue it is important to be aware of some details with ACFS. If you use ACFS, be careful since the HAIP IP is used in the ASM proxy and needs to be updated; and if ACFS is in place, this procedure is valid just for GI 12.2+.
Environment
In this scenario, I am removing HAIP from ZDLRA GI. The HAIP has configured automatically and we are using ACFS.
The focus is to disable HAIP and remove it from CRS. Another detail is that we need to remove the dependencies between the CRS resources (like ASM that need HAIP) too.
One important detail is that since we are removing the HAIP, we need to swap the HAIP IP that it is used by services (like ASM Proxy) for something that exists. So, you need to have high availability IP’s to use. For Exadata, we use the interconnect IP’s, but if you are using in another environment, be careful that your network has the needed requirements (throughput, failover, and others).
GI Upgrade
During the GI upgrade from 12.2 to 19c we need to run the runcluvfy and it will detect if HAIP is enabled:
....
....
Checks did not pass for the following nodes:
zeroserv02,zeroserv01
Failures were encountered during execution of CVU verification request "stage -pre crsinst".
Verifying Node Connectivity ...FAILED
zeroserv02: PRVG-11068 : Highly Available IP (HAIP) is enabled on the nodes
"zeroserv01,zeroserv02".
zeroserv01: PRVG-11068 : Highly Available IP (HAIP) is enabled on the nodes
"zeroserv01,zeroserv02".
Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.
CVU operation performed: stage -pre crsinst
Date: Dec 9, 2019 1:38:46 PM
CVU home: /u01/app/19.0.0.0/grid/
User: oracle
[root@zeroserv01 ~]#
Remove HAIP
The steps need to be executed in order to avoid errors during the procedure. Unfortunately, it is needed to have one maintenance window due to the unavailability of the services. The running databases need to be shutdown due to the CRS restart by an example.
Bellow, the steps are summarized and explained to be followed. Please be aware of the IP’s involved (they will be different in your case).
1 – Shutdown services
So, this first point is to shutdown all databases running in this cluster:
[oracle@zeroserv01 ~]$ srvctl stop database -d zdlras -o immediate
[oracle@zeroserv01 ~]$
2- Check HAIP, ASM Proxy, and ACFS
It is needed to check if HAIP is enabled. Another detail is check for ASM proxy, and ACFS too:
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res ora.cluster_interconnect.haip -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cluster_interconnect.haip
1 ONLINE ONLINE zeroserv01 STABLE
--------------------------------------------------------------------------------
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res ora.proxy_advm -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.proxy_advm
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
--------------------------------------------------------------------------------
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/crsctl stat res -w "TYPE = ora.acfs.type" -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.catalog.raadmin.acfs
ONLINE ONLINE zeroserv01 mounted on /raacfs/r
aadmin,STABLE
ONLINE ONLINE zeroserv02 mounted on /raacfs/r
aadmin,STABLE
ora.catalog.raosbadmin.acfs
ONLINE ONLINE zeroserv01 mounted on /osbcat,S
TABLE
ONLINE ONLINE zeroserv02 mounted on /osbcat,S
TABLE
--------------------------------------------------------------------------------
[oracle@zeroserv01 ~]$
3 – Check New IP
The next step is to check the new IP that will be used. Since ASM does not use HAIP, we can (at Exadata) pickup the same used IP:
[oracle@zeroserv01 ~]$ export ORACLE_SID=+ASM1
[oracle@zeroserv01 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Mon Dec 9 15:00:55 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> show parameter cluster_interconnects
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects string 192.168.10.1:192.168.10.2
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ $ORACLE_HOME/bin/oifcfg getif
bondeth0 10.208.68.0 global public
ib0 192.168.8.0 global cluster_interconnect,asm
ib1 192.168.8.0 global cluster_interconnect,asm
[oracle@zeroserv01 ~]$
In this case, the IP that will substitute HAIP IP is 192.168.10.1/192.168.10.2. And, as you can see above, they are from the interconnect network.
4 – Fixing IP for ASM Proxy
The next step is to fix the IP that is used by the ASM Proxy instance. But default, it pickup the HAIP IP during the startup. The idea here is to force the same IP than the ASM instance for cluster_interconnect parameter.
Connecting at APX instance
Connect at APX instance, check the parameters (and see that are without values). After that we create one init file from memory to have the backup:
[oracle@zeroserv01 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv01 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv01 ~]$ export ORACLE_SID=+APX1
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:20:59 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> show parameter instance_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
instance_name string +APX1
SQL> show parameter instance_type
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
instance_type string ASMPROXY
SQL> show parameter cluster_interconnects
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects string
SQL> create pfile = '/tmp/pfileapx1' from memory;
File created.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv01 ~]$
Changing the IP
For ASM Proxy instance we need to set the IP for cluster_interconnect parameter with the same IP used by ASM instance.
To do that, we need to set it in init for ASM Proxy. But since the ASM Proxy instance doesn’t have one, we create one at $GI_HOME/dbs with the connect name (look below the folder and instance name). Below I set just the cluster to interconnect parameter and dedicated to the instance running in this node.
[oracle@zeroserv01 ~]$ echo "+APX1.cluster_interconnects='192.168.10.1:192.168.10.2'" > $ORACLE_HOME/dbs/init+APX1.ora
[oracle@zeroserv01 ~]$
[oracle@zeroserv01 ~]$ cat $ORACLE_HOME/dbs/init+APX1.ora
+APX1.cluster_interconnects='192.168.10.1:192.168.10.2'
[oracle@zeroserv01 ~]$
Restart the ASM Proxy
After creating the init, we restart the ASM Proxy in this node.
[root@zeroserv01 ~]# export ORACLE_HOME=/u01/app/12.2.0.1/grid
[root@zeroserv01 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# $ORACLE_HOME/bin/srvctl stop asm -proxy -node zeroserv01 -force
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# $ORACLE_HOME/bin/srvctl start asm -proxy -node zeroserv01
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# ps -ef |grep APX
oracle 267811 1 0 10:43 ? 00:00:00 apx_pmon_+APX1
oracle 267813 1 0 10:43 ? 00:00:00 apx_clmn_+APX1
oracle 267815 1 0 10:43 ? 00:00:00 apx_psp0_+APX1
oracle 267820 1 1 10:43 ? 00:00:00 apx_vktm_+APX1
oracle 267824 1 0 10:43 ? 00:00:00 apx_gen0_+APX1
oracle 267826 1 0 10:43 ? 00:00:00 apx_mman_+APX1
oracle 267830 1 0 10:43 ? 00:00:00 apx_gen1_+APX1
oracle 267834 1 0 10:43 ? 00:00:00 apx_diag_+APX1
oracle 267836 1 0 10:43 ? 00:00:00 apx_dskm_+APX1
oracle 267838 1 0 10:43 ? 00:00:00 apx_pman_+APX1
oracle 267840 1 0 10:43 ? 00:00:00 apx_dia0_+APX1
oracle 267842 1 0 10:43 ? 00:00:00 apx_lreg_+APX1
oracle 267845 1 0 10:43 ? 00:00:00 apx_pxmn_+APX1
oracle 267847 1 0 10:43 ? 00:00:00 apx_rbal_+APX1
oracle 267849 1 0 10:43 ? 00:00:00 apx_vbg0_+APX1
oracle 267851 1 0 10:43 ? 00:00:00 apx_vdbg_+APX1
oracle 267853 1 0 10:43 ? 00:00:00 apx_vubg_+APX1
root 267979 32720 0 10:43 pts/0 00:00:00 grep --color=auto APX
[root@zeroserv01 ~]#
Look that instance restarted correctly and it is up.
And we can check the parameter if it is OK with the IP:
[oracle@zeroserv01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:51:59 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> show parameter cluster_interconnects
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects string 192.168.10.1:192.168.10.2
SQL> show parameter instance_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
instance_name string +APX1
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv01 ~]$
If the instance not startup (or the IP’s are wrong) you need to check the init file that you created in $GIHOME/dbs folder.
Restart ACFS
Since we restarted the ASM Proxy instance, the ACFS mountpoints went down and need to be up again:
[root@zeroserv01 ~]# $ORACLE_HOME/bin/crsctl start res -w "TYPE = ora.acfs.type" -n zeroserv01
CRS-2672: Attempting to start 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01'
CRS-2672: Attempting to start 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01'
CRS-2676: Start of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.catalog.raosbadmin.acfs' on 'zeroserv01'
CRS-2676: Start of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.catalog.raadmin.acfs' on 'zeroserv01'
CRS-2676: Start of 'ora.catalog.raosbadmin.acfs' on 'zeroserv01' succeeded
CRS-2676: Start of 'ora.catalog.raadmin.acfs' on 'zeroserv01' succeeded
[root@zeroserv01 ~]#
I started up using the CRS resource for that.
Other Nodes
After it is Ok for node 1, we can do it for other nodes. Here, since I made for ZDLRA, I have just the second node to fix ASM Proxy. Be careful with the IP’s used, they are specific for this node and remember to use the correct init file.
[root@zeroserv02 ~]# su - oracle
Last login: Tue Dec 10 10:53:02 CET 2019
[oracle@zeroserv02 ~]$ export ORACLE_SID=+ASM2
[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:56:24 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> show parameter cluster_interconnects
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects string 192.168.10.3:192.168.10.4
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX1
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX2
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 10:57:15 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL>
SQL> show parameter instance_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
instance_name string +APX2
SQL> show parameter cluster_interconnects
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects string
SQL> create pfile = '/tmp/pfileapx2' from memory;
File created.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ ls -l $ORACLE_HOME/dbs/init?APX*
ls: cannot access /u01/app/12.2.0.1/grid/dbs/init?APX*: No such file or directory
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ echo "+APX2.cluster_interconnects='192.168.10.3:192.168.10.4'" > $ORACLE_HOME/dbs/init+APX2.ora
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ cat $ORACLE_HOME/dbs/init+APX2.ora
+APX2.cluster_interconnects='192.168.10.3:192.168.10.4'
[oracle@zeroserv02 ~]$
[oracle@zeroserv02 ~]$ exit
[root@zeroserv02 ~]# export ORACLE_HOME=/u01/app/12.2.0.1/grid
[root@zeroserv02 ~]# export PATH=$ORACLE_HOME/bin:$PATH
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# $ORACLE_HOME/bin/srvctl stop asm -proxy -node zeroserv02 -force
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# $ORACLE_HOME/bin/srvctl start asm -proxy -node zeroserv02
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# $ORACLE_HOME/bin/crsctl start res -w "TYPE = ora.acfs.type" -n zeroserv02
CRS-2672: Attempting to start 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02'
CRS-2676: Start of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.catalog.raosbadmin.acfs' on 'zeroserv02'
CRS-2676: Start of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.catalog.raadmin.acfs' on 'zeroserv02'
CRS-2676: Start of 'ora.catalog.raosbadmin.acfs' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.catalog.raadmin.acfs' on 'zeroserv02' succeeded
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# su - oracle
Last login: Tue Dec 10 10:54:10 CET 2019
[oracle@zeroserv02 ~]$ export ORACLE_HOME=/u01/app/12.2.0.1/grid
[oracle@zeroserv02 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX1
[oracle@zeroserv02 ~]$ export ORACLE_SID=+APX2
[oracle@zeroserv02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 10 11:00:46 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> show parameter cluster_interconnects
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_interconnects string 192.168.10.3:192.168.10.4
SQL> show parameter instance_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
instance_name string +APX2
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
You have new mail in /var/spool/mail/oracle
[oracle@zeroserv02 ~]$
5 – ASM Dependency
Since the ASM since the CRS depends on HAIP we need to remove this dependency. Here we have a tricky part. It is needed to completely change the dependencies for ASM, setting it to CRS/CSS directly.
First, at first node we check the current dependency for START and STOP:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res ora.asm -p |grep START_DEPENDENCIES=
START_DEPENDENCIES=hard(ora.ASMNET1LSNR_ASM.lsnr,ora.ASMNET2LSNR_ASM.lsnr) weak(ora.LISTENER.lsnr) pullup(ora.ASMNET1LSNR_ASM.lsnr,ora.ASMNET2LSNR_ASM.lsnr) dispersion:active(site:type:ora.asm.type)
[root@zeroserv01 ~]#
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res ora.asm -p |grep STOP_DEPENDENCIES=
STOP_DEPENDENCIES=hard(intermediate:ora.ASMNET1LSNR_ASM.lsnr,intermediate:ora.ASMNET2LSNR_ASM.lsnr)
[root@zeroserv01 ~]#
And now we change it. Look at the parameters values. They are completely different, and need to be with these specific values:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init
[root@zeroserv01 ~]#
If you see closely the HAIP it is not listed there as a dependency, setting with the values, we completely remove HAIP dependency and inheritance.
And we need to do the same for the other node:
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd) pullup(ora.cssd,ora.ctssd) weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init
[root@zeroserv02 ~]#
6 – Disable HAIP Resource
The next step is to disable the HAIP resource from startup at CRS. We do this in both nodes.
Node 1:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
[root@zeroserv01 ~]#
Node 2:
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
[root@zeroserv02 ~]#
But check that we do not stop yet the HAIP resource. It is needed to remain up at this moment.
If we check with the init for CRS, it is still there in node 1:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE zeroserv01 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE zeroserv01 STABLE
ora.crf
1 ONLINE ONLINE zeroserv01 STABLE
ora.crsd
1 ONLINE ONLINE zeroserv01 STABLE
ora.cssd
1 ONLINE ONLINE zeroserv01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE zeroserv01 STABLE
ora.ctssd
1 ONLINE ONLINE zeroserv01 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE zeroserv01 STABLE
ora.drivers.acfs
1 ONLINE ONLINE zeroserv01 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE zeroserv01 STABLE
ora.gipcd
1 ONLINE ONLINE zeroserv01 STABLE
ora.gpnpd
1 ONLINE ONLINE zeroserv01 STABLE
ora.mdnsd
1 ONLINE ONLINE zeroserv01 STABLE
ora.storage
1 ONLINE ONLINE zeroserv01 STABLE
--------------------------------------------------------------------------------
[root@zeroserv01 ~]#
And for the other node too:
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE zeroserv02 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE zeroserv02 STABLE
ora.crf
1 ONLINE ONLINE zeroserv02 STABLE
ora.crsd
1 ONLINE ONLINE zeroserv02 STABLE
ora.cssd
1 ONLINE ONLINE zeroserv02 STABLE
ora.cssdmonitor
1 ONLINE ONLINE zeroserv02 STABLE
ora.ctssd
1 ONLINE ONLINE zeroserv02 ACTIVE:0,STABLE
ora.diskmon
1 ONLINE ONLINE zeroserv02 STABLE
ora.drivers.acfs
1 ONLINE ONLINE zeroserv02 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE zeroserv02 STABLE
ora.gipcd
1 ONLINE ONLINE zeroserv02 STABLE
ora.gpnpd
1 ONLINE ONLINE zeroserv02 STABLE
ora.mdnsd
1 ONLINE ONLINE zeroserv02 STABLE
ora.storage
1 ONLINE ONLINE zeroserv02 STABLE
--------------------------------------------------------------------------------
[root@zeroserv02 ~]#
7 – Restart CRS
After we configure the dependencies for ASM we can restart CRS to shutfown HAIP (and test the changes that we made). I prefer to execute, first, in just one node and if everything goes well, I do for the others.
Stop Cluster
So, first the STOP CLUSTER in the first node:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -f
CRS-2673: Attempting to stop 'ora.crsd' on 'zeroserv01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'zeroserv01'
CRS-2673: Attempting to stop 'ora.catalog.raadmin.acfs' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.catalog.raosbadmin.acfs' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.DELTA.dg' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.CATALOG.dg' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.cvu' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'zeroserv01'
CRS-2677: Stop of 'ora.DELTA.dg' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.CATALOG.dg' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv01'
CRS-2677: Stop of 'ora.cvu' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.asm' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv01'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'zeroserv01'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'zeroserv01'
CRS-2677: Stop of 'ora.catalog.raosbadmin.acfs' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01'
CRS-2677: Stop of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.catalog.raadmin.acfs' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01'
CRS-2677: Stop of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'zeroserv01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.zeroserv01.vip' on 'zeroserv01'
CRS-2677: Stop of 'ora.scan3.vip' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.zeroserv01.vip' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'zeroserv01' succeeded
CRS-2675: Stop of 'ora.chad' on 'zeroserv02' failed
CRS-2679: Attempting to clean 'ora.chad' on 'zeroserv02'
CRS-2675: Stop of 'ora.chad' on 'zeroserv01' failed
CRS-2679: Attempting to clean 'ora.chad' on 'zeroserv01'
CRS-2681: Clean of 'ora.chad' on 'zeroserv02' succeeded
CRS-2681: Clean of 'ora.chad' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'zeroserv01'
CRS-2677: Stop of 'ora.mgmtdb' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'zeroserv01'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.qosmserver' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.cvu' on 'zeroserv02'
CRS-2672: Attempting to start 'ora.zeroserv01.vip' on 'zeroserv02'
CRS-2676: Start of 'ora.cvu' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.zeroserv01.vip' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv02'
CRS-2676: Start of 'ora.scan3.vip' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv02'
CRS-2676: Start of 'ora.MGMTLSNR' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'zeroserv02'
CRS-2676: Start of 'ora.qosmserver' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'zeroserv02' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.chad' on 'zeroserv02'
CRS-2676: Start of 'ora.chad' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'zeroserv01'
CRS-2677: Stop of 'ora.ons' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'zeroserv01'
CRS-2677: Stop of 'ora.net1.network' on 'zeroserv01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'zeroserv01' has completed
CRS-2677: Stop of 'ora.crsd' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.evmd' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.storage' on 'zeroserv01'
CRS-2677: Stop of 'ora.storage' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv01'
CRS-2677: Stop of 'ora.ctssd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.asm' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'zeroserv01'
CRS-2677: Stop of 'ora.cssd' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'zeroserv01'
CRS-2677: Stop of 'ora.diskmon' on 'zeroserv01' succeeded
[root@zeroserv01 ~]#
Stop/Start CRS
And if everything was successful, we can stop CRS. This is needed because HAIP is from init from CRS:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.crf' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zeroserv01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'zeroserv01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.crf' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'zeroserv01'
CRS-2677: Stop of 'ora.mdnsd' on 'zeroserv01' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'zeroserv01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zeroserv01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@zeroserv01 ~]#
After that we can start again CRS in this node:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@zeroserv01 ~]#
Check CRS Init
And after some time we can see if the HAIP not restarted (as expected):
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE zeroserv01 Started,STABLE
ora.cluster_interconnect.haip
1 OFFLINE OFFLINE STABLE
ora.crf
1 ONLINE ONLINE zeroserv01 STABLE
ora.crsd
1 ONLINE ONLINE zeroserv01 STABLE
ora.cssd
1 ONLINE ONLINE zeroserv01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE zeroserv01 STABLE
ora.ctssd
1 ONLINE ONLINE zeroserv01 ACTIVE:0,STABLE
ora.diskmon
1 ONLINE ONLINE zeroserv01 STABLE
ora.drivers.acfs
1 ONLINE ONLINE zeroserv01 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE zeroserv01 STABLE
ora.gipcd
1 ONLINE ONLINE zeroserv01 STABLE
ora.gpnpd
1 ONLINE ONLINE zeroserv01 STABLE
ora.mdnsd
1 ONLINE ONLINE zeroserv01 STABLE
ora.storage
1 ONLINE ONLINE zeroserv01 STABLE
--------------------------------------------------------------------------------
[root@zeroserv01 ~]#
As you can see above, the HAIP not started during the init of CRS. If the HAIP still up, please check the topic 5.
Other nodes
After that, we can do the same in the other nodes.
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -f
CRS-2673: Attempting to stop 'ora.crsd' on 'zeroserv02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'zeroserv02'
CRS-2673: Attempting to stop 'ora.DELTA.dg' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.catalog.raadmin.acfs' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.chad' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.catalog.raosbadmin.acfs' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv02'
CRS-2677: Stop of 'ora.DELTA.dg' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.dg' on 'zeroserv02'
CRS-2677: Stop of 'ora.CATALOG.dg' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.zeroserv02.vip' on 'zeroserv02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'zeroserv02'
CRS-2677: Stop of 'ora.catalog.raadmin.acfs' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02'
CRS-2677: Stop of 'ora.CATALOG.RAADMIN.advm' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.catalog.raosbadmin.acfs' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02'
CRS-2677: Stop of 'ora.CATALOG.RAOSBADMIN.advm' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'zeroserv02'
CRS-2677: Stop of 'ora.asm' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv02'
CRS-2677: Stop of 'ora.zeroserv02.vip' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.ASMNET2LSNR_ASM.lsnr' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.chad' on 'zeroserv02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'zeroserv01'
CRS-2672: Attempting to start 'ora.zeroserv02.vip' on 'zeroserv01'
CRS-2676: Start of 'ora.zeroserv02.vip' on 'zeroserv01' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'zeroserv01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv01'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'zeroserv01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'zeroserv02'
CRS-2677: Stop of 'ora.ons' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'zeroserv02'
CRS-2677: Stop of 'ora.net1.network' on 'zeroserv02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'zeroserv02' has completed
CRS-2677: Stop of 'ora.crsd' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.evmd' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.storage' on 'zeroserv02'
CRS-2677: Stop of 'ora.storage' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'zeroserv02'
CRS-2677: Stop of 'ora.ctssd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.asm' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'zeroserv02'
CRS-2677: Stop of 'ora.cssd' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'zeroserv02'
CRS-2677: Stop of 'ora.diskmon' on 'zeroserv02' succeeded
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.crf' on 'zeroserv02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'zeroserv02'
CRS-2677: Stop of 'ora.drivers.acfs' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.crf' on 'zeroserv02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'zeroserv02'
CRS-2677: Stop of 'ora.mdnsd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'zeroserv02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'zeroserv02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'zeroserv02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@zeroserv02 ~]#
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@zeroserv02 ~]#
[root@zeroserv02 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE zeroserv02 STABLE
ora.cluster_interconnect.haip
1 OFFLINE OFFLINE STABLE
ora.crf
1 ONLINE ONLINE zeroserv02 STABLE
ora.crsd
1 ONLINE ONLINE zeroserv02 STABLE
ora.cssd
1 ONLINE ONLINE zeroserv02 STABLE
ora.cssdmonitor
1 ONLINE ONLINE zeroserv02 STABLE
ora.ctssd
1 ONLINE ONLINE zeroserv02 ACTIVE:0,STABLE
ora.diskmon
1 ONLINE ONLINE zeroserv02 STABLE
ora.drivers.acfs
1 ONLINE ONLINE zeroserv02 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE zeroserv02 STABLE
ora.gipcd
1 ONLINE ONLINE zeroserv02 STABLE
ora.gpnpd
1 ONLINE ONLINE zeroserv02 STABLE
ora.mdnsd
1 ONLINE ONLINE zeroserv02 STABLE
ora.storage
1 ONLINE ONLINE zeroserv02 STABLE
--------------------------------------------------------------------------------
[root@zeroserv02 ~]#
8 – Check if everything is UP
After doing the changes in both nodes we can see if everything is up and running:
[root@zeroserv01 ~]# /u01/app/12.2.0.1/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ob_dbfs
OFFLINE OFFLINE zeroserv01 STABLE
OFFLINE OFFLINE zeroserv02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.ASMNET2LSNR_ASM.lsnr
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.CATALOG.RAADMIN.advm
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.CATALOG.RAOSBADMIN.advm
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.CATALOG.dg
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.DELTA.dg
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.catalog.raadmin.acfs
ONLINE ONLINE zeroserv01 mounted on /raacfs/r
aadmin,STABLE
ONLINE ONLINE zeroserv02 mounted on /raacfs/r
aadmin,STABLE
ora.catalog.raosbadmin.acfs
ONLINE ONLINE zeroserv01 mounted on /osbcat,S
TABLE
ONLINE ONLINE zeroserv02 mounted on /osbcat,S
TABLE
ora.chad
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.net1.network
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.ons
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
ora.proxy_advm
ONLINE ONLINE zeroserv01 STABLE
ONLINE ONLINE zeroserv02 STABLE
rep_dbfs
OFFLINE OFFLINE zeroserv01 STABLE
OFFLINE OFFLINE zeroserv02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE zeroserv02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE zeroserv01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE zeroserv01 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE zeroserv01 192.168.10.1 192.168
.10.2,STABLE
ora.asm
1 ONLINE ONLINE zeroserv01 Started,STABLE
2 ONLINE ONLINE zeroserv02 Started,STABLE
ora.cvu
1 ONLINE ONLINE zeroserv01 STABLE
ora.mgmtdb
1 ONLINE ONLINE zeroserv01 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE zeroserv01 STABLE
ora.scan1.vip
1 ONLINE ONLINE zeroserv02 STABLE
ora.scan2.vip
1 ONLINE ONLINE zeroserv01 STABLE
ora.scan3.vip
1 ONLINE ONLINE zeroserv01 STABLE
ora.zdlras.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.zeroserv01.vip
1 ONLINE ONLINE zeroserv01 STABLE
ora.zeroserv02.vip
1 ONLINE ONLINE zeroserv02 STABLE
--------------------------------------------------------------------------------
[root@zeroserv01 ~]#
As you can see, everything is up. ASM is online in both nodes, as the listeners for ASM too.
9 – Start Databases
To finish, we need to restart all databases that were stopped at topic 1.
[oracle@zeroserv01 ~]$ srvctl start database -d zdlras
[oracle@zeroserv01 ~]$
Clean up
After you complete the GI upgrade for 19c, you can remove the ASM Proxy init file (from $GI_HOME/dbs) that was created before. It is not needed anymore because the upgrade removes completely the HAIP from CRS because it is not in use.
Conclusion
As told before, HAIP is not supported by the Exadata environment. If you check the definition for HAIP and the hardware design for the Exadata/Engineering System, it is completely redundant. The InfiniBand network and the dual-channel/ports have the same effect (even better)
But unfortunately, HAIP can be up for several reasons (like ZDLRA) and because of the way that ASM Proxy starts and pick up the interconnect IP, HAIP can be selected. Remove HAIP it is a little tricky because of the ASM dependency, we need to set some specific parameters.
As you saw before, there are several steps. But you can use and adapt them to your environment. If it is Exadata/Engineering System (like ZDLRA) they can be used almost as-is (just check the IPs). If it is not, read and adapt.
References:
Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip (Doc ID 1210883.1)
HOWTO: Remove/Disable HAIP on Exadata (Doc ID 2524069.1)
Grid infrastructure (GI):HAIP on RDS is not supported (Doc ID 2328941.1)
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”
