Oracle Groundbreakers EMEA - Virtual Tour 2020 (LuxOUG)
This year is different, in all it’s aspects. For the OGB EMEA Virtual Tour 2020, this year is about diversity, is about #BackToBasics.
For two weeks, speakers from all over the world will come to our houses bringing one of the most valuable gift: their knowledge. Everyone will learn something, regardless how many years of experience we have or how “fresh and new” we are.
LuxOUG will be part of this great event. See the agenda and register for all events you wish to participate !!!
Link to event registration: http://ogbemea.com/02-oct-2020/
See you at the event !!
opatch fails with Error: 'Archive Action: Source file "$ORACLE_HOME/.patch_storage/..." does not exist.'
Another quick article about a problem I had yesterday during an update patch in ODA. I found an error during patch in the Oracle Binary.
[Sep 14, 2020 2:40:58 PM] [INFO] add CopyAction for olsrelod.sql
[Sep 14, 2020 2:40:58 PM] [INFO] OPatchSessionHelper::sortOnOverlay() Given list - 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998
[Sep 14, 2020 2:40:58 PM] [INFO] size of PatchObject list: 8
[Sep 14, 2020 2:40:59 PM] [INFO] Patch 24425998:
Achive Action: Directory "/u01/app/oracle/product/12.1.0.2/dbhome_3/.patch_storage/24425998_Sep_28_2016_12_31_24" does not exists or is not readable.
'oracle.rdbms, 12.1.0.2.0': Cannot update file '/u01/app/oracle/product/12.1.0.2/dbhome_3/lib/libserver12.a' with '/ksfd.o'
[Sep 14, 2020 2:40:59 PM] [INFO] Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
The details are:
Patch 24425998:
Achive Action: Directory "/u01/app/oracle/product/12.1.0.2/dbhome_3/.patch_storage/24425998_Sep_28_2016_12_31_24" does not exists or is not readable.
'oracle.rdbms, 12.1.0.2.0': Cannot update file '/u01/app/oracle/product/12.1.0.2/dbhome_3/lib/libserver12.a' with '/ksfd.o'
[Sep 14, 2020 2:40:59 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
[Sep 14, 2020 2:40:59 PM] [INFO] --------------------------------------------------------------------------------
[Sep 14, 2020 2:40:59 PM] [INFO] The following warnings have occurred during OPatch execution:
[Sep 14, 2020 2:40:59 PM] [INFO] 1) OUI-67303:
Patches [ 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998 ] will be rolled back.
[Sep 14, 2020 2:40:59 PM] [INFO] 2) OUI-67303:
Patches [ 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998 ] will be rolled back.
[Sep 14, 2020 2:40:59 PM] [INFO] --------------------------------------------------------------------------------
[Sep 14, 2020 2:40:59 PM] [INFO] Finishing UtilSession at Mon Sep 14 14:40:59 CEST 2020
[Sep 14, 2020 2:40:59 PM] [INFO] Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_3/cfgtoollogs/opatchauto/core/opatch/opatch2020-09-14_14-39-20PM_1.log
According to the document: opatch fails with Error: ‘Archive Action: Source file “$ ORACLE_HOME / .patch_storage / …” does not exist.’ or ‘Achive Action: Directory “$ ORACLE_HOME / .patch_storage / …” does not exist or is not readable’. (Doc ID 1244414.1)
The reason for this is:
Files needed to rollback existing subset patch (es) are missing from $ORACLE_HOME/.patch_storage.
BACKGROUND
==============
When an Oracle software patch is installed, the first step is to place an unmodified copy of each affected $ORACLE_HOME file into a directory under the “$ORACLE_HOME/.patch_storage”. These file copies will be used if the software patch is ever manually, or automatically rolled back.
Well after I saw this error, I was sure that nothing had been removed before the patch.
The only supported solutions:
The missing directories and files can be restored from a backup of the ORACLE_HOME.
If no backups exist then re-install the $ORACLE_HOME.
Clone from another ORACLE_HOME of a like installation.
In my case I always have the backup of the following directories $ORACLE_HOME/inventory/oneoffs and $ORACLE_HOME/.patch_storage.
I found the patch folder /.patch_storage/24425998_Sep_28_2016_12_31_24 that was needed to perform the rollback and after that the patch was applied successfully.
2020-09-15 07:47:04 Patch 29972716 is successfully already applied on the Home: /u01/app/oracle/product/12.1.0.2/db_home3
2020-09-15 07:47:04 SUCCESS: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/db_home4, /u01/app/oracle/product/12.1.0.2/db_home1, /u01/app/oracle/product/12.1.0.2/db_home3.
I hope this helps you!!!
Stay tuned, following on twitter @aontalba and on Linkedin
Andre Luiz Dutra Ontalba
Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”
rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched
Yesterday during an ODA upgrade, I came across an error during the cluster upgrade process, where this error was presented.
.
.
2020/03/03 14:34:00 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2020/03/03 14:34:04 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/03/03 14:34:32 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched.
CRS-4000: Command Start failed, or completed with errors.
2020/03/03 14:34:32 CLSRSC-511: failed to set Oracle Clusterware and ASM to rolling migration mode
Died at /u01/app/18.0.0.0/grid/crs/install/oraasm.pm line 1455.
Well following Oracle’s note rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched (Doc ID 2494827.1) .
I found the solution to the problem by following the steps below.
Run the commands below to identify the versions of crs, releasepatch and softwarepatch to see if there are any differences.
bash-4.3# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [2660242823].
bash-4.3#
bash-4.3# crsctl query crs releasepatch
Oracle Clusterware release patch level is [1953265745] and the complete list of patches [23600818 26839277 27001739 27105253 27128906 27144050 27335416 ] have been applied on the local node.
bash-4.3#
bash-4.3# crsctl query crs softwarepatch
Oracle Clusterware patch level on node odatest1 is [1953265745]
We can see that the crs has a different version than the releasepatch and softwarepatch.
Well done that we will fix the problem.
1 – Run stop rollingpatch as root user, which will update OCR with correct values
<GRID_HOME>/bin/crsctl stop rollingpatch
root@odatest1:~# /u01/app/12.1.0.2/grid/bin/crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [1953265745].
root@odatest1:~#
2 – Verify software/release patch levels and retry rootupgrade.sh.
bash-4.3# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1953265745].
bash-4.3#
bash-4.3# crsctl query crs releasepatch
Oracle Clusterware release patch level is [1953265745] and the complete list of patches [23600818 26839277 27001739 27105253 27128906 27144050 27335416 ] have been applied on the local node.
bash-4.3#
bash-4.3# crsctl query crs softwarepatch
Oracle Clusterware patch level on node odatest1 is [1953265745]
root@odatest1:~# /u01/app/18.0.0.0/grid/rootupgrade.sh
.
.
2020/03/03 15:34:00 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2020/03/03 15:34:04 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/03/03 15:34:32 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2020/03/03 15:35:10 CLSRSC-482: Running command: '/u01/app/18.0.0.0/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.1.0.2/grid -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '
ASM configuration upgraded in local node successfully.
2020/03/03 15:34:20 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
.
.
2020/03/03 15:54:00 CLSRSC-595: Executing upgrade step 8 of 19: 'UpgradeNode'.
2020/03/03 15:54:04 CLSRSC-474: Initiating upgrade of resource types
2020/03/03 15:56:20 CLSRSC-475: Upgrade of resource types successfully initiated.
2020/03/03 15:56:44 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2020/03/03 15:57:05 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
I hope this helps you!!!
Stay tuned, following on twitter @aontalba and on Linkedin
Andre Luiz Dutra Ontalba
Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”
First LuxOUG Virtual Tech Days event for the Oracle community.
We will hold our first online event, covering various technologies such as DevOps, Engineered System, Middleware, Cloud and others.
Event Schedule:
Day 1 – 22/06
Speaker: Toon Koppelaars
Session: Database Core Performance Principles.
Speaker: Piet Visser
Session: Partitioning – Positives and Pitfalls. (database, development) – this one will include an SQL demo.
Speaker: Bruno Reis
Session: Beginner-friendly Python for Oracle DBAs.
Speaker: Heli Helskyaho
Session: Introduction to Machine Learning.
Day 2 – 23/06
Speaker: Kamran Agayev
Session: From DBA to Data Engineer – How to survive a career transition.
Speaker: Nikitas Xenakis
Session: Building a Highly Available & Scalable Logistics Platform with Oracle 19c & Goldengate 19c.
Speaker: Franky Weber
Session: Cheating your application code with Oracle Database.
Speaker: Rodrigo Jorge
Session: Scanning Oracle Database for Malicious Changes.
Day 3 – 24/06
Speaker: Sandesh Rao
Session: Introduction to AutoML and Data Science using the Oracle Autonomous Database.
Speaker: Robert Marz
Session: Oracle Cloud Infrastructure – Network Setup for DBAs.
Speaker: Alex Zaballa
Session: Exploring All Options to Move your Oracle Databases to the Oracle Cloud.
Speaker: Mariami Kupatadze
Session: Main components, memory structures, physical and logical structures and more.
Day 4 – 25/06
Speaker: Erik Van Roon
Session: Handling errors during bulk DML operations.
Speaker: Mohamed Houri
Session: Cursor Optimization under Adaptive and Extended Cursor Sharing.
Speaker: Y V Ravi Kumar
Session: Oracle Sharding Technical Deep Dive.
Speaker: Lonneke Dikmans
Session: Oracle Blockchain Platform – a case study.
===================================================================================
For registration and participation in the virtual event room, please CLICK HERE
Enrollment open until 20/06.
After this period the event will be broadcast on our Youtube channel – CLICK HERE
We have split the videos by speaker this is the playlist for LuxOUG Virtual Tech Days: CLICK HERE
====================================================================================
Presentations for download :
Speaker: Heli Helskyaho – PDF
Speaker: Mariami Kupatadze – PDF
Speaker: Toon Koppelaars – PDF
Speaker: Y V Ravi Kumar – PDF
Speaker: Rodrigo Jorge – PDF
Speaker: Erik Van Roon – PDF and Scripts
Speaker: Alex Zaballa – PDF
Speaker: Bruno Reis – PDF
Speaker: Robert Marz – PDF
Speaker: Mohamed Houri – PDF
Speaker: Sandesh Rao – PDF
Speaker: Lonneke Dikmans – PDF
Speaker: Piet Visser – PDF
Speaker: Franky Weber – PDF
See you at the event
LuxOUG Board
DB_UNIQUE_NAME, PDB, and Data Guard
When you change the parameters for the database is possible to specify the db_unique_name and allow more control where you want to apply/use it. This is very useful to limit the scope, but you need to be aware of some collateral effects. Even not present at the official doc, you can use it. But check here some details that you need to take care of.
Unplug and Plug
The environment below is a simple 19c single instance database. There, exists 3 PDB’s and you can see the db_unique_name for this CDB:
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDBDG READ WRITE NO
5 PDBMOVE READ WRITE NO
SQL> show parameter db_unique_name;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_unique_name string or19dg
SQL>
And we can connect in the PDBMOVE and change the parameters. First, as usual, normal set without any special parameter:
SQL> alter system set open_cursors = 300 scope = spfile;
System altered.
SQL>
But, we can specify the db_unique_name for alter system. ALTER SYSTEM… DB_UNIQUE_NAME = ‘VALUE’. Here you can see that I used the same that as defined for root level.
SQL> alter system set sessions = 100 scope = spfile db_unique_name = 'or19dg';
System altered.
SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'or19dg';
System altered.
SQL>
Detail #1 here, if you specify the DB_UNIQUE_NAME the scope can be JUST spfile. Otherwise you will receive error.
SQL> alter system set cursor_sharing = FORCE scope = both db_unique_name = 'or19dg';
alter system set cursor_sharing = FORCE scope = both db_unique_name = 'or19dg'
*
ERROR at line 1:
ORA-65147: DB_UNIQUE_NAME specified without SPFILE scope
SQL>
In this example I set two parameters for PDB level and you can see at PDB_SPFILE$ inside of root cdb that values was set:
SQL> alter session set container = cdb$root;
Session altered.
SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2549618825;
DB_UNIQ_NAME NAME VALUE$ PDB_UID
------------------------------ ------------------------------ ---------------------------------------- ----------
* open_cursors 300 2549618825
or19dg cursor_sharing 'FORCE' 2549618825
or19dg sessions 100 2549618825
SQL>
And after the reload of the database the values are set:
[oracle@orcl19p ~]$ srvctl stop database -d or19dg
[oracle@orcl19p ~]$ srvctl start database -d or19dg
[oracle@orcl19p ~]$
[oracle@orcl19p ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 17:33:29 2020
Version 19.5.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.5.0.0.0
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDBDG READ WRITE NO
5 PDBMOVE READ WRITE NO
SQL> alter session set container = PDBMOVE;
Session altered.
SQL> show parameter cursor_sharing;
NAME TYPE VALUE
------------------------------------ -------------------------------- ------------------------------
cursor_sharing string FORCE
SQL>
UNPLUG
But if I need to unplug the PDB, the values will be exported too, right? Yes, but no as you imagine.
So, doing a simple unplug to XML:
SQL> alter pluggable database PDBMOVE close immediate;
Pluggable database altered.
SQL> alter pluggable database PDBMOVE unplug into '/tmp/pdbmove_par.xml';
Pluggable database altered.
SQL> drop pluggable database PDBMOVE keep datafiles;
Pluggable database dropped.
SQL>
And checking the file for these parameters at generated XML:
[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep open_cursors
<spfile>*.open_cursors=300#HWM:300,</spfile>
[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep cursor_sharing
<spfile>*.cursor_sharing='FORCE'</spfile>
[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep sessions
<spfile>*.sessions=100#HWM:100,flag:1,</spfile>
[oracle@orcl19p ~]$
As you can see here, the parameters was exported, but the parameter changed to “*”. The definition to specific db_unique_name was cleared.
And if we try to plug again the same PDB, we can see that parameter was loaded as “*”:
SQL> create pluggable database PDBMOVE USING '/tmp/pdbmove_par.xml' NOCOPY TEMPFILE REUSE;
Pluggable database created.
SQL> col name format a30
SQL> col VALUE$ format a40
SQL> set linesize 250
SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;
CON_ID DBID CON_UID GUID NAME
---------- ---------- ---------- -------------------------------- ------------------------------
2 826464235 826464235 9B151A78B6DB533AE0533205A00AFC30 PDB$SEED
3 1340021208 1340021208 9B155010BEAC661BE0533205A00AF21B PDBDG
4 2549618825 3364812106 A7082B6C610C5E1DE0533205A00AF7FE PDBMOVE
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDBDG READ WRITE NO
4 PDBMOVE MOUNTED
SQL> alter pluggable database PDBMOVE open;
Pluggable database altered.
SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 3364812106;
DB_UNIQ_NAME NAME VALUE$ PDB_UID
------------------------------ ------------------------------ ---------------------------------------- ----------
* cursor_sharing 'FORCE' 3364812106
* open_cursors 300 3364812106
* sessions 100 3364812106
SQL>
DATAGUARD
As you can imagine, using the db_unique_name for an alter system can affect the dataguard environments. If you know, the db_unique_name is different between primary and standby. So, if you define some parameter, depending on the way that you made, after you, switchover/failover can occur that parameter returns to the default value.
It is not the case that this is wrong, maybe you need to set some parameters for one server/side in specific. Maybe because of less memory, less CPU power, or whatever the reason.
In this example, I have gold19c as primary, and golds19c as standby. And I have the GOLD19P as used PDB here:
[oracle@goldpn1 ~]$ dgmgrl sys/oracle@gold19c
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jun 1 19:13:39 2020
Version 19.7.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "GOLD19C"
Connected as SYSDBA.
DGMGRL> show configuration;
Configuration - gold19c
Protection Mode: MaxAvailability
Members:
gold19c - Primary database
golds19c - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 72 seconds ago)
DGMGRL> exit
[oracle@goldpn1 ~]$
[oracle@goldpn1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 19:16:50 2020
Version 19.7.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0
SQL> show parameter db_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_name string gold19c
SQL> show parameter db_unique_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_unique_name string GOLD19C
SQL>
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 GOLD19P READ WRITE NO
SQL> col name format a30
SQL> col VALUE$ format a40
SQL> set linesize 250
SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;
CON_ID DBID CON_UID GUID NAME
---------- ---------- ---------- -------------------------------- ------------------------------
2 2661745537 2661745537 A6B13C68753F63D3E0530A07A00A6303 PDB$SEED
3 2141275600 2141275600 A6B19FBF38306073E0530C07A00ACE20 GOLD19P
SQL>
And to set the parameters I used the same than before, with and without db_unique_name:
SQL> alter session set container = GOLD19P;
Session altered.
SQL> alter system set open_cursors = 300 scope = spfile;
System altered.
SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'GOLD19C';
System altered.
SQL>
And as you can see, the parameter was stored.
SQL> alter session set container = cdb$root;
Session altered.
SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;
DB_UNIQ_NAME NAME VALUE$ PDB_UID
------------------------------ ------------------------------ ---------------------------------------- ----------
* open_cursors 300 2141275600
GOLD19C cursor_sharing 'FORCE' 2141275600
SQL>
But if I made the switchover:
[oracle@goldpn1 ~]$ dgmgrl sys/oracle@gold19c
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jun 1 19:24:15 2020
Version 19.7.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "GOLD19C"
Connected as SYSDBA.
DGMGRL> switchover to golds19c;
Performing switchover NOW, please wait...
Operation requires a connection to database "golds19c"
Connecting ...
Connected to "GOLDS19C"
Connected as SYSDBA.
New primary database "golds19c" is opening...
Oracle Clusterware is restarting database "gold19c" ...
Connected to "GOLD19C"
Connected to "GOLD19C"
Switchover succeeded, new primary is "golds19c"
DGMGRL>
You can see that now, the value for the parameter cursor_sharing now have the default value because the db_unique_name does not hit what was defined:
[oracle@goldsn1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 19:35:01 2020
Version 19.7.0.0.0
Copyright (c) 1982, 2020, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.7.0.0.0
SQL> show parameter db_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_name string gold19c
SQL> show parameter db_unique_name
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_unique_name string GOLDS19C
SQL>
SQL> alter session set container = GOLD19P;
Session altered.
SQL> show parameter cursor_sharing;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cursor_sharing string EXACT
SQL> show parameter open_cursors;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
open_cursors integer 300
SQL>
But you can see that the parameter is defined at PDB_SPFILE$:
SQL> alter session set container = cdb$root;
Session altered.
SQL>
SQL> col name format a30
SQL> col VALUE$ format a40
SQL> set linesize 250
SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;
CON_ID DBID CON_UID GUID NAME
---------- ---------- ---------- -------------------------------- ------------------------------
2 2661745537 2661745537 A6B13C68753F63D3E0530A07A00A6303 PDB$SEED
3 2141275600 2141275600 A6B19FBF38306073E0530C07A00ACE20 GOLD19P
SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;
DB_UNIQ_NAME NAME VALUE$ PDB_UID
------------------------------ ------------------------------ ---------------------------------------- ----------
* open_cursors 300 2141275600
GOLD19C cursor_sharing 'FORCE' 2141275600
SQL>
As showed, the value at standby is not defied, but as told before this can be expected behaviour. Or no. If you want to set the parameter and it became value in both, primary and standby, you don’t need to specify the db_unique_name, otherwise when the switchover/failover occurs the value will not be there. And, this can lead to some unexpected behaviours (at application side as and example) and need to troubleshoot (and until find this about set parameter, can be a long path).
Others points
The db_unique_name option does not check what you specify, so, take care to set the correct value. As you can see below I defined it as SIMON, and was accepted and saved:
SQL> alter session set container = GOLD19P;
Session altered.
SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'SIMON';
System altered.
SQL> alter session set container = cdb$root;
Session altered.
SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;
DB_UNIQ_NAME NAME VALUE$ PDB_UID
------------------------------ ------------------------------ ---------------------------------------- ----------
* open_cursors 300 2141275600
GOLD19C cursor_sharing 'FORCE' 2141275600
SIMON cursor_sharing 'FORCE' 2141275600
SQL>
And to reset, the process is similar. As you see below, you can specify the db_unique_name to delete specific. But if you specify nothing, the reset will be done for all.
SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';
System altered.
SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';
alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C'
*
ERROR at line 1:
ORA-32010: cannot find entry to delete in SPFILE
SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'GOLD19C';
System altered.
SQL> alter system reset cursor_sharing scope = spfile;
System altered.
SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';
alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C'
*
ERROR at line 1:
ORA-32010: cannot find entry to delete in SPFILE
SQL>
Conclusion
Sometimes a simple definition can lead to some strange behaviors. Think that your application is running ok and after a switchover change completely the way how it works (cursor_sharing as above). And until you dig the solution can take time.
Db_unique_name for set parameter needs to be used carefully to avoid these cases. It is not the case that you never need to use it, sometimes this definition needs to be used. You can use it to prepare everything BEFORE the switchover to avoid some error or to tune the database since the beginning.
References
Initialization parameters in a Multitenant database – FAQ and Examples (Doc ID 2101638.1)
Initialization parameters in a Multitenant database – Facts and additional information (Doc ID 2101596.1)
2.7 Managing Initialization Parameters Using a Server Parameter File
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”
ASM, REPLACE DISK Command
The REPLACE DISK command was released with 12.1 and allow to do an online replacement for a failed disk. This command is important because it reduces the rebalance time doing just the SYNC phase. Comparing with normal disk replacement (DROP and ADD in the same command), the REPLACE just do mirror resync.
Basically, when the REPLACE command is called, the rebalance just copy/sync the data from the survivor disk (the partner disk from the mirror). It is faster since the previous way with drop/add execute a complete rebalance from all AU of the diskgroup, doing REBALANCE and SYNC phase.
The replace disk command is important for the SWAP disk process for Exadata (where you add the new 14TB disks) since it is faster to do the rebalance of the diskgroup.
Below one example from this behavior. Look that AU from DISK01 was SYNCED with the new disk:
And compare with the previous DROP/ADD disk, where all AU from all disks was rebalanced:
Actual Environment And Simulate The failure
In this post, to simulate and show how the replace disk works I have the DATA diskgroup with 6 disks (DISK01-06). The DISK07 it is not in use.
SQL> select NAME,FAILGROUP,LABEL,PATH from v$asm_disk order by FAILGROUP, label;
NAME FAILGROUP LABEL PATH
------------------------------ ------------------------------ ---------- -----------
DISK01 FAIL01 DISK01 ORCL:DISK01
DISK02 FAIL01 DISK02 ORCL:DISK02
DISK03 FAIL02 DISK03 ORCL:DISK03
DISK04 FAIL02 DISK04 ORCL:DISK04
DISK05 FAIL03 DISK05 ORCL:DISK05
DISK06 FAIL03 DISK06 ORCL:DISK06
RECI01 RECI01 RECI01 ORCL:RECI01
SYSTEMIDG01 SYSTEMIDG01 SYSI01 ORCL:SYSI01
DISK07 ORCL:DISK07
9 rows selected.
SQL>
And to simulate the error I disconnected the disk from Operational system (since I used iSCSI, I just log off the target for DISK02:
[root@asmrec ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.eff4683320e8 -p 172.16.0.3:3260 -u
Logging out of session [sid: 41, target: iqn.2006-01.com.openfiler:tsn.eff4683320e8, portal: 172.16.0.3,3260]
Logout of [sid: 41, target: iqn.2006-01.com.openfiler:tsn.eff4683320e8, portal: 172.16.0.3,3260] successful.
[root@asmrec ~]#
At the same moment, the alertlog from ASM detected the error and informed that the mirror was found in another disk (DISK06):
2020-03-29T00:42:11.160695+01:00
WARNING: Read Failed. group:3 disk:1 AU:29 offset:0 size:4096
path:ORCL:DISK02
incarnation:0xf0f0c113 synchronous result:'I/O error'
subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f3df8db35b8 bufp:0x7f3df8c9c000 osderr1:0x3 osderr2:0x2e
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: cache failed reading from group=3(DATA) fn=8 blk=0 count=1 from disk=1 (DISK02) mirror=0 kfkist=0x20 status=0x02 osderr=0x3 file=kfc.c line=13317
WARNING: cache succeeded reading from group=3(DATA) fn=8 blk=0 count=1 from disk=5 (DISK06) mirror=1 kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=13366
So, at this moment the DQISK02 will not be removed instantly, but after the disk_repair_time finish:
WARNING: Started Drop Disk Timeout for Disk 1 (DISK02) in group 3 with a value 43200
WARNING: Disk 1 (DISK02) in group 3 will be dropped in: (43200) secs on ASM inst 1
cluster guid (e4db41a22bd95fc6bf79d2e2c93360c7) generated for PST Hbeat for instance 1
If you want to check the full output from ASM alertlog you can access here at ASM-ALERTLOG-Output-Online-Disk-Error.txt
So, the actual diskgroup is
SQL> select NAME,FAILGROUP,LABEL,PATH from v$asm_disk order by FAILGROUP, label;
NAME FAILGROUP LABEL PATH
------------------------------ ------------------------------ ---------- -----------
DISK01 FAIL01 DISK01 ORCL:DISK01
DISK02 FAIL01
DISK03 FAIL02 DISK03 ORCL:DISK03
DISK04 FAIL02 DISK04 ORCL:DISK04
DISK05 FAIL03 DISK05 ORCL:DISK05
DISK06 FAIL03 DISK06 ORCL:DISK06
RECI01 RECI01 RECI01 ORCL:RECI01
SYSTEMIDG01 SYSTEMIDG01 SYSI01 ORCL:SYSI01
DISK07 ORCL:DISK07
9 rows selected.
SQL>
REPLACE DISK
Since the old disk was lost (by HW or something similar), it is impossible to put it again online. A new disk was attached to the server (DISK07 in this example) and this is added in the diskgroup.
So, we just need to execute the REPLACE DISK command:
SQL> alter diskgroup DATA
2 REPLACE DISK DISK02 with 'ORCL:DISK07'
3 power 2;
Diskgroup altered.
SQL>
The command is easy, we replace disk failed disk with the new disk path. And it is possible to replace more than one at the same time and specify the power of the rebalance too.
At ASM alertlog we can see a lot of messages about this replacement, but look that resync of the disk. The full output can be found here at ASM-ALERTLOG-Output-Replace-Disk.txt
Some points here:
2020-03-29T00:44:31.602826+01:00
SQL> alter diskgroup DATA
replace disk DISK02 with 'ORCL:DISK07'
power 2
2020-03-29T00:44:31.741335+01:00
NOTE: cache closing disk 1 of grp 3: (not open) DISK02
2020-03-29T00:44:31.742068+01:00
NOTE: GroupBlock outside rolling migration privileged region
2020-03-29T00:44:31.742968+01:00
NOTE: client +ASM1:+ASM:asmrec no longer has group 3 (DATA) mounted
2020-03-29T00:44:31.746444+01:00
NOTE: Found ORCL:DISK07 for disk DISK02
NOTE: initiating resync of disk group 3 disks
DISK02 (1)
NOTE: process _user20831_+asm1 (20831) initiating offline of disk 1.4042309907 (DISK02) with mask 0x7e in group 3 (DATA) without client assisting
2020-03-29T00:44:31.747191+01:00
NOTE: sending set offline flag message (2044364809) to 1 disk(s) in group 3
…
…
2020-03-29T00:44:34.558097+01:00
NOTE: PST update grp = 3 completed successfully
2020-03-29T00:44:34.559806+01:00
SUCCESS: alter diskgroup DATA
replace disk DISK02 with 'ORCL:DISK07'
power 2
2020-03-29T00:44:36.805979+01:00
NOTE: Attempting voting file refresh on diskgroup DATA
NOTE: Refresh completed on diskgroup DATA. No voting file found.
2020-03-29T00:44:36.820900+01:00
NOTE: starting rebalance of group 3/0xf99030d7 (DATA) at power 2
After that, we can see the rebalance just take the SYNC phase:
SQL> select * from gv$asm_operation;
INST_ID GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE CON_ID
---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------
1 3 REBAL COMPACT WAIT 2 2 0 0 0 0 0
1 3 REBAL REBALANCE WAIT 2 2 0 0 0 0 0
1 3 REBAL REBUILD WAIT 2 2 0 0 0 0 0
1 3 REBAL RESYNC RUN 2 2 231 1350 513 2 0
SQL>
SQL> /
INST_ID GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE CON_ID
---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------
1 3 REBAL COMPACT WAIT 2 2 0 0 0 0 0
1 3 REBAL REBALANCE WAIT 2 2 0 0 0 0 0
1 3 REBAL REBUILD WAIT 2 2 0 0 0 0 0
1 3 REBAL RESYNC RUN 2 2 373 1350 822 1 0
SQL>
SQL> /
INST_ID GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE CON_ID
---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------
1 3 REBAL COMPACT REAP 2 2 0 0 0 0 0
1 3 REBAL REBALANCE DONE 2 2 0 0 0 0 0
1 3 REBAL REBUILD DONE 2 2 0 0 0 0 0
1 3 REBAL RESYNC DONE 2 2 1376 1350 0 0 0
SQL> /
no rows selected
SQL>
In the end, after the rebalance we have:
SQL> select NAME,FAILGROUP,LABEL,PATH from v$asm_disk order by FAILGROUP, label;
NAME FAILGROUP LABEL PATH
------------------------------ ------------------------------ ---------- -----------
DISK01 FAIL01 DISK01 ORCL:DISK01
DISK02 FAIL01 DISK07 ORCL:DISK07
DISK03 FAIL02 DISK03 ORCL:DISK03
DISK04 FAIL02 DISK04 ORCL:DISK04
DISK05 FAIL03 DISK05 ORCL:DISK05
DISK06 FAIL03 DISK06 ORCL:DISK06
RECI01 RECI01 RECI01 ORCL:RECI01
SYSTEMIDG01 SYSTEMIDG01 SYSI01 ORCL:SYSI01
8 rows selected.
SQL>
An important detail is that the NAME for the disk will not change, it is impossible to change using REPLACE DISK command. As you can see above, the disk named DISK02 has the label DISK07 (here this came from asmlib disk).
Know Issues
There is a known issue for REPLACE DISK for 18c and higher for GI where the rebalance can take AGES to finish. This occurs because (when replacing more than one disk per time), it executes the SYNC disk by disk. One example, for one Exadata the replace for a complete cell took more than 48 hours, while a DROP/ADD took just 12 hours for the same disks.
So, it is recommended to have the fix for Bug 30582481 and Bug 31062010 applied. The detail it is that patch 30582481 (Patch 30582481: ASM REPLACE DISK COMMAND EXECUTED ON ALL CELLDISKS OF A FAILGROUP, ASM RUNNING RESYNC ONE DISK AT A TIME) was withdraw and replaced by bug/patch 31062010 that it is not available (at the moment that I write this port – March 2020).
So, be careful to do this in one engineering system or when you need to replace a lot of disks at the same time.
Some reference for reading:
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”