Database
BUG – High “row Cache Mutex” Waits – Oracle 19C
Category: Database Author: Andre Luiz Dutra Ontalba (Board Member) Date: 4 years ago Comments: 0

BUG - High "row Cache Mutex" Waits - Oracle 19c

Today, during an analysis of a performance problem in Oracle 19c, I found an event strange that I had not seen before.
 
I started looking at the event “ROW CACHE MUTEX” with a large number of events in 1 hour of process execution.

 

 

We can see above in 1 hour of processing we had 2,184,882 events with consumption of 51.92% of DB time. 
 
 I can found this bug in Oracle Support
 
Bug 31933451 – High “row Cache Mutex” Waits (Doc ID 31933451.8)
 
Symptoms:
  • Hang (Involving Shared Resource)
  • Internal Error May Occur (ORA-600)
  • Waits for “Row Cache Mutex”
 
Description
Too many row cache objects are being copied or ‘cloned’.  This fix ensures that only the ‘hot’ objects are cloned, not every object on the hash chain.
 
This problem affects all versions 19.x. 
 
There is no workaround for this problem.
 
In version 21.1 is fixed the problem.
 
There is already a specific one-off patch for this released in December.
After the patch was applied, an AWR collection was performed again and we could see that it resolved with an impressive result.
We had a 95.72% reduction in the “ROW CACHE MUTEX” event contention.
 
I hope I helped with this article
 
Stay tuned, following on twitter @aontalba and on Linkedin
 
Andre Ontalba

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


Oracle Groundbreakers EMEA – Virtual Tour 2020 (LuxOUG)
Category: Database Author: Andre Luiz Dutra Ontalba (Board Member) Date: 4 years ago Comments: 0

Oracle Groundbreakers EMEA - Virtual Tour 2020 (LuxOUG)

This year is different, in all it’s aspects. For the OGB EMEA Virtual Tour 2020, this year is about diversity, is about #BackToBasics.
 
For two weeks, speakers from all over the world will come to our houses bringing one of the most valuable gift: their knowledge. Everyone will learn something, regardless how many years of experience we have or how “fresh and new” we are.
 
LuxOUG will be part of this great event. See the agenda and register for all events you wish to participate !!!

 

Link to event registration:  http://ogbemea.com/02-oct-2020/
 
See you at the event !!
 

opatch fails with Error: ‘Archive Action: Source file “$ORACLE_HOME/.patch_storage/…” does not exist.’
Category: Database,Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 4 years ago Comments: 0

opatch fails with Error: 'Archive Action: Source file "$ORACLE_HOME/.patch_storage/..." does not exist.'

 
Another quick article about a problem I had yesterday during an update patch in ODA. I found an error during patch in the Oracle Binary.

 

[Sep 14, 2020 2:40:58 PM] [INFO] add CopyAction for olsrelod.sql
[Sep 14, 2020 2:40:58 PM] [INFO] OPatchSessionHelper::sortOnOverlay() Given list - 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998
[Sep 14, 2020 2:40:58 PM] [INFO] size of PatchObject list: 8
[Sep 14, 2020 2:40:59 PM] [INFO] Patch 24425998:
Achive Action: Directory "/u01/app/oracle/product/12.1.0.2/dbhome_3/.patch_storage/24425998_Sep_28_2016_12_31_24" does not exists or is not readable.
'oracle.rdbms, 12.1.0.2.0': Cannot update file '/u01/app/oracle/product/12.1.0.2/dbhome_3/lib/libserver12.a' with '/ksfd.o'
[Sep 14, 2020 2:40:59 PM] [INFO] Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
The details are:

Patch 24425998:
Achive Action: Directory "/u01/app/oracle/product/12.1.0.2/dbhome_3/.patch_storage/24425998_Sep_28_2016_12_31_24" does not exists or is not readable.
'oracle.rdbms, 12.1.0.2.0': Cannot update file '/u01/app/oracle/product/12.1.0.2/dbhome_3/lib/libserver12.a' with '/ksfd.o'
[Sep 14, 2020 2:40:59 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
[Sep 14, 2020 2:40:59 PM] [INFO] --------------------------------------------------------------------------------
[Sep 14, 2020 2:40:59 PM] [INFO] The following warnings have occurred during OPatch execution:
[Sep 14, 2020 2:40:59 PM] [INFO] 1) OUI-67303:
Patches [ 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998 ] will be rolled back.
[Sep 14, 2020 2:40:59 PM] [INFO] 2) OUI-67303:
Patches [ 25897615 25034396 26477255 20370037 21688501 18430870 27435440 24425998 ] will be rolled back.
[Sep 14, 2020 2:40:59 PM] [INFO] --------------------------------------------------------------------------------
[Sep 14, 2020 2:40:59 PM] [INFO] Finishing UtilSession at Mon Sep 14 14:40:59 CEST 2020
[Sep 14, 2020 2:40:59 PM] [INFO] Log file location: /u01/app/oracle/product/12.1.0.2/dbhome_3/cfgtoollogs/opatchauto/core/opatch/opatch2020-09-14_14-39-20PM_1.log

 

According to the document: opatch fails with Error: ‘Archive Action: Source file “$ ORACLE_HOME / .patch_storage / …” does not exist.’ or ‘Achive Action: Directory “$ ORACLE_HOME / .patch_storage / …” does not exist or is not readable’. (Doc ID 1244414.1)
The reason for this is:
Files needed to rollback existing subset patch (es) are missing from $ORACLE_HOME/.patch_storage.

BACKGROUND
==============
When an Oracle software patch is installed, the first step is to place an unmodified copy of each affected $ORACLE_HOME file into a directory under the “$ORACLE_HOME/.patch_storage”.  These file copies will be used if the software patch is ever manually, or automatically rolled back.

 

Well after I saw this error, I was sure that nothing had been removed before the patch.
 
The only supported solutions:

 

  1. The missing directories and files can be restored from a backup of the ORACLE_HOME.
  2.  
  3. If no backups exist then re-install the $ORACLE_HOME.
  4.  
  5. Clone from another ORACLE_HOME of a like installation.

 

 

In my case I always have the backup of the following directories $ORACLE_HOME/inventory/oneoffs and    $ORACLE_HOME/.patch_storage.
 
I found the patch folder /.patch_storage/24425998_Sep_28_2016_12_31_24 that was needed to perform the rollback and after that the patch was applied successfully.
 
2020-09-15 07:47:04 Patch 29972716 is successfully already applied on the Home: /u01/app/oracle/product/12.1.0.2/db_home3
2020-09-15 07:47:04 SUCCESS: Successfully applied the patch on the Home : /u01/app/oracle/product/12.1.0.2/db_home4, /u01/app/oracle/product/12.1.0.2/db_home1, /u01/app/oracle/product/12.1.0.2/db_home3.

 

I hope this helps you!!!
 
Stay tuned, following on twitter @aontalba and on Linkedin

 

Andre Luiz Dutra Ontalba

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched
Category: Database,Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 4 years ago Comments: 0

rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched

Yesterday during an ODA upgrade, I came across an error during the cluster upgrade process, where this error was presented.

 

.
.
2020/03/03 14:34:00 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2020/03/03 14:34:04 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/03/03 14:34:32 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched.
CRS-4000: Command Start failed, or completed with errors.
2020/03/03 14:34:32 CLSRSC-511: failed to set Oracle Clusterware and ASM to rolling migration mode
Died at /u01/app/18.0.0.0/grid/crs/install/oraasm.pm line 1455.

 

Well following Oracle’s note rootupgrade.sh Fails with CRS-1136: Rejecting the rolling upgrade mode change because the cluster is being patched (Doc ID 2494827.1)    .
 
I found the solution to the problem by following the steps below.
 
Run the commands below to identify the versions of crs, releasepatch and softwarepatch to see if there are any differences.

 

bash-4.3# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [2660242823].
bash-4.3#

bash-4.3# crsctl query crs releasepatch
Oracle Clusterware release patch level is [1953265745] and the complete list of patches [23600818 26839277 27001739 27105253 27128906 27144050 27335416 ] have been applied on the local node.
bash-4.3#

bash-4.3# crsctl query crs softwarepatch
Oracle Clusterware patch level on node odatest1 is [1953265745]

 

We can see that the crs has a different version than the releasepatch and softwarepatch.
 
Well done that we will fix the problem.
 
1 – Run stop rollingpatch as root user, which will update OCR with correct values
<GRID_HOME>/bin/crsctl stop rollingpatch  

 

root@odatest1:~# /u01/app/12.1.0.2/grid/bin/crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [1953265745].
root@odatest1:~# 

 

2 – Verify software/release patch levels and retry rootupgrade.sh.

 

bash-4.3# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1953265745].
bash-4.3#

bash-4.3# crsctl query crs releasepatch
Oracle Clusterware release patch level is [1953265745] and the complete list of patches [23600818 26839277 27001739 27105253 27128906 27144050 27335416 ] have been applied on the local node.
bash-4.3#

bash-4.3# crsctl query crs softwarepatch
Oracle Clusterware patch level on node odatest1 is [1953265745]

 

root@odatest1:~# /u01/app/18.0.0.0/grid/rootupgrade.sh







.
.
2020/03/03 15:34:00 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2020/03/03 15:34:04 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/03/03 15:34:32 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2020/03/03 15:35:10 CLSRSC-482: Running command: '/u01/app/18.0.0.0/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.1.0.2/grid -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2020/03/03 15:34:20 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

.
.

2020/03/03 15:54:00 CLSRSC-595: Executing upgrade step 8 of 19: 'UpgradeNode'.

2020/03/03 15:54:04 CLSRSC-474: Initiating upgrade of resource types

2020/03/03 15:56:20 CLSRSC-475: Upgrade of resource types successfully initiated.

2020/03/03 15:56:44 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.

2020/03/03 15:57:05 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

 

 

I hope this helps you!!!

 

Stay tuned, following on twitter @aontalba and on Linkedin
 
Andre Luiz Dutra Ontalba

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


LuxOUG – Virtual Tech Days
Category: Database Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0
First LuxOUG Virtual Tech Days event for the Oracle community.
 
We will hold our first online event, covering various technologies such as DevOps, Engineered System, Middleware, Cloud and others.
 
 Event Schedule:

Day 1 – 22/06

Speaker: Toon Koppelaars
Session:  Database Core Performance Principles.
Speaker: Piet Visser
Session: Partitioning – Positives and Pitfalls. (database, development) – this one will include an SQL demo.
Speaker: Bruno Reis
Session: Beginner-friendly Python for Oracle DBAs.
Speaker: Heli Helskyaho
Session: Introduction to Machine Learning.

Day 2 – 23/06

Speaker: Kamran Agayev
Session: From DBA to Data Engineer – How to survive a career transition.
Speaker: Nikitas Xenakis
Session: Building a Highly Available & Scalable Logistics Platform with Oracle 19c & Goldengate 19c.
Speaker: Franky Weber
Session: Cheating your application code with Oracle Database.
Speaker: Rodrigo Jorge
Session: Scanning Oracle Database for Malicious Changes.

Day 3 – 24/06

Speaker: Sandesh Rao 
Session: Introduction to AutoML and Data Science using the Oracle Autonomous Database.
Speaker: Robert Marz
Session: Oracle Cloud Infrastructure – Network Setup for DBAs.
Speaker: Alex Zaballa
Session: Exploring All Options to Move your Oracle Databases to the Oracle Cloud.
Speaker: Mariami Kupatadze
Session: Main components, memory structures, physical and logical structures and more.

Day 4 – 25/06

 
Speaker: Erik Van Roon 
Session:  Handling errors during bulk DML operations.
Speaker: Mohamed Houri 
Session:  Cursor Optimization under Adaptive and Extended Cursor Sharing.
Speaker: Y V Ravi Kumar
Session:  Oracle Sharding Technical Deep Dive.
Speaker: Lonneke Dikmans
Session: Oracle Blockchain Platform – a case study.
===================================================================================
For registration and participation in the virtual event room, please CLICK HERE
 
Enrollment open until 20/06. 
 
After this period the event will be broadcast on our Youtube channel – CLICK HERE
 
We have split the videos by speaker this is the playlist for LuxOUG Virtual Tech Days: CLICK HERE
====================================================================================
 
Presentations for download :
 
Speaker: Heli Helskyaho – PDF
Speaker: Mariami Kupatadze – PDF
Speaker: Toon Koppelaars – PDF
Speaker: Y V Ravi Kumar – PDF
Speaker: Rodrigo Jorge – PDF
Speaker: Erik Van Roon – PDF and Scripts 
Speaker: Alex Zaballa – PDF
Speaker: Bruno Reis – PDF
Speaker: Robert Marz – PDF
Speaker: Mohamed Houri  – PDF
Speaker: Sandesh Rao – PDF
Speaker: Lonneke Dikmans – PDF
Speaker: Piet Visser – PDF
Speaker: Franky Weber – PDF
 
 
 
 
See you at the event
 
 
LuxOUG Board

DB_UNIQUE_NAME, PDB, and Data Guard
Category: Database Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

DB_UNIQUE_NAME, PDB, and Data Guard

When you change the parameters for the database is possible to specify the db_unique_name and allow more control where you want to apply/use it. This is very useful to limit the scope, but you need to be aware of some collateral effects. Even not present at the official doc, you can use it. But check here some details that you need to take care of.

 

 

Unplug and Plug

The environment below is a simple 19c single instance database. There, exists 3 PDB’s and you can see the db_unique_name for this CDB:

 

SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDBDG                          READ WRITE NO

         5 PDBMOVE                        READ WRITE NO

SQL> show parameter db_unique_name;




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_unique_name                       string      or19dg

SQL>

 

And we can connect in the PDBMOVE and change the parameters. First, as usual, normal set without any special parameter:

 

SQL> alter system set open_cursors = 300 scope = spfile;




System altered.




SQL>

 

But, we can specify the db_unique_name for alter system. ALTER SYSTEM… DB_UNIQUE_NAME = ‘VALUE’. Here you can see that I used the same that as defined for root level.

 

SQL> alter system set sessions = 100  scope = spfile db_unique_name = 'or19dg';




System altered.




SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'or19dg';




System altered.




SQL>

 

Detail #1 here, if you specify the DB_UNIQUE_NAME the scope can be JUST spfile. Otherwise you will receive error.

 

SQL> alter system set cursor_sharing = FORCE scope = both db_unique_name = 'or19dg';

alter system set cursor_sharing = FORCE scope = both db_unique_name = 'or19dg'

*

ERROR at line 1:

ORA-65147: DB_UNIQUE_NAME specified without SPFILE scope

SQL>

 

In this example I set two parameters for PDB level and you can see at PDB_SPFILE$ inside of root cdb that values was set:

 

SQL> alter session set container = cdb$root;




Session altered.




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2549618825;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2549618825

or19dg                         cursor_sharing                 'FORCE'                                  2549618825

or19dg                         sessions                       100                                      2549618825




SQL>

 

And after the reload of the database the values are set:

 

[oracle@orcl19p ~]$ srvctl stop database -d or19dg

[oracle@orcl19p ~]$ srvctl start database -d or19dg

[oracle@orcl19p ~]$

[oracle@orcl19p ~]$ sqlplus / as sysdba




SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 17:33:29 2020

Version 19.5.0.0.0




Copyright (c) 1982, 2019, Oracle.  All rights reserved.







Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.5.0.0.0




SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDBDG                          READ WRITE NO

         5 PDBMOVE                        READ WRITE NO

SQL> alter session set container = PDBMOVE;




Session altered.




SQL> show parameter cursor_sharing;




NAME                                 TYPE                             VALUE

------------------------------------ -------------------------------- ------------------------------

cursor_sharing                       string                           FORCE

SQL>

 

 

UNPLUG

But if I need to unplug the PDB, the values will be exported too, right? Yes, but no as you imagine.
So, doing a simple unplug to XML:

 

SQL> alter pluggable database PDBMOVE close immediate;




Pluggable database altered.




SQL> alter pluggable database PDBMOVE unplug into '/tmp/pdbmove_par.xml';




Pluggable database altered.




SQL> drop pluggable database PDBMOVE keep datafiles;




Pluggable database dropped.




SQL>

 

And checking the file for these parameters at generated XML:

 

[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep open_cursors

      <spfile>*.open_cursors=300#HWM:300,</spfile>

[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep cursor_sharing

      <spfile>*.cursor_sharing='FORCE'</spfile>

[oracle@orcl19p ~]$ cat /tmp/pdbmove_par.xml |grep sessions

      <spfile>*.sessions=100#HWM:100,flag:1,</spfile>

[oracle@orcl19p ~]$

 

As you can see here, the parameters was exported, but the parameter changed to “*”. The definition to specific db_unique_name was cleared.
And if we try to plug again the same PDB, we can see that parameter was loaded as “*”:

 

SQL> create  pluggable database PDBMOVE USING '/tmp/pdbmove_par.xml' NOCOPY TEMPFILE REUSE;




Pluggable database created.




SQL> col name format a30

SQL> col VALUE$ format a40

SQL> set linesize 250

SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;




    CON_ID       DBID    CON_UID GUID                             NAME

---------- ---------- ---------- -------------------------------- ------------------------------

         2  826464235  826464235 9B151A78B6DB533AE0533205A00AFC30 PDB$SEED

         3 1340021208 1340021208 9B155010BEAC661BE0533205A00AF21B PDBDG

         4 2549618825 3364812106 A7082B6C610C5E1DE0533205A00AF7FE PDBMOVE




SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDBDG                          READ WRITE NO

         4 PDBMOVE                        MOUNTED

SQL> alter pluggable database PDBMOVE open;




Pluggable database altered.




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 3364812106;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              cursor_sharing                 'FORCE'                                  3364812106

*                              open_cursors                   300                                      3364812106

*                              sessions                       100                                      3364812106




SQL>

 

DATAGUARD

As you can imagine, using the db_unique_name for an alter system can affect the dataguard environments. If you know, the db_unique_name is different between primary and standby. So, if you define some parameter, depending on the way that you made, after you, switchover/failover can occur that parameter returns to the default value.
It is not the case that this is wrong, maybe you need to set some parameters for one server/side in specific. Maybe because of less memory, less CPU power, or whatever the reason.
In this example, I have gold19c as primary, and golds19c as standby. And I have the GOLD19P as used PDB here:

 

[oracle@goldpn1 ~]$ dgmgrl sys/oracle@gold19c

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jun 1 19:13:39 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.




Welcome to DGMGRL, type "help" for information.

Connected to "GOLD19C"

Connected as SYSDBA.

DGMGRL> show configuration;




Configuration - gold19c




  Protection Mode: MaxAvailability

  Members:

  gold19c  - Primary database

    golds19c - Physical standby database




Fast-Start Failover:  Disabled




Configuration Status:

SUCCESS   (status updated 72 seconds ago)




DGMGRL> exit

[oracle@goldpn1 ~]$

[oracle@goldpn1 ~]$ sqlplus / as sysdba




SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 19:16:50 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2020, Oracle.  All rights reserved.







Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0




SQL> show parameter db_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_name                              string      gold19c

SQL> show parameter db_unique_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_unique_name                       string      GOLD19C

SQL>

SQL> show pdbs




    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 GOLD19P                        READ WRITE NO

SQL> col name format a30

SQL> col VALUE$ format a40

SQL> set linesize 250

SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;




    CON_ID       DBID    CON_UID GUID                             NAME

---------- ---------- ---------- -------------------------------- ------------------------------

         2 2661745537 2661745537 A6B13C68753F63D3E0530A07A00A6303 PDB$SEED

         3 2141275600 2141275600 A6B19FBF38306073E0530C07A00ACE20 GOLD19P




SQL>

 

And to set the parameters I used the same than before, with and without db_unique_name:

 

SQL> alter session set container = GOLD19P;




Session altered.




SQL> alter system set open_cursors = 300 scope = spfile;




System altered.




SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'GOLD19C';




System altered.




SQL>

 

And as you can see, the parameter was stored.

 

SQL> alter session set container = cdb$root;




Session altered.




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2141275600

GOLD19C                        cursor_sharing                 'FORCE'                                  2141275600




SQL>

 

But if I made the switchover:

 

[oracle@goldpn1 ~]$ dgmgrl sys/oracle@gold19c

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Jun 1 19:24:15 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.




Welcome to DGMGRL, type "help" for information.

Connected to "GOLD19C"

Connected as SYSDBA.

DGMGRL> switchover to golds19c;

Performing switchover NOW, please wait...

Operation requires a connection to database "golds19c"

Connecting ...

Connected to "GOLDS19C"

Connected as SYSDBA.

New primary database "golds19c" is opening...

Oracle Clusterware is restarting database "gold19c" ...

Connected to "GOLD19C"

Connected to "GOLD19C"

Switchover succeeded, new primary is "golds19c"

DGMGRL>

 

You can see that now, the value for the parameter cursor_sharing now have the default value because the db_unique_name does not hit what was defined:

 

[oracle@goldsn1 ~]$ sqlplus / as sysdba




SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 1 19:35:01 2020

Version 19.7.0.0.0




Copyright (c) 1982, 2020, Oracle.  All rights reserved.







Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0




SQL> show parameter db_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_name                              string      gold19c

SQL> show parameter db_unique_name




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_unique_name                       string      GOLDS19C

SQL>

SQL> alter session set container = GOLD19P;




Session altered.




SQL> show parameter cursor_sharing;




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

cursor_sharing                       string      EXACT

SQL> show parameter open_cursors;




NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

open_cursors                         integer     300

SQL>

 

But you can see that the parameter is defined at PDB_SPFILE$:

 

SQL> alter session set container = cdb$root;




Session altered.




SQL>

SQL> col name format a30

SQL> col VALUE$ format a40

SQL> set linesize 250

SQL> select CON_ID,DBID,CON_UID,GUID,NAME from v$pdbs;




    CON_ID       DBID    CON_UID GUID                             NAME

---------- ---------- ---------- -------------------------------- ------------------------------

         2 2661745537 2661745537 A6B13C68753F63D3E0530A07A00A6303 PDB$SEED

         3 2141275600 2141275600 A6B19FBF38306073E0530C07A00ACE20 GOLD19P




SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;




DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2141275600

GOLD19C                        cursor_sharing                 'FORCE'                                  2141275600




SQL>

 

As showed, the value at standby is not defied, but as told before this can be expected behaviour. Or no. If you want to set the parameter and it became value in both, primary and standby, you don’t need to specify the db_unique_name, otherwise when the switchover/failover occurs the value will not be there. And, this can lead to some unexpected behaviours (at application side as and example) and need to troubleshoot (and until find this about set parameter, can be a long path).

 

Others points

The db_unique_name option does not check what you specify, so, take care to set the correct value. As you can see below I defined it as SIMON, and was accepted and saved:

 

SQL> alter session set container = GOLD19P;

Session altered.

SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'SIMON';

System altered.

SQL> alter session set container = cdb$root;

Session altered.

SQL> select DB_UNIQ_NAME, NAME, VALUE$, PDB_UID from PDB_SPFILE$ where PDB_UID = 2141275600;

DB_UNIQ_NAME                   NAME                           VALUE$                                      PDB_UID

------------------------------ ------------------------------ ---------------------------------------- ----------

*                              open_cursors                   300                                      2141275600

GOLD19C                        cursor_sharing                 'FORCE'                                  2141275600

SIMON                          cursor_sharing                 'FORCE'                                  2141275600

SQL>

 

And to reset, the process is similar. As you see below, you can specify the db_unique_name to delete specific. But if you specify nothing, the reset will be done for all.

 

SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';

System altered.

SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';

alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C'

*

ERROR at line 1:

ORA-32010: cannot find entry to delete in SPFILE

SQL> alter system set cursor_sharing = FORCE scope = spfile db_unique_name = 'GOLD19C';

System altered.

SQL> alter system reset cursor_sharing scope = spfile;

System altered.

SQL> alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C';

alter system reset cursor_sharing scope = spfile db_unique_name = 'GOLD19C'

*

ERROR at line 1:

ORA-32010: cannot find entry to delete in SPFILE

SQL>

 

Conclusion

Sometimes a simple definition can lead to some strange behaviors. Think that your application is running ok and after a switchover change completely the way how it works (cursor_sharing as above). And until you dig the solution can take time.
Db_unique_name for set parameter needs to be used carefully to avoid these cases. It is not the case that you never need to use it, sometimes this definition needs to be used. You can use it to prepare everything BEFORE the switchover to avoid some error or to tune the database since the beginning.

References

Initialization parameters in a Multitenant database – FAQ and Examples (Doc ID 2101638.1)
Initialization parameters in a Multitenant database – Facts and additional information (Doc ID 2101596.1)
2.7 Managing Initialization Parameters Using a Server Parameter File

 

 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

 


ASM, REPLACE DISK Command
Category: Database Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ASM, REPLACE DISK Command

The REPLACE DISK command was released with 12.1 and allow to do an online replacement for a failed disk. This command is important because it reduces the rebalance time doing just the SYNC phase. Comparing with normal disk replacement (DROP and ADD in the same command), the REPLACE just do mirror resync.
Basically, when the REPLACE command is called, the rebalance just copy/sync the data from the survivor disk (the partner disk from the mirror). It is faster since the previous way with drop/add execute a complete rebalance from all AU of the diskgroup, doing REBALANCE and SYNC phase.
The replace disk command is important for the SWAP disk process for Exadata (where you add the new 14TB disks) since it is faster to do the rebalance of the diskgroup.
Below one example from this behavior. Look that AU from DISK01 was SYNCED with the new disk:
And compare with the previous DROP/ADD disk, where all AU from all disks was rebalanced:

Actual Environment And Simulate The failure

In this post, to simulate and show how the replace disk works I have the DATA diskgroup with 6 disks (DISK01-06). The DISK07 it is not in use.

 

SQL> select NAME,FAILGROUP,LABEL,PATH from v$asm_disk order by FAILGROUP, label;




NAME                           FAILGROUP                      LABEL      PATH

------------------------------ ------------------------------ ---------- -----------

DISK01                         FAIL01                         DISK01     ORCL:DISK01

DISK02                         FAIL01                         DISK02     ORCL:DISK02

DISK03                         FAIL02                         DISK03     ORCL:DISK03

DISK04                         FAIL02                         DISK04     ORCL:DISK04

DISK05                         FAIL03                         DISK05     ORCL:DISK05

DISK06                         FAIL03                         DISK06     ORCL:DISK06

RECI01                         RECI01                         RECI01     ORCL:RECI01

SYSTEMIDG01                    SYSTEMIDG01                    SYSI01     ORCL:SYSI01

                                                              DISK07     ORCL:DISK07




9 rows selected.




SQL>

 

And to simulate the error I disconnected the disk from Operational system (since I used iSCSI, I just log off the target for DISK02:

 

[root@asmrec ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.eff4683320e8 -p 172.16.0.3:3260 -u

Logging out of session [sid: 41, target: iqn.2006-01.com.openfiler:tsn.eff4683320e8, portal: 172.16.0.3,3260]

Logout of [sid: 41, target: iqn.2006-01.com.openfiler:tsn.eff4683320e8, portal: 172.16.0.3,3260] successful.

[root@asmrec ~]#

 

At the same moment, the alertlog from ASM detected the error and informed that the mirror was found in another disk (DISK06):

 

2020-03-29T00:42:11.160695+01:00

WARNING: Read Failed. group:3 disk:1 AU:29 offset:0 size:4096

path:ORCL:DISK02

         incarnation:0xf0f0c113 synchronous result:'I/O error'

         subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so krq:0x7f3df8db35b8 bufp:0x7f3df8c9c000 osderr1:0x3 osderr2:0x2e

         IO elapsed time: 0 usec Time waited on I/O: 0 usec

WARNING: cache failed reading from group=3(DATA) fn=8 blk=0 count=1 from disk=1 (DISK02) mirror=0 kfkist=0x20 status=0x02 osderr=0x3 file=kfc.c line=13317

WARNING: cache succeeded reading from group=3(DATA) fn=8 blk=0 count=1 from disk=5 (DISK06) mirror=1 kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=13366

 

So, at this moment the DQISK02 will not be removed instantly, but after the disk_repair_time finish:

 

WARNING: Started Drop Disk Timeout for Disk 1 (DISK02) in group 3 with a value 43200

WARNING: Disk 1 (DISK02) in group 3 will be dropped in: (43200) secs on ASM inst 1

cluster guid (e4db41a22bd95fc6bf79d2e2c93360c7) generated for PST Hbeat for instance 1

 

If you want to check the full output from ASM alertlog you can access here at ASM-ALERTLOG-Output-Online-Disk-Error.txt
So, the actual diskgroup is

 

SQL> select NAME,FAILGROUP,LABEL,PATH from v$asm_disk order by FAILGROUP, label;




NAME                           FAILGROUP                      LABEL      PATH

------------------------------ ------------------------------ ---------- -----------

DISK01                         FAIL01                         DISK01     ORCL:DISK01

DISK02                         FAIL01

DISK03                         FAIL02                         DISK03     ORCL:DISK03

DISK04                         FAIL02                         DISK04     ORCL:DISK04

DISK05                         FAIL03                         DISK05     ORCL:DISK05

DISK06                         FAIL03                         DISK06     ORCL:DISK06

RECI01                         RECI01                         RECI01     ORCL:RECI01

SYSTEMIDG01                    SYSTEMIDG01                    SYSI01     ORCL:SYSI01

                                                              DISK07     ORCL:DISK07




9 rows selected.




SQL>

 

REPLACE DISK

Since the old disk was lost (by HW or something similar), it is impossible to put it again online. A new disk was attached to the server (DISK07 in this example) and this is added in the diskgroup.
So, we just need to execute the REPLACE DISK command:

 

SQL> alter diskgroup DATA

  2  REPLACE DISK DISK02 with 'ORCL:DISK07'

  3  power 2;




Diskgroup altered.




SQL>

 

The command is easy, we replace disk failed disk with the new disk path. And it is possible to replace more than one at the same time and specify the power of the rebalance too.
At ASM alertlog we can see a lot of messages about this replacement, but look that resync of the disk. The full output can be found here at ASM-ALERTLOG-Output-Replace-Disk.txt
Some points here:

 

2020-03-29T00:44:31.602826+01:00

SQL> alter diskgroup DATA

replace disk DISK02 with 'ORCL:DISK07'

power 2

2020-03-29T00:44:31.741335+01:00

NOTE: cache closing disk 1 of grp 3: (not open) DISK02

2020-03-29T00:44:31.742068+01:00

NOTE: GroupBlock outside rolling migration privileged region

2020-03-29T00:44:31.742968+01:00

NOTE: client +ASM1:+ASM:asmrec no longer has group 3 (DATA) mounted

2020-03-29T00:44:31.746444+01:00

NOTE: Found ORCL:DISK07 for disk DISK02

NOTE: initiating resync of disk group 3 disks

DISK02 (1)




NOTE: process _user20831_+asm1 (20831) initiating offline of disk 1.4042309907 (DISK02) with mask 0x7e in group 3 (DATA) without client assisting

2020-03-29T00:44:31.747191+01:00

NOTE: sending set offline flag message (2044364809) to 1 disk(s) in group 3





2020-03-29T00:44:34.558097+01:00

NOTE: PST update grp = 3 completed successfully

2020-03-29T00:44:34.559806+01:00

SUCCESS: alter diskgroup DATA

replace disk DISK02 with 'ORCL:DISK07'

power 2

2020-03-29T00:44:36.805979+01:00

NOTE: Attempting voting file refresh on diskgroup DATA

NOTE: Refresh completed on diskgroup DATA. No voting file found.

2020-03-29T00:44:36.820900+01:00

NOTE: starting rebalance of group 3/0xf99030d7 (DATA) at power 2

 

After that, we can see the rebalance just take the SYNC phase:

 

SQL> select * from gv$asm_operation;




   INST_ID GROUP_NUMBER OPERA PASS      STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE     CON_ID

---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------

         1            3 REBAL COMPACT   WAIT          2          2          0          0          0           0                     0

         1            3 REBAL REBALANCE WAIT          2          2          0          0          0           0                     0

         1            3 REBAL REBUILD   WAIT          2          2          0          0          0           0                     0

         1            3 REBAL RESYNC    RUN           2          2        231       1350        513           2                     0




SQL>

SQL> /




   INST_ID GROUP_NUMBER OPERA PASS      STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE     CON_ID

---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------

         1            3 REBAL COMPACT   WAIT          2          2          0          0          0           0                     0

         1            3 REBAL REBALANCE WAIT          2          2          0          0          0           0                     0

         1            3 REBAL REBUILD   WAIT          2          2          0          0          0           0                     0

         1            3 REBAL RESYNC    RUN           2          2        373       1350        822           1                     0




SQL>

SQL> /




   INST_ID GROUP_NUMBER OPERA PASS      STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE     CON_ID

---------- ------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------

         1            3 REBAL COMPACT   REAP          2          2          0          0          0           0                     0

         1            3 REBAL REBALANCE DONE          2          2          0          0          0           0                     0

         1            3 REBAL REBUILD   DONE          2          2          0          0          0           0                     0

         1            3 REBAL RESYNC    DONE          2          2       1376       1350          0           0                     0




SQL> /




no rows selected




SQL>

 

In the end, after the rebalance we have:

 

SQL> select NAME,FAILGROUP,LABEL,PATH from v$asm_disk order by FAILGROUP, label;




NAME                           FAILGROUP                      LABEL      PATH

------------------------------ ------------------------------ ---------- -----------

DISK01                         FAIL01                         DISK01     ORCL:DISK01

DISK02                         FAIL01                         DISK07     ORCL:DISK07

DISK03                         FAIL02                         DISK03     ORCL:DISK03

DISK04                         FAIL02                         DISK04     ORCL:DISK04

DISK05                         FAIL03                         DISK05     ORCL:DISK05

DISK06                         FAIL03                         DISK06     ORCL:DISK06

RECI01                         RECI01                         RECI01     ORCL:RECI01

SYSTEMIDG01                    SYSTEMIDG01                    SYSI01     ORCL:SYSI01




8 rows selected.




SQL>

 

An important detail is that the NAME for the disk will not change, it is impossible to change using REPLACE DISK command. As you can see above, the disk named DISK02 has the label DISK07 (here this came from asmlib disk).

 

Know Issues

There is a known issue for REPLACE DISK for 18c and higher for GI where the rebalance can take AGES to finish. This occurs because (when replacing more than one disk per time), it executes the SYNC disk by disk. One example, for one Exadata the replace for a complete cell took more than 48 hours, while a DROP/ADD took just 12 hours for the same disks.
So, it is recommended to have the fix for Bug 30582481 and Bug 31062010 applied. The detail it is that patch 30582481 (Patch 30582481: ASM REPLACE DISK COMMAND EXECUTED ON ALL CELLDISKS OF A FAILGROUP, ASM RUNNING RESYNC ONE DISK AT A TIME) was withdraw and replaced by bug/patch 31062010 that it is not available (at the moment that I write this port – March 2020).
So, be careful to do this in one engineering system or when you need to replace a lot of disks at the same time.
Some reference for reading:
 
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


1 2 3 4 7