ZDLRA, ORDERING_WAIT task state
Tasks for ZDLRA are the pillar of how the backups are processed, everything is one task. So, when you ingest incremental backup one task is created but can occur that it get a freeze at ORDERING_WAIT state. These tasks are hard to identify and can create a big problem for your virtual full backup and backup strategy. Below I will show how they occur and how to solve the problem.
Incremental backups
To understand how they appear I need to show a little how the incremental backups work. Basically, look at the example below and one datafile with some blocks. When you do backup level 0, all the blocks are copied, and if I do backup incremental level 1 just the “new blocks” are copied (just blocks that changed).
As you can see above, the first backup of datafile (level 0), have SCN 3333. So, the consequent backup level 1 will copy everything that changed from that (and in this example have SCN 4444). And, the next incremental backup will pick up everything since the last backup, in this example, every block change since SCN 4444. So, it will generate a new backup with/until SCN 5555.
As you know, this is the definition of incremental backup. As you can see in the definition at “About RMAN Incremental Backups” from docs it is:
“An incremental backup copies only those data blocks that have changed since a previous backup. You can use RMAN to create incremental backups of data files, tablespaces, or the whole database.”
But the point it is that database/rman knows the scn from the last backup, and when does incremental backup it copy everything since from the last scn. Each incremental backup “knows” (internally, with the database blocks that are inside) the start scn and endpoint scn. So, to “reconstruct” the datafile, the database/rman uses the full backup and all subsequent incremental backups.
Incremental backup and ZDLRA
As I already wrote about incremental backups and ZDLRA, they are used to construct the “Virtual Full Backup”. In a very resumed way, ZDLRA merge the stored backups and create the virtual full backup (as I explained here too).
But even with this virtual backup, the way how the incremental backups work not change. The procedure is the same, check the scn from the last backup and copy all block change since that. As you can see in the image above, the full backups (blue in the image) are created merging the previous full with the ingested backup and are used as the base for the subsequent incremental.
ORDERING_WAIT
The ORDERING_WAIT occurs when the task INDEX_BACKUP that creates the index (and the virtual full backup) can’t finish because it doesn’t have all the required data. And this occurs because (by some reason) one backup is created and not stored at ZDLRA. And can be even a duplicate to create the standby (remember that basically the duplicate is rman backup copied from one side to another).
Look the image above, when after the SCN 4444 (that was the last backup stored at ZDLRA), another backup was taken and it is not inside of ZDLRA. So, when the new incremental backup is taken, it will copy all blocks changed from the last backup, but this last backup was the one that is not at ZDLRA (for rman side, it don’t care where it is the backup. By definition, incremental backup it is from the last backup, whatever or wherever it is).
And when this incremental backup it is ingested at ZDLRA, it will try to create the virtual full. But since the last stored backup have the SCN 4444, and the new incremental pickup blocks changed since SCN 5555 and go until the SCN 6666, ZDLRA knows that it is a gap when opens this ingested backup. ZDLRA doesn’t have the blocks that are between SCN 4444 and 5555 (look the yellow block, backup exists just outside of ZDLRA).
So, it is impossible to create the virtual full backup for SCN 6666, and the INDEX_BACKUP task will be at hold in state ORDERING_WAIT. To solve, there is two option, you can take a new level 0 backup or use BACKUP [CUMULATIVE] INCREMENTAL LEVEL 1 … FOR RECOVER OF TAG ‘<TAG>’ command. I will show you below how to do that.
How this occurs and how to solve
Bellow, I will show how you can identify and solve the problem. I will use the solution wrote in the last paragraph. But you can check the internal details of how occurs, to identify, and how to solve the issue.
In this scenario, I will use one database (number 12 – tablespace users for one PDB). So, first, check the backups for datafile 12:
RMAN> list backup of datafile 12 completed after "sysdate - 20/1440";
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24283 Incr 1 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24284 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282I Media:
List of Datafiles in backup set 24283
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24287 Incr 0 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24288 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282_12 Media:
List of Datafiles in backup set 24287
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
As you can see, we have Incr 1 and Incr 0 levels with the same scn. And the name of the handle starts with VB$. This means that virtual full backup it is OK and created by ZDLRA.
And if I do a new incremental backup, ZDLRA generates a new virtual full backup:
RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;
Starting backup at 20/04/2020 23:51:58
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 20/04/2020 23:51:59
channel ORA_SBT_TAPE_1: finished piece 1 at 20/04/2020 23:52:02
piece handle=ORCL18C_a1uu5dsv_1_1 tag=TAG20200420T235158 comment=API Version 2.0,MMS Version 12.2.0.2
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03
Finished backup at 20/04/2020 23:52:02
Starting Control File and SPFILE Autobackup at 20/04/2020 23:52:02
piece handle=c-558466555-20200420-0b comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 20/04/2020 23:52:10
RMAN> list backup of datafile 12 completed after "sysdate - 20/1440";
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24283 Incr 1 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24284 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282I Media:
List of Datafiles in backup set 24283
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24287 Incr 0 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24288 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282_12 Media:
List of Datafiles in backup set 24287
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24355 Incr 1 40.00K SBT_TAPE 00:00:03 20/04/2020 23:52:02
BP Key: 24356 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24354I Media:
List of Datafiles in backup set 24355
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013810 20/04/2020 23:51:59 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24359 Incr 0 40.00K SBT_TAPE 00:00:03 20/04/2020 23:52:02
BP Key: 24360 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24354_12 Media:
List of Datafiles in backup set 24359
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013810 20/04/2020 23:51:59 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
Simulating the error
Just to remember that this new incremental have all scn from 2013573 until 2013810.
But by some reason one full backup it takes to another place (like disk, or duplicate for standby). Look that the channel type it is not ZDLRA:
RMAN> BACKUP INCREMENTAL LEVEL 0 DEVICE TYPE disk format '/tmp/%U' DATAFILE 12 TAG 'BKP-DBF-TO-DISK';
Starting backup at 20/04/2020 23:54:01
released channel: ORA_SBT_TAPE_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=65 device type=DISK
channel ORA_DISK_1: starting incremental level 0 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_DISK_1: starting piece 1 at 20/04/2020 23:54:01
channel ORA_DISK_1: finished piece 1 at 20/04/2020 23:54:02
piece handle=/tmp/a3uu5e0p_1_1 tag=BKP-DBF-TO-DISK comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 20/04/2020 23:54:02
Starting Control File and SPFILE Autobackup at 20/04/2020 23:54:02
piece handle=/u01/app/oracle/oradata/ORCL18C/autobackup/2020_04_20/o1_mf_s_1038268443_h9w6hwht_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 20/04/2020 23:54:10
RMAN> list backup tag = 'BKP-DBF-TO-DISK';
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24391 Incr 0 1.18M DISK 00:00:00 20/04/2020 23:54:01
BP Key: 24394 Status: AVAILABLE Compressed: NO Tag: BKP-DBF-TO-DISK
Piece Name: /tmp/a3uu5e0p_1_1
List of Datafiles in backup set 24391
Container ID: 3, PDB Name: ORCL18P
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013972 20/04/2020 23:54:01 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
Now, if I try to execute the incremental level 1, the ingested backup does not generate a new virtual backup:
RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;
Starting backup at 20/04/2020 23:55:23
released channel: ORA_DISK_1
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=65 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: RA Library (ZDLRAS1) SID=A3C0F4C16DAA11FAE053010310ACC1C4
channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 20/04/2020 23:55:24
channel ORA_SBT_TAPE_1: finished piece 1 at 20/04/2020 23:55:27
piece handle=ORCL18C_a5uu5e3c_1_1 tag=TAG20200420T235524 comment=API Version 2.0,MMS Version 12.2.0.2
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03
Finished backup at 20/04/2020 23:55:27
Starting Control File and SPFILE Autobackup at 20/04/2020 23:55:27
piece handle=c-558466555-20200420-0d comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 20/04/2020 23:55:35
RMAN> list backup of datafile 12 completed after "sysdate - 20/1440";
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24283 Incr 1 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24284 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282I Media:
List of Datafiles in backup set 24283
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24287 Incr 0 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24288 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282_12 Media:
List of Datafiles in backup set 24287
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24355 Incr 1 40.00K SBT_TAPE 00:00:03 20/04/2020 23:52:02
BP Key: 24356 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24354I Media:
List of Datafiles in backup set 24355
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013810 20/04/2020 23:51:59 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24359 Incr 0 40.00K SBT_TAPE 00:00:03 20/04/2020 23:52:02
BP Key: 24360 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24354_12 Media:
List of Datafiles in backup set 24359
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013810 20/04/2020 23:51:59 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24391 Incr 0 1.18M DISK 00:00:00 20/04/2020 23:54:01
BP Key: 24394 Status: AVAILABLE Compressed: NO Tag: BKP-DBF-TO-DISK
Piece Name: /tmp/a3uu5e0p_1_1
List of Datafiles in backup set 24391
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013972 20/04/2020 23:54:01 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24430 Incr 1 256.00K SBT_TAPE 00:00:00 20/04/2020 23:55:24
BP Key: 24431 Status: AVAILABLE Compressed: NO Tag: TAG20200420T235524
Handle: ORCL18C_a5uu5e3c_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24430
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2014089 20/04/2020 23:55:24 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
Look above that the last incremental backup not generated a new level 0, it is the backupset 24430 (and backuppiece 24431).
Task inside ZDLRA
If we enter inside of ZDLRA, we can see that the task responsible to index the backupiece 23286 is in state ORDEING_WAIT. You can use query over ra_task and check using the state column..
SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where db_unique_name = 'ORCL18C' and state = 'ORDERING_WAIT' order by 5,2,7,10,11,12,13;
TASK_ID TASK_TYPE STATE WAITING_ON DB_KEY DB_UNIQUE_NAME CREATION_TIME ERROR_COUNT INTERRUPT_COUNT BP_KEY BS_KEY DF_KEY VB_KEY
---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------
40203 INDEX_BACKUP ORDERING_WAIT 15993 ORCL18C 20-APR-20 11.55.26.779191 PM +02:00 0 1 24431
SQL>
And there is no incident for this error or state:
SQL> select incident_id, error_text last_seen from ra_incident_log where db_unique_name = 'ORCL18C' and status not in ('FIXED', 'RESET') order by last_seen desc;
no rows selected
SQL>
Unfortunately, this occurs for ZDLRA, even if there is a task in the ORDEING_WAIT state, it is not reported as an error. And if you think about, the virtual full backup it is not generated and the feature that it is the virtual full backup is not in place for these datafiles.
But even with tasks in this state, you will not be unprotected, the backup is there and can be restored.
ZDLRA Internals
If we check inside of ZDLRA we can check more details (this is one example how to navigate through internal tables of ZDLRA rman catalog and find the information – same that you find with list backupset inside rman):
SQL> select bs_key, db_key, pdb_key from bp where bp_key = 24431;
BS_KEY DB_KEY PDB_KEY
---------- ---------- ----------
24430 15993 16000
SQL> select * from rc_database where db_key = 15993;
DB_KEY DBINC_KEY DBID NAME RESETLOGS_CHANGE# RESETLOGS FINAL_CHANGE#
---------- ---------- ---------- -------- ----------------- --------- -------------
15993 15994 558466555 ORCL18C 1477662 11-AUG-19
SQL> select df_key from df where dbinc_key = 15994 and file# = 12;
DF_KEY
----------
16026
SQL> select bdf_key, ckp_scn from bdf where bs_key = 24430;
BDF_KEY CKP_SCN
---------- ----------
24432 2014089
SQL>
###################################################################
RMAN> list backupset 24430;
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24430 Incr 1 256.00K SBT_TAPE 00:00:00 20/04/2020 23:55:24
BP Key: 24431 Status: AVAILABLE Compressed: NO Tag: TAG20200420T235524
Handle: ORCL18C_a5uu5e3c_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24430
Container ID: 3, PDB Name: ORCL18P
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2014089 20/04/2020 23:55:24 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
Using the BP table we can find the DB_KEY, and with that, we can go to RC_DATABASE to get the incarnation key (DBINC_KEY). With that, at DF table we can discover the DF_KEY for the datafile 12 and for this database incarnation. And with the DBF table, we can find the CKP_SCN for this datafile.
But if we go to the BLOCKS tables (ZDLRA table that stores the index – and “more or less” the virtual full), there are no database blocks with the SCN 2014089 for the backupset 24430. If you want to understand who it works, you can read my post about this.
SQL> select * from blocks where df_key = 16026 and scn >= 2014089;
no rows selected
SQL>
And if we check for SCN 2013972 (there came from backup was put in disk), nothing too (as expected):
SQL> select * from blocks where df_key = 16026 and scn >= 2013972;
no rows selected
SQL>
As if we check with the last SCN 2013810 that are know by ZDLRA (last virtual full backup, backupset 24359), we can see which blocks are there:
SQL> select * from blocks where df_key = 16026 and scn >= 2013810 order by scn, chunkno ;
DF_KEY BLOCKNO SCN CKP_ID CHUNKNO COFFSET USED DBINC_KEY ENDBLK
---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
16026 1 2013810 2013810 25601 33121 338 15994
16026 0 2013810 2013810 25601 8192 24576 15994
SQL>
And there is just PLANS for this virtual backup (nothing generated for the next existing backups):
SQL> select VB_KEY, DF_KEY, CKP_SCN, SRCBP_KEY, VCBP_KEY from vbdf where CKP_SCN >= 2013810 and df_key = 16026;
VB_KEY DF_KEY CKP_SCN SRCBP_KEY VCBP_KEY
---------- ---------- ---------- ---------- ----------
24354 16026 2013810 24293 24356
SQL> select * from plans_details where VB_KEY = 24354;
DF_KEY TYPE VB_KEY BLKRANK BLOCKNO CHUNKNO NUMBLKS COFFSET NUMBYTES
---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
16026 1 24354 1 0 25601 1 8192 24576
16026 1 24354 1 2 16385 2 32788 256
16026 1 24354 1 4 14337 132 33045 14000
16026 1 24354 1 136 16385 3 33044 553
16026 1 24354 1 139 23553 1 32788 327
16026 1 24354 1 140 16385 2 33786 264
16026 1 24354 1 142 25601 1 32788 333
16026 1 24354 1 143 16385 1 34182 132
16026 1 24354 1 191 14337 3 47045 252
16026 1 24354 1 4294967295 25601 1 33121 338
10 rows selected.
SQL>
As you can figure out, ZDLRA can’t fill the gap to create the virtual full backup.
Recurring error
If you not solve the problem, and continue to ingest backups, the task will remain in ORDERING_WAIT:
RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;
Starting backup at 21/04/2020 00:04:00
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 21/04/2020 00:04:01
channel ORA_SBT_TAPE_1: finished piece 1 at 21/04/2020 00:04:04
piece handle=ORCL18C_a7uu5ejh_1_1 tag=TAG20200421T000400 comment=API Version 2.0,MMS Version 12.2.0.2
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03
Finished backup at 21/04/2020 00:04:04
Starting Control File and SPFILE Autobackup at 21/04/2020 00:04:04
piece handle=c-558466555-20200421-00 comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 21/04/2020 00:04:13
RMAN> list backup of datafile 12 completed after "sysdate - 5/1440";
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24511 Incr 1 256.00K SBT_TAPE 00:00:01 21/04/2020 00:04:02
BP Key: 24512 Status: AVAILABLE Compressed: NO Tag: TAG20200421T000400
Handle: ORCL18C_a7uu5ejh_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24511
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2021757 21/04/2020 00:04:01 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
####################################
SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where db_unique_name = 'ORCL18C' and state = 'ORDERING_WAIT' order by 5,2,7,10,11,12,13;
TASK_ID TASK_TYPE STATE WAITING_ON DB_KEY DB_UNIQUE_NAME CREATION_TIME ERROR_COUNT INTERRUPT_COUNT BP_KEY BS_KEY DF_KEY VB_KEY
---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------
40203 INDEX_BACKUP ORDERING_WAIT 15993 ORCL18C 20-APR-20 11.55.26.779191 PM +02:00 0 1 24431
40210 INDEX_BACKUP ORDERING_WAIT 15993 ORCL18C 21-APR-20 12.04.05.207342 AM +02:00 0 1 24512
SQL>
Even if you try to do a cumulative incremental backup, the problem will be the same:
RMAN> BACKUP CUMULATIVE INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 12;
Starting backup at 21/04/2020 00:07:13
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 21/04/2020 00:07:13
channel ORA_SBT_TAPE_1: finished piece 1 at 21/04/2020 00:07:20
piece handle=ORCL18C_a9uu5eph_1_1 tag=TAG20200421T000713 comment=API Version 2.0,MMS Version 12.2.0.2
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:07
Finished backup at 21/04/2020 00:07:20
Starting Control File and SPFILE Autobackup at 21/04/2020 00:07:20
piece handle=c-558466555-20200421-01 comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 21/04/2020 00:07:28
RMAN> list backup of datafile 12 completed after "sysdate - 2/1440";
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24604 Incr 1 256.00K SBT_TAPE 00:00:02 21/04/2020 00:07:15
BP Key: 24605 Status: AVAILABLE Compressed: NO Tag: TAG20200421T000713
Handle: ORCL18C_a9uu5eph_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24604
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2021959 21/04/2020 00:07:13 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
#############################################
SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where db_unique_name = 'ORCL18C' and state = 'ORDERING_WAIT' order by 5,2,7,10,11,12,13;
TASK_ID TASK_TYPE STATE WAITING_ON DB_KEY DB_UNIQUE_NAME CREATION_TIME ERROR_COUNT INTERRUPT_COUNT BP_KEY BS_KEY DF_KEY VB_KEY
---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------
40203 INDEX_BACKUP ORDERING_WAIT 15993 ORCL18C 20-APR-20 11.55.26.779191 PM +02:00 0 1 24431
40210 INDEX_BACKUP ORDERING_WAIT 15993 ORCL18C 21-APR-20 12.04.05.207342 AM +02:00 0 1 24512
40215 INDEX_BACKUP ORDERING_WAIT 15993 ORCL18C 21-APR-20 12.07.18.140618 AM +02:00 0 1 24605
SQL>
Solving ORDERING_WAIT
There are two ways to solve the issue, for both the idea is the same: ingest the database blocks inside of ZDLRA. And we do this performing backup.
FOR RECOVER OF TAG
The first is to use the command “BACKUP [CUMULATIVE] INCREMENTAL LEVEL 1 … FOR RECOVER OF TAG ‘<TAG>’”. The idea here is to create one incremental backup that recovers since one specific tag. You can check the documentation for more details if you want, it exists since Oracle 10g.
As you can imagine, the critical point here is to define the correct TAG do be used as a reference. And in this case, the tag is the last full backup (virtual or no) that it is inside of ZDLRA. Doing this, we ingest all changed blocks and fill the gap that is holding the task.
In this case, I used normal incremental. Look that the tag is from the last virtual full backup that is inside of ZDLRA for this datafile:
RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FOR RECOVER OF TAG 'TAG20200420T235158' DATAFILE 12;
Starting backup at 21/04/2020 00:12:17
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 21/04/2020 00:12:17
channel ORA_SBT_TAPE_1: finished piece 1 at 21/04/2020 00:12:20
piece handle=ORCL18C_abuu5f31_1_1 tag=TAG20200420T235158 comment=API Version 2.0,MMS Version 12.2.0.2
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03
Finished backup at 21/04/2020 00:12:20
Starting Control File and SPFILE Autobackup at 21/04/2020 00:12:20
piece handle=c-558466555-20200421-02 comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 21/04/2020 00:12:28
RMAN>
And we can see that a new virtual full backup was created with this incremental backup (look the last 5 backupsets):
RMAN> list backup of datafile 12 completed after "sysdate - 40/1440";
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24283 Incr 1 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24284 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282I Media:
List of Datafiles in backup set 24283
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24287 Incr 0 40.00K SBT_TAPE 00:00:04 20/04/2020 23:49:26
BP Key: 24288 Status: AVAILABLE Compressed: YES Tag: TAG20200420T234921
Handle: VB$_1891149551_24282_12 Media:
List of Datafiles in backup set 24287
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013573 20/04/2020 23:49:22 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24355 Incr 1 40.00K SBT_TAPE 00:00:03 20/04/2020 23:52:02
BP Key: 24356 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24354I Media:
List of Datafiles in backup set 24355
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2013810 20/04/2020 23:51:59 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24359 Incr 0 40.00K SBT_TAPE 00:00:03 20/04/2020 23:52:02
BP Key: 24360 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24354_12 Media:
List of Datafiles in backup set 24359
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013810 20/04/2020 23:51:59 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24391 Incr 0 1.18M DISK 00:00:00 20/04/2020 23:54:01
BP Key: 24394 Status: AVAILABLE Compressed: NO Tag: BKP-DBF-TO-DISK
Piece Name: /tmp/a3uu5e0p_1_1
List of Datafiles in backup set 24391
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2013972 20/04/2020 23:54:01 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24430 Incr 1 256.00K SBT_TAPE 00:00:00 20/04/2020 23:55:24
BP Key: 24431 Status: AVAILABLE Compressed: NO Tag: TAG20200420T235524
Handle: ORCL18C_a5uu5e3c_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24430
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2014089 20/04/2020 23:55:24 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24511 Incr 1 256.00K SBT_TAPE 00:00:01 21/04/2020 00:04:02
BP Key: 24512 Status: AVAILABLE Compressed: NO Tag: TAG20200421T000400
Handle: ORCL18C_a7uu5ejh_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24511
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2021757 21/04/2020 00:04:01 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24604 Incr 1 256.00K SBT_TAPE 00:00:02 21/04/2020 00:07:15
BP Key: 24605 Status: AVAILABLE Compressed: NO Tag: TAG20200421T000713
Handle: ORCL18C_a9uu5eph_1_1 Media: Recovery Appliance (ZDLRAS1)
List of Datafiles in backup set 24604
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2021959 21/04/2020 00:07:13 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24814 Incr 1 40.00K SBT_TAPE 00:00:03 21/04/2020 00:12:20
BP Key: 24815 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24813I Media:
List of Datafiles in backup set 24814
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 1 Incr 2025613 21/04/2020 00:12:17 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
24818 Incr 0 40.00K SBT_TAPE 00:00:03 21/04/2020 00:12:20
BP Key: 24819 Status: AVAILABLE Compressed: YES Tag: TAG20200420T235158
Handle: VB$_1891149551_24813_12 Media:
List of Datafiles in backup set 24818
File LV Type Ckp SCN Ckp Time Abs Fuz SCN Sparse Name
---- -- ---- ---------- ------------------- ----------- ------ ----
12 0 Incr 2025613 21/04/2020 00:12:17 NO /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
RMAN>
And inside of ZDLRA, we can see that task previously in ORDERING_WAIT finished (check the BP_KEY column):
SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, COMPLETION_TIME, ERROR_COUNT, INTERRUPT_COUNT, BP_KEY,BS_KEY,DF_KEY,VB_KEY from rasys.ra_task where task_id IN (40203,40210,40215);
TASK_ID TASK_TYPE STATE WAITING_ON DB_KEY DB_UNIQUE_NAME CREATION_TIME COMPLETION_TIME ERROR_COUNT INTERRUPT_COUNT BP_KEY BS_KEY DF_KEY VB_KEY
---------- --------------- ------------------------- ---------- ---------- ------------------------------ ----------------------------------- ----------------------------------- ----------- --------------- ---------- ---------- ---------- ----------
40203 INDEX_BACKUP COMPLETED 15993 ORCL18C 20-APR-20 11.55.26.779191 PM +02:00 21-APR-20 12.13.09.253916 AM +02:00 0 1 24431
40210 INDEX_BACKUP COMPLETED 15993 ORCL18C 21-APR-20 12.04.05.207342 AM +02:00 21-APR-20 12.13.21.663869 AM +02:00 0 1 24512
40215 INDEX_BACKUP COMPLETED 15993 ORCL18C 21-APR-20 12.07.18.140618 AM +02:00 21-APR-20 12.13.42.179087 AM +02:00 0 1 24605
SQL>
And if we check for the BLOCKS for this datafile, we can see that was registered new that are higher with the last full backup before the error, and they go until the last backup made
SQL> select * from blocks where df_key = 16026 and scn >= 2013810 order by scn, chunkno ;
DF_KEY BLOCKNO SCN CKP_ID CHUNKNO COFFSET USED DBINC_KEY ENDBLK
---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
16026 0 2013810 2013810 25601 8192 24576 15994
16026 1 2013810 2013810 25601 33121 338 15994
16026 142 2021942 2025613 26625 32788 414 15994
16026 1 2025613 2025613 26625 33202 336 15994
16026 0 2025613 2025613 26625 8192 24576 15994
SQL>
And we can see that now exists PLANS for the backupset of virtual full backup that exists before the error (SCN 2013810) and after we fix (SCN 2025613):
SQL> select VB_KEY, DF_KEY, CKP_SCN, SRCBP_KEY, VCBP_KEY from vbdf where CKP_SCN >= 2013810 and DF_KEY = 16026;
VB_KEY DF_KEY CKP_SCN SRCBP_KEY VCBP_KEY
---------- ---------- ---------- ---------- ----------
24354 16026 2013810 24293 24356
24813 16026 2025613 24706 24815
SQL>
SQL> select * from plans_details where VB_KEY IN (24354,24813) order by VB_KEY,BLOCKNO;
DF_KEY TYPE VB_KEY BLKRANK BLOCKNO CHUNKNO NUMBLKS COFFSET NUMBYTES
---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
16026 1 24354 1 0 25601 1 8192 24576
16026 1 24354 1 2 16385 2 32788 256
16026 1 24354 1 4 14337 132 33045 14000
16026 1 24354 1 136 16385 3 33044 553
16026 1 24354 1 139 23553 1 32788 327
16026 1 24354 1 140 16385 2 33786 264
16026 1 24354 1 142 25601 1 32788 333
16026 1 24354 1 143 16385 1 34182 132
16026 1 24354 1 191 14337 3 47045 252
16026 1 24354 1 4294967295 25601 1 33121 338
16026 1 24813 1 0 26625 1 8192 24576
16026 1 24813 1 2 16385 2 32788 256
16026 1 24813 1 4 14337 132 33045 14000
16026 1 24813 1 136 16385 3 33044 553
16026 1 24813 1 139 23553 1 32788 327
16026 1 24813 1 140 16385 2 33786 264
16026 1 24813 1 142 26625 1 32788 414
16026 1 24813 1 143 16385 1 34182 132
16026 1 24813 1 191 14337 3 47045 252
16026 1 24813 1 4294967295 26625 1 33202 336
20 rows selected.
SQL>
So, this means that the incremental backup that we made with FOR RECOVERY OF TAG was ingested and used to fix the needed gap.
And if we try to recover the datafile 12, we can do without a problem. Check that the used backup to recover was the last virtual full backup generated from the “RECOVERY OF TAG” command:
RMAN> run{
2> ALTER PLUGGABLE DATABASE ORCL18P CLOSE IMMEDIATE INSTANCES=ALL;
3> RESTORE DATAFILE 12;
4> RECOVER DATAFILE 12;
5> ALTER PLUGGABLE DATABASE ORCL18P OPEN INSTANCES=ALL;
6> }
Statement processed
starting full resync of recovery catalog
full resync complete
Starting restore at 21/04/2020 00:57:53
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=88 device type=DISK
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=70 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: RA Library (ZDLRAS1) SID=A3C1D44670B428B0E053010310AC5DA9
channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00012 to /u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: reading from backup piece VB$_1891149551_24813_12
channel ORA_SBT_TAPE_1: piece handle=VB$_1891149551_24813_12 tag=TAG20200420T235158
channel ORA_SBT_TAPE_1: restored backup piece 1
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:25
Finished restore at 21/04/2020 00:58:21
Starting recover at 21/04/2020 00:58:21
using channel ORA_DISK_1
using channel ORA_SBT_TAPE_1
starting media recovery
media recovery complete, elapsed time: 00:00:00
Finished recover at 21/04/2020 00:58:22
Statement processed
starting full resync of recovery catalog
full resync complete
RMAN>
Unfortunately, backup tags for ZDLRA can be tricky when you directly specify it during the backup phase. They can be the same and the usage “FOR TAG” can be more difficult to define. One option is to merge and execute the command BACKUP CUMULATIVE INCREMENTAL LEVEL 1 DEVICE TYPE SBT FOR RECOVER OF TAG ‘<TAG>’ DATAFILE XX.
Doing this, the command will pick up all the blocks from the last full backup that have the tag that you defined. The result is the same:
RMAN> BACKUP CUMULATIVE INCREMENTAL LEVEL 1 DEVICE TYPE SBT FOR RECOVER OF TAG 'TAG20200419T232006' DATAFILE 12;
Starting backup at 19/04/2020 23:38:44
using channel ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set
channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
input datafile file number=00012 name=/u01/app/oracle/oradata/ORCL18C/ORCL18P/users01.dbf
channel ORA_SBT_TAPE_1: starting piece 1 at 19/04/2020 23:38:44
channel ORA_SBT_TAPE_1: finished piece 1 at 19/04/2020 23:38:59
piece handle=ORCL18C_97uu2oo4_1_1 tag=TAG20200419T232006 comment=API Version 2.0,MMS Version 12.2.0.2
channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:15
Finished backup at 19/04/2020 23:38:59
Starting Control File and SPFILE Autobackup at 19/04/2020 23:39:00
piece handle=c-558466555-20200419-04 comment=API Version 2.0,MMS Version 12.2.0.2
Finished Control File and SPFILE Autobackup at 19/04/2020 23:39:16
RMAN>
BACKUP FULL
The other option to solve the ORDERING_WAIT is to do a full backup of the datafile. With this, all the blocks are read and ingested at ZDLRA.
The procedure is the same as above and the result the same. The only point is that for huge files this can take a long time (since it is full) and the above incremental approach can be more suitable.
Monitoring the Tasks
So, as you can see above, the ORDERING_WAIT state can have a lot of collateral effects for you ZDLRA. Unfortunately, this not generates incidents that are reported, you need to write a query to check this directly at ra_task table.
Whatever the method that you choose to solve the problem (RECOVERY OF TAG, or FULL BACKUP) always verify if the new virtual full backup is generated. It is a good practice to do this to avoid errors and do a double cross-check over the tasks
It is a simple query, and a simple monitoring thing to do. But this will avoid a huge problem.
Reference:
Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”
