ZDLRA

ZDLRA

ZDLRA Internals, INDEX_BACKUP task in details

Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA Internals, INDEX_BACKUP task in details

For ZDLRA, the task type INDEX_BACKUP it is important (if it is not the most) because it is responsible to create the virtual full backup. This task runs for every backup that you ingest at ZDLRA and here, I will show with more details what occurs at ZDLRA: internals steps, phases, and tables involved.

I recommend that you check my previous post about ZDLRA: ZDLRA Internals, Tables and Storage, ZDLRA, Virtual Full Backup and Incremental Forever, and Understanding ZDLRA. They provide a good base to understand some aspects of ZDLRA architecture and features.

As you saw in my previous post, ZDLRA opens every backup that you sent and read every block of it to generate one new virtual full backup. And this backup is validated block a block (physically and logically) against corruption. It differs from a snapshot because it is content-aware (in this case it is proprietary Oracle datafile blocks inside another proprietary Oracle rman block) and Oracle it is the only that can do this guaranteeing that result is valid.

Backup

This generation is done thought INDEX_BACKUP task and occurs for every backup (level 0 or 1) that enter in the system. For this post, I already have a full backup level 0 inside ZDLRA and I made one new incremental level 1:

RMAN> run{

2> backup device type sbt cumulative incremental level 1 filesperset 1 datafile 29;

3> }

Starting backup at 2019-08-16_15-01-11

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=406 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRA) SID=903C95C93085DC14E0531E43B20AB30F

allocated channel: ORA_SBT_TAPE_2

channel ORA_SBT_TAPE_2: SID=451 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_2: RA Library (ZDLRA) SID=903C95D1C201DC19E0531E43B20A4C44

allocated channel: ORA_SBT_TAPE_3

channel ORA_SBT_TAPE_3: SID=502 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_3: RA Library (ZDLRA) SID=903C95E1BFD9DC38E0531E43B20A0038

allocated channel: ORA_SBT_TAPE_4

channel ORA_SBT_TAPE_4: SID=651 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_4: RA Library (ZDLRA) SID=903C95EAC1ACDC48E0531E43B20AC79D

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00029 name=+DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

channel ORA_SBT_TAPE_1: starting piece 1 at 2019-08-16_15-01-25

channel ORA_SBT_TAPE_1: finished piece 1 at 2019-08-16_15-03-10

piece handle=SIMON_s0u9c0a5_1_1 tag=TAG20190816T150116 comment=API Version 2.0,MMS Version 3.17.1.26

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:01:45

Finished backup at 2019-08-16_15-03-10

Starting Control File and SPFILE Autobackup at 2019-08-16_15-03-11

piece handle=c-4165931009-20190816-06 comment=API Version 2.0,MMS Version 3.17.1.26

Finished Control File and SPFILE Autobackup at 2019-08-16_15-03-18

RMAN>

As you can see, it is a normal backup. And the details for this backup:

RMAN> list backupset 52141830;

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52141830 Incr 1  30.75M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52141831   Status: AVAILABLE  Compressed: NO  Tag: TAG20190816T150116

        Handle: SIMON_s0u9c0a5_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 52141830

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

RMAN> list backup tag = 'TAG20190816T150116';

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52141830 Incr 1  30.75M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52141831   Status: AVAILABLE  Compressed: NO  Tag: TAG20190816T150116

        Handle: SIMON_s0u9c0a5_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 52141830

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

Here it is important to remember the TAG, backupset(BS)/backup piece (BP) numbers, and the SCN for the backup made.

INDEX_BACKUP

The INDEX_BACKUP appears as a single task inside ZDLRA, but have two major phases: fixup_unordered and q_restore_fast. And each one impacts differently inside the rman catalog and in the internal tables.

So, after the backup finish, inside of ZDLRA we can see the index task running:

SQL> l

  1  SELECT s.ba_session_id, s.instance_id, s.sid, s.serial#, s.job_name

  2         , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state, rt.waiting_on

  3         , rt.elapsed_seconds

  4         , gs.module, gs.sql_id, gs.action

  5         , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

  6  FROM sessions s

  7  JOIN ra_task rt

  8  ON rt.task_id = s.current_task

  9  AND rt.task_type = 'INDEX_BACKUP'

 10  JOIN gv$session gs

 11  ON gs.inst_id = s.instance_id

 12  AND gs.sid = s.sid

 13  AND gs.serial# = s.serial#

 14* ORDER BY rt.LAST_EXECUTE_TIME DESC

SQL> /

BA_SESSION_ID INSTANCE_ID    SID    SERIAL# JOB_NAME              TASK_ID     DB_KEY DB_UNIQUE_NAME  TASK_TYPE       STATE    WAITING_ON ELAPSED_SECONDS MODULE           SQL_ID        ACTION            BP_KEY     BS_KEY     DF_KEY     VB_KEY

------------- ----------- ------ ---------- ------------------ ---------- ---------- --------------- --------------- -------- ---------- --------------- ---------------- ------------- ------------- ---------- ---------- ---------- ----------

     56215535           2   4228      30075 RA$_EXEC_56216288    56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING                  345.305895 fixup_unordered  1q0zr86n25pvu scan 6% done    52141831

SQL>

With this SQL you can check all the INDEX_BACKUP tasks running at ZDLRA. And the SQL use the tables and report:

RA_TASK: All tasks running inside ZDLRA.
SESSIONS: Sessions running inside ZDLRA, used to get the database session running the task
GV$SESSION: Databases sessions, used to get the sql_id and others info
BP_KEY: Identify the backup piece key that it is indexing. It is the start point to identify everything here.

Above you can see that task 56242962 is running, processing the BP_KEY (backup piece key) 52141831, running the module fixup_unordered, processed 6% of this step, and running SQL_ID 1q0zr86n25pvu.

INDEX_BACKUP, fixup_unordered

As showed, the index task is running the fixup_unordered module, and thus represents the first phase where ZDLRA it is checking all the blocks that are necessary to build the virtual full backup. Based on the report from the query above we can see that BP_KEY processed it is 52141831 and we can get more info from backup piece rman table:

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024 from bp where bp_key = 52141831;

    BP_KEY     BS_KEY TAG                              HANDLE               BYTES/1024/1024/1024

---------- ---------- -------------------------------- -------------------- --------------------

  52141831   52141830 TAG20190816T150116               SIMON_s0u9c0a5_1_1             .030029297

SQL>

SQL> select bs_key, INCREMENTAL_LEVEL, HANDLE from rc_backup_piece where bp_key = 52141831;

    BS_KEY INCREMENTAL_LEVEL HANDLE

---------- ----------------- --------------------

  52141830                 1 SIMON_s0u9c0a5_1_1

SQL>

This tells us that the task it is processing the backup that we made before (look the result of column BP_KEY from SQL in the first topic). The BP, and rc_backup_piece are the tables from rman catalog that store info about the backup pieces, check the handle and tag to compare from rman list output.

Going further we can check the VBDF (virtual backup data file table – store the virtual full backups information) table and verify more details:

SQL> SELECT vb_key, ckp_id, df_key, db_key, blocks, vcbp_key, srcbp_key, file#, ckp_scn FROM VBDF WHERE srcbp_key = 52141831;

    VB_KEY         CKP_ID     DF_KEY     DB_KEY     BLOCKS   VCBP_KEY  SRCBP_KEY      FILE#        CKP_SCN

---------- -------------- ---------- ---------- ---------- ---------- ---------- ---------- --------------

  52141864  4932560891039     752220     752186 1655177216              52141831         29  4932560891039

SQL>

At VBDF table we can discover:

VB_KEY: Virtual backup key.
CKP_ID: Checkpoint and SCN for this virtual backup.
BLOCKS: Number of the blocks for this virtual backup.
VCBP_KEY: Is null at fixup_unordered because the virtual backup piece (VCBP) was not yet generated (it is doing by fixup_unordered). This will change in the future and I will show more details.
SRCBP_KEY: The source/original backup piece used to create this virtual full backup.

So, we can check that the backu is in the middle of the index the process since the VCBP_KEY is not yet available. But besides that, we can validate/check more details about the backups.

Check that does not exist a backup piece associated with this virtual backup key:

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where vb_key = 52141864;

no rows selected

SQL>

And that exists just one backup of datafile for this SCN and FILE#:

SQL> SELECT bdf_key, bs_key, file#, ckp_scn, incr_level from bdf where CKP_SCN = 4932560891039 and file# = 29;

   BDF_KEY     BS_KEY      FILE#        CKP_SCN INCR_LEVEL

---------- ---------- ---------- -------------- ----------

  52141832   52141830         29  4932560891039          1

SQL>

SQL> SELECT db_key, bdf_key, file#, CHECKPOINT_CHANGE#, incremental_level from rc_backup_datafile where bs_key IN(52141830);

    DB_KEY    BDF_KEY      FILE# CHECKPOINT_CHANGE# INCREMENTAL_LEVEL

---------- ---------- ---------- ------------------ -----------------

    752186   52141832         29      4932560891039                 1

SQL> SELECT db_key, bdf_key, file#, CHECKPOINT_CHANGE#, incremental_level from rc_backup_datafile where CHECKPOINT_CHANGE# = 4932560891039;

    DB_KEY    BDF_KEY      FILE# CHECKPOINT_CHANGE# INCREMENTAL_LEVEL

---------- ---------- ---------- ------------------ -----------------

    752186   52141832         29      4932560891039                 1

SQL>

Digging more, we can see that are zero plans for this virtual backup (will be zero until fixup_unordered finish – more detail further):

SQL> select count(*) from plans where VB_KEY = 52141864;

  COUNT(*)

----------

         0

SQL>

During this phase of INDEX_BACKUP, the major SQL_ID is 1q0zr86n25pvu:

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = '1q0zr86n25pvu';

SQL_FULLTEXT

----------------------------------------------------------------------------------------------------

SELECT * FROM (SELECT /*+

                     QB_NAME(c)

                     INDEX_RS_ASC(@c b@c)

                     NO_INDEX_FFS(@c b@c)

                     NO_INDEX_SS(@c b@c)

                     OPT_PARAM('optimizer_dynamic_sampling' 0)

                     OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

                     OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

                     OPT_PARAM('_optimizer_use_feedback' 'false')

                    */ BLOCKNO, SCN, CKP_ID, ENDBLK, CHUNKNO, COFFSET FROM BLOCKS B WHERE DF_KEY = :

B4 AND BLOCKNO BETWEEN :B3 AND :B2 AND SCN < :B1 ORDER BY BLOCKNO, SCN, CKP_ID ) WHERE ROWNUM < :B5

SQL>

And the bind variables in this case are:

SQL> l

  1  SELECT s.ba_session_id, s.instance_id

  2  , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state

  3  , gs.module, gs.sql_id, gs.action

  4  , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

  5  , sbc.name, sbc.value_string

  6  FROM sessions s

  7  JOIN ra_task rt

  8  ON rt.task_id = s.current_task

  9  AND RT.TASK_ID = 56242962

 10  JOIN gv$session gs

 11  ON gs.inst_id = s.instance_id

 12  AND gs.sid = s.sid

 13  AND gs.serial# = s.serial#

 14  join gv$sql_bind_capture sbc

 15  on sbc.inst_id = gs.inst_id

 16  and sbc.sql_id = gs.sql_id

 17* ORDER BY rt.LAST_EXECUTE_TIME DESC

SQL>

SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE           MODULE           SQL_ID        ACTION              BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- -------------------- --------------- --------------- ---------------- ------------- --------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B4   752220

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B3   302875642

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B2   1655177216

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B1   4932554641877

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B5   1048576

SQL>

Explaining a little. So, based on the SQL text and variables we can see that is retrieving from BLOCKS table all the blocks for the DF_KEY 752220 (you can get it in VBDF), that are between 302875642 (actual block of processing) and 1655177216 (max number of blocks for the ingested backup), and have SCN bellow 4932554641877 (from the ingested backup) and process this in package of 1048576 blocks. One information here it is that the number 302875642 will vary while it is running this phase.

So, the idea here for ZDLRA index task is processing get/check all the blocks that have SCN below of the SCN from ingested backup. And since the INDEX_BACKUP result creates one virtual full backup, all the blocks for this datafile need to be processed.

If you check this query time to time, you will see the increase of scan % increasing until the :B3 and :B4 are equal (compare with the previous execution):

SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE           MODULE           SQL_ID        ACTION             BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- -------------------- --------------- --------------- ---------------- ------------- -------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B4   752220

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B3   1641637658

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B2   1655177216

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B1   4932554641877

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B5   1048576

SQL> select rt.elapsed_seconds from ra_task rt where task_id = 56242962;

ELAPSED_SECONDS

---------------

     6618.53526

SQL>

After this hit 100% the next phase of the INDEX_BAKUP (q_restore_fast) will start. The phase fixup_unordered depends on the size of your datafile and not from the ingested backup. In this example, the backup has just 30MB, but the datafile has more than 16TB, so, the virtual full will have the same size and this takes a lot of time reading all the blocks. For you ZDLRA the DF_KEY, BS_KEY, and others will be different. Just remember that BP_KEY for the index task it is the start point for the analyses.

INDEX_BACKUP, q_restore_fast

After the fixup_unordered reach 100% of the scan, the phase q_restore_fast starts and this will create the virtual backup itself. But before let’s see what’s happened at rman catalog after the previous phase finished:

##########################################################

This is list took right after the backup finished:

RMAN> list backup tag = 'TAG20190816T150116';

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52141830 Incr 1  30.75M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52141831   Status: AVAILABLE  Compressed: NO  Tag: TAG20190816T150116

        Handle: SIMON_s0u9c0a5_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 52141830

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

##########################################################

And now after the fixup_unordered finish:

RMAN> list backup tag = 'TAG20190816T150116';

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52146235 Incr 1  28.00M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52146236   Status: AVAILABLE  Compressed: YES  Tag: TAG20190816T150116

        Handle: VB$_90959062_52141864I   Media:

  List of Datafiles in backup set 52146235

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

As you can see, the backupset (52141830 before, and 52146235 now) and backup piece changed (the handle changed too). But look that TAG and SCN remains the same. This means that a new backup was created because ZDLRA scanned the original backup. But as you can see, the virtual full was not created yet (there is any level 0 available in the list). If you see this in your ZDLRA, that means that INDEX_BACKUP task is in process.

If we continue to check more details and use the same query than before, we can see that backup changed the BS_KEY and BP_KEY in the internal tables (look that old keys disappears):

SQL> select bs_key, INCREMENTAL_LEVEL, HANDLE from rc_backup_piece where bp_key = 52141831;

no rows selected

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where bp_key = 52141831;

no rows selected

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024 from bp where tag = 'TAG20190816T150116';

    BP_KEY     BS_KEY TAG                 HANDLE                  BYTES/1024/1024/1024

---------- ---------- ------------------- ----------------------- --------------------

  52146236   52146235 TAG20190816T150116  VB$_90959062_52141864I             .02734375

SQL>

And now, checking in the VBDF table we can see that info at VCBP_KEY column appears

SQL> SELECT vb_key, ckp_id, df_key, db_key, blocks, vcbp_key, srcbp_key, file#, ckp_scn FROM VBDF WHERE srcbp_key = 52141831;

    VB_KEY         CKP_ID     DF_KEY     DB_KEY     BLOCKS   VCBP_KEY  SRCBP_KEY      FILE#        CKP_SCN

---------- -------------- ---------- ---------- ---------- ---------- ---------- ---------- --------------

  52141864  4932560891039     752220     752186 1655177216   52146236   52141831         29  4932560891039

SQL>

And if you search now at backup piece table you can see that exists one listed for this virtual backup key (check in the topic before that this info does not exist):

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where vb_key = 52141864;

    BP_KEY     BS_KEY TAG                 HANDLE                  BYTES/1024/1024/1024     VB_KEY

---------- ---------- ------------------- ----------------------- -------------------- ----------

  52146236   52146235 TAG20190816T150116  VB$_90959062_52141864I             .02734375   52141864

SQL>

And check that exists just one backup of this datafile at this SCN:

SQL> SELECT bdf_key, bs_key, file#, ckp_scn, incr_level from bdf where CKP_SCN = 4932560891039 and file# = 29;

   BDF_KEY     BS_KEY      FILE#        CKP_SCN INCR_LEVEL

---------- ---------- ---------- -------------- ----------

  52146237   52146235         29  4932560891039          1

SQL>

In this phase, you can see that the details for index and virtual full backup starts to appear internally. If you check, now you have plans for this virtual full backup:

SQL> select count(*) from plans where VB_KEY = 52141864;

  COUNT(*)

----------

         1

SQL>

SQL> select count(*), to_char(sysdate, 'HH24:MI') as time from plans_details where VB_KEY = 52141864;

  COUNT(*) TIME

---------- -----

   2294010 18:25

SQL> select count(*), to_char(sysdate, 'HH24:MI') as time from plans_details where VB_KEY = 52141864;

  COUNT(*) TIME

---------- -----

   5773720 18:43

SQL>

As you can see above, the plans_details for this virtual backup is increasing from time to time. This means that the q_restore_fast is running and it is reading blocks to create the virtual full backup/index (and inserting at plan_details).

We can figure out this, while the INDEX_BACKUP task is running, checking the sql_id for the session. While the task is running, we have three major sql’s (I used the same query than before – where we checked the bind for the session/task):

SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME  TASK_TYPE       STATE    MODULE          SQL_ID        ACTION                   BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- --------------- --------------- -------- --------------- ------------- -------------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B1

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B12

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B2

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B1

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B2

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B3

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B4

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B1   752220

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B3   1

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B4   1051071

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B11  0

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B10  4931849117847

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B9   4931848304742

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B8   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B7   752187

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B6   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B5   4932554641877

17 rows selected.                                                                                       

SQL>                                                                                                    

SQL> /                                                                                                 

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME  TASK_TYPE       STATE    MODULE          SQL_ID        ACTION                   BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- --------------- --------------- -------- --------------- ------------- -------------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B1   752220

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B3   403380104

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B4   404440007

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B11  0

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B10  4931849117847

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B9   4931848304742

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B8   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B7   752187

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B6   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B5   4932554641877

10 rows selected.

SQL>

SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE      MODULE                         SQL_ID        ACTION                   BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- -------------------- --------------- ---------- ------------------------------ ------------- -------------------- ---------- ---------- ---------- ---------- ----- --------------------

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B1

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B3   752220

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B2   4389973931270

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B1   4932554641877

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B4   1048576

SQL>

And the SQL text for them:

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = '9ma189ab8x9c7';

SQL_FULLTEXT

--------------------------------------------------------------------------------------------------------------------------------

INSERT /*+

             QB_NAME(q_restore_fast)

             OPT_PARAM('optimizer_dynamic_sampling' 0)

             OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

             OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

             OPT_PARAM('_optimizer_use_feedback' 'false')

           */ INTO PLANS_DETAILS (DF_KEY, TYPE, VB_KEY, BLKRANK, BLOCKNO, CHUNKNO, NUMBLKS, COFFSET, NUMBYTES) SELECT :B1 , :B12

, :B2 , 1 BLKRANK, BLOCKNO, CHUNKNO, NUMBLKS, COFFSET, NUMBYTES FROM TABLE( DBMS_RA_POOL.COLLAPSE(:B1 , :B2 , :B3 , :B4 ,

 CURSOR( SELECT /*+

                     QB_NAME(q_rfast_c)

                     INDEX_RS_ASC(@q_rfast_c b@q_rfast_c)

                     NO_INDEX_FFS(@q_rfast_c b@q_rfast_c)

                     NO_INDEX_SS(@q_rfast_c b@q_rfast_c)

                     OPT_PARAM('optimizer_dynamic_sampling' 0)

                     OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

                     OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

                     OPT_PARAM('_optimizer_use_feedback' 'false')

                   */ CKP_ID, BLOCKNO, SCN, CHUNKNO, USED, COFFSET, ENDBLK FROM BLOCKS B WHERE DF_KEY = :B1 AND BLOCKNO BETWEEN

:B3 AND :B4 AND SCN BETWEEN :B11 AND :B10 AND CKP_ID >= :B9 AND (CKP_ID <= :B8 OR DBINC_KEY <> :B7 ) AND (SCN < :B6 OR CKP_ID =

:B5 ) ORDER BY BLOCKNO, SCN, CKP_ID, CHUNKNO ) ) )

SQL>

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = 'brt6uuhzacdnu';

SQL_FULLTEXT

-------------------------------------------------------------------------------------------------------------------------------

SELECT /*+ QB_NAME ("Q_RFAST_C") NO_INDEX_SS ("B") NO_INDEX_FFS ("B") INDEX_RS_ASC ("B") */ "B"."CKP_ID" "CKP_ID","B"."BLOCKNO"

"BLOCKNO","B"."SCN" "SCN","B"."CHUNKNO" "CHUNKNO","B"."USED" "USED","B"."COFFSET" "COFFSET","B"."ENDBLK" "ENDBLK" FROM "BLOCKS"

"B" WHERE "B"."DF_KEY"=:B1 AND "B"."BLOCKNO">=:B3 AND "B"."BLOCKNO"<=:B4 AND "B"."SCN">=:B11 AND "B"."SCN"<=:B10 AND "B"."CKP_ID

">=:B9 AND ("B"."CKP_ID"<=:B8 OR "B"."DBINC_KEY"<>:B7) AND ("B"."SCN"<:B6 OR "B"."CKP_ID"=:B5) ORDER BY "B"."BLOCKNO","B"."SCN",

"B"."CKP_ID","B"."CHUNKNO"

SQL>

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = '8t81c0j1w9jqu';

SQL_FULLTEXT

--------------------------------------------------------------------------------------------------------------------------------

SELECT NVL(MAX(BLOCKNO), :B1 ), COUNT(*) FROM (SELECT /*+

                 QB_NAME(nbr)

                 INDEX_RS_ASC(@nbr b@nbr)

                 NO_INDEX_FFS(@nbr b@nbr)

                 NO_INDEX_SS(@nbr b@nbr)

                 OPT_PARAM('optimizer_dynamic_sampling' 0)

                 OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

                 OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

                 OPT_PARAM('_optimizer_use_feedback' 'false')

               */ BLOCKNO FROM BLOCKS B WHERE DF_KEY = :B3 AND BLOCKNO BETWEEN :B2 AND :B1 ORDER BY BLOCKNO) WHERE ROWNUM <= :B4

SQL>

As you can see, the sql_id 9ma189ab8x9c7 is responsible to insert into PLANS_DETAILS the blocks that are needed to create this virtual full backup. This is based in SCN, and means that ZDLRA create the index for all blocks that are below the SCN from the backup that was ingested originally. The sql_id brt6uuhzacdnu returns all the blocks that are needed (based on the SCN), and it is the same for sql_id 8t81c0j1w9jqu that verify if we have a block to index. These sql’s are the base for the q_restore_fast and they alternate itself until we reach 100%.

The idea for ZDLRA in this phase is to create the index that will be the base for the virtual full backup. And “create the index” means all the blocks that are needed to create the plan for this virtual full backup key.

INDEX_BACKUP, 100%

So, after the INDEX_BACKUP finish at ZDLRA, we have this at rman catalog:

RMAN> list backup of datafile 29 tag = TAG20190816T150116;

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52146235 Incr 1  28.00M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52146236   Status: AVAILABLE  Compressed: YES  Tag: TAG20190816T150116

        Handle: VB$_90959062_52141864I   Media:

  List of Datafiles in backup set 52146235

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52152458 Incr 0  12.67T     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52152459   Status: AVAILABLE  Compressed: YES  Tag: TAG20190816T150116

        Handle: VB$_90959062_52141864_29   Media:

  List of Datafiles in backup set 52152458

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   0  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

Look that for the same TAG we have now two backups, and one it is the virtual full backup. Check that SCN and TAG for both are the same. And if we check internally, at rman catalog view, about the backup pieces linked to the virtual backup key that was created by the index task we can see:

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where vb_key = 52141864;

    BP_KEY     BS_KEY TAG                 HANDLE                    BYTES/1024/1024/1024     VB_KEY

---------- ---------- ------------------- ------------------------- -------------------- ----------

  52152459   52152458 TAG20190816T150116  VB$_90959062_52141864_29            12975.3698   52141864

  52146236   52146235 TAG20190816T150116  VB$_90959062_52141864I               .02734375   52141864

SQL>

Look above that now we have two backups linked to the same virtual full backup (compare with the same SQL from the previous phases). The backup set key 52152458 is the virtual full backup and was generated during the phase q_restore_fast. The backup set key 52146235 was generated during the phase fixup_unordered.

And there is now, two backups for this datafile:

SQL> SELECT bdf_key, bs_key, file#, ckp_scn, incr_level from bdf where CKP_SCN = 4932560891039 and file# = 29;

   BDF_KEY     BS_KEY      FILE#         CKP_SCN INCR_LEVEL

---------- ---------- ---------- --------------- ----------

  52146237   52146235         29   4932560891039          1

  52152460   52152458         29   4932560891039          0

SQL>

Just to show that the task took a lot of time to finish:

SQL> select rt.elapsed_seconds from ra_task rt where task_id = 56242962;

ELAPSED_SECONDS

---------------

     14563.7986

SQL>

SQL> select count(*) from plans_details where VB_KEY = 52141864;

  COUNT(*)

----------

  74505579

SQL>

INDEX_BACKUP, The internals

As you can see above, the INDEX_BACKUP task it is responsible to generate the virtual full backup and “fix” the rman catalog views to represent the new backups. This is done in two major phases (fixup_unordered and q_restore_fast) to read the backup that was ingested at ZDLRA and index all the blocks. The other phases are process_allfiles and plan_blocks, but they are fast to execute.

About the blocks, it is important to hint that it is completely based in SCN/CKP number. The index creation will search for all blocks that are bellow of the SCN of ingested backup. If you check the details, the backup file does not exist, by the way, it is just a huge list of the blocks need to this SCN (information stored at plans_details table).

The procedures, SQL, steps, phases that I showed before will have different values in your case. The DF_KEY, BS_KEY, and others will be different. But I think that you can understand the logic of how ZDLRA index the backup and internally generate the virtual full backup. I know that it is not easy to read all the numbers and link between sql’s and tables, but the logic it is simple: verify all the blocks that belong to datafile and create one plan that points to every block needed by the virtual full backup for one scn.

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

ZDLRA Internals, Tables and Storage

Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA Internals, Tables and Storage

ZDLRA tables are owned by rasys user and are one extension of rman recovery catalog views. The internal tables that were added are just a few, but are important to understand how tasks works (mainly INDEX_BACKUP). In this post, I will show tables like plans, plans_details, blocks, and chunks. And besides that, show a little glimpse about how the files are stored.

Extra tables

If you check, officially, the rman recovery catalog already includes some columns from ZDLRA, the column VB_KEY identify the virtual backup key inside ZDLRA. It exists in RC_BACKUP_PIECE as an example. The extra tables added are linked with SBT tasks (clone to tape or cloud), replication (to other ZDLRA), storage (chunks), vbdf and plans (virtual full backup).

The most import tables are listed below.

RASYS.VBDF: This table store the reference for virtual full backups created for the datafile. Have the information about the backup piece used as the source(base) for virtual full backup and the reference to the new backup piece that was generated at virtual full creation. The most import columns are:

VB_KEY: The unique identifier to virtual backup.
DF_KEY: The reference to datafile key for virtual full backup.
CKP_ID and CHP_SCN: Checkpoint information (id/scn) for this virtual full backup.
SRCBP_KEY: The reference to the original backup piece (source) that was used to generate the virtual full backup. Links to table BP.
VCBP_KEY: Reference to the derived backup piece that was created during the index of the original (source) backup piece.

RASYS.PLANS: This table store the plans that link the datafile and the virtual full backup. Basically, is the reference “index” and links: virtual backup, datafile, and the number of chunks that store it. Important columns:

VB_KEY: The unique identifier to virtual backup.
DF_KEY: The reference to datafile key for virtual full backup.
NUMCHUNKS: Chunks used to store (or store) this virtual full backup.

RASYS.PLANS_DETAILS: IOT table that stores the detailed information for the plan for the “index”. It is partitioned over the DF_KEY column. Link the virtual full backup, datafile, block number, and chunk identification. Tells you in which chuck it is stored each block that are part of the virtual full backup. Import columns:

VB_KEY: The unique identifier to virtual backup.
DF_KEY: The reference to datafile key for virtual full backup.
BLOCKNO: Block number that are part of the datafile.
NUMBLKS: Number of the blocks in sequential mode starting from BLOCKNO that are equal.
CHUNKNO: What is the chunk id that store the BLOCKNO + NUMBKS.

RASYS.BLOCKS: IOT table that stores the info for each block that belongs to the datafile. It is partitioned over the DF_KEY column. There is no link to any backup, but store each version of a single block that is needed to restore any backup (the block number needed can be found in the PLAN_DETAILS and SCN come from other tables). Import columns:

DF_KEY: The reference to datafile key for virtual full backup.
BLOCKNO: Block number that are part of the datafile.
SCN and CKP_ID: Checkpoint and SCN for this specific version of the block.

RASYS.CHUNK: IOT table that has the information about the chunks that was used to store the backups for the datafile. Important columns:

DF_KEY: The reference to datafile key for virtual full backup.
CHUNKNO: What is the chunk id.
FILESIZE: Size for this chunk
FILENAME: Derived file name that store the chunk (no represent the file inside ASM).

These tables are the most important for ZDLRA for the internal part of ZDLRA. They are not needed in daily basis activities but I think that is useful if you want to understand how things work internally. The list above is just a glimpse for internal tables, but based on that you can easily: identify what are the datafile blocks that belongs for specific virtual backup, where the blocks are stored (which chunk), and how this virtual full backup was build (the plan that contains all the blocks).

Storage

As you saw above, the blocks are stored in chunks and in CHUNKS table we have filename information. But this is not related where the data are really stored, there are other places used too. All of these tables are stored at RA_POOL tablespace but don’t store the backup itself. The structure that does this is “CONTAINERS” and represent files inside ASM diskgroup.

If you see the ZDLRA doc (look the image below) this is called DELTA POOLS (CONTAINERS and CHUNKS) and is stored inside one “Delta Store” (ASM diskgroup). For ASM diskgroups you can see the CATALOG and DELTA. The CATALOG, as I wrote in this post, store the metadata information about the backups (mostly the rman catalog views) and the DELTA (the Delta Store) have the info related the backups itself.

For tablespaces, there are two important RA_POOL and RA_CATALOG. The tables listed before (in the “Extra Tables topic”) resides at RA_POOL tablespace. But the backup data itself is stored in CONTAINER files at DELTA diskgroup and you can check them in the table SYS.AMCONT$. Another option is going in ASM diskgroup and check the list at DELTA/CONTAINER files (each file have the default of 2TB):

ASMCMD> pwd

+DELTA/ZDLRA/CONTAINER

ASMCMD> ls -s

Block_Size      Blocks          Bytes          Space  Name

       512  4294959104  2199019061248  4398059094016  con.561.935405781

       512  4294959104  2199019061248  4398059094016  con.562.935405789

       512  4294959104  2199019061248  4398059094016  con.563.935405797

       512  4294959104  2199019061248  4398059094016  con.564.935405805

…

…

       512  4294959104  2199019061248  4398059094016  con.656.935406489

       512   308051968   157722607616   315692679168  con.657.935406495

ASMCMD>

You can see more info checking tables SYS.AM* tables and views (SYS.AMAU$, SYS.AMCONT$, and SYS.AM$CONTAINER are examples). The list for containers files are not even listed as ZDLRA database (as datafile in dba_data_files) but are created directly by ZDLRA library.

You can even test the performance (before starting to use ZDLRA usage) to create these files using rastat.pl: https://docs.oracle.com/cd/E88198_01/AMAGD/amagd_monitor.htm#AMAGD1503. And this is the only info in the docs about CONTAINERS files.

Last info about ZDLRA database and storage, the database operates in NOARCHIVLOG mode. The backups for it are daily exports from the metadata information and in case of corruption/lost you can reconstruct the information reading the con files.

Conclusion

Understand the tables involved for ZDLRA are important to see what’s happening internally, how the index is stored and how works virtual full backup. The tables listed above are not used in daily bases for ZDLRA, you usually go to rman catalog view (RC_*), everything is there.

But, PLANS, PLANS_DETAILS, and VCBP are the base for a task like INDEX_BACKUP and to sustain virtual full backups. I needed to pass over these tables before explaining in more details the virtual full creation and INDEX_BACKUP task.

The storage part, containers and ASM diskgroup are needed just to help support to identify some bottleneck or problem. You don’t need to use it but I think that it is important to show you, at least superficially, how its work.

Understanding ZDLRA

Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Understanding ZDLRA

Zero Data Loss Recovery Appliance (ZDLRA) deliver to you the capacity to improve the reliability of your environment in more than one way. You can improve the RPO (Recovery Point Objective) for your databases until you reach zero, zero data loss. And besides that, adding a lot of new cool features (virtual backups) on the way how you do that your backups (incremental forever), and backup strategy. And again, besides that, improve the MAA at the highest level that you can hit.

But this is just marketing, right? No, really, works pretty well! My history with ZDLRA starts with Oracle Open World 2014 when they released the ZDLRA and I watched the session/presentation. At that moment I figure out how good the solution was. In that moment, hit exactly the problem that I was suffering for databases: deduplication (bad dedup). One year later, in 2015 at OOW I made the presentation for a big project that I coordinate (from definition implementation, and usage) with 2 Sites + 2 ZDLRA + N Exadata’s + Zero RPO and RTO + DG + Replication. And at the end of 2017 moved to a new continent, but still involved with MAA until today.

This post is just a little start point about ZDLRA, I will do a quick review about some key points but will write about each one (with examples) in several other dedicates posts. I will not cover the bureaucratic steps to build the project like that, POC, scope definition, key turn points, and budget. I will talk technically about ZDLRA.

ZDLRA is one Engineered System built over Exadata. Contains Exadata Storage and Database nodes, InfiniBand network, SAS drivers (for tape connection). So, a good machine with redundancy components. Look the base config:

But as Exadata, it is not just hardware, it is software too (and a lot of in this case). ZDLRA solve problems in more than one topic, it is not just a backup appliance where you redirect your backups, it is an appliance that provides zero data loss. To do that the key features are:

Virtual backups: Based on the input backups (incremental) generate a virtual full backup for you.
Real-time Redo: ZDLRA can be an archive log destination for your database, RPO zero.
Replication: You can replicate data between ZDLRA’s, eliminating single point of failure.
Tape integration: ZDLRA used OSB to copy backups to tape.
Rman: totally compatible with rman commands, and deliver to you rman catalog self-managed.
Database and MAA integration: fully integrated with MAA environments, including DG.

Since it is built over Exadata, the updates follow the same procedure, IB, Storage, Db’s, and after that ZDLRA software. But in this case, you don’t need to concern about integration or tuning for Exadata, everything is done by Oracle. For Storage, ZDLRA uses two diskgroups: DELTA (to store the backups) and CATALOG (to store the rman catalog tables).

The integration between ZDLRA and your databases are simple. You have an SBT_TAPE library to use and the rman catalog database. The library communicates with ZDLRA appliance and sends the blocks, and the rman catalog store the metadata.

Talking about rman catalog, using ZDLRA you need to use the rman catalog. The catalog database that you connect it is the ZDLRA itself. And the catalog it is managed by ZDLRA software, this means that when you send the backup to ZDLRA tape driver, and the ZDLRA process it, the catalog reflect all the new virtual full backups that are available. You don’t need to manage the backups, everything is done by ZDLRA software (delete backup, crosscheck, validation, etc).

Virtual backups

Probably this is the most know feature for ZDLRA. In the most basic description, it “simple” create to you a new backup level 0 just using incremental level 1 backups. The functionality is simple:

Ingest one first rman level 0 backup.
Ingest subsequent rman level 1 backup.
A new “virtual level 0” appears available to you.

But how it’s work? Basically, ZDLRA open the rman block and read the datafile blocks that are inside of rman block. After that, using the already existed backup (stored inside ZDLRA) for that datafile, generate to you a new level 0 backup. This is called “incremental forever strategy”, where (after first level 0) you just need to do incremental backups.

If you are thinking about deduplication at this moment, you are almost right. But is really different of other vendors that just index the rman block (by hash, md5, whatever), ZDLRA it is the only one that opens the block and look what it is inside. It’s pick up the content of rman block, the data blocks for each datafile that is inside, and index it.

But why it is better than deduplication? Because it is sensitive to the context. And in this case, it is Oracle context. Work’s better too because can handle every kind of database blocks sizes, any kind of filesperset and chunks that you specify. You don’t need to “black magic” to minimally have good values for dedup, like DataDomain as example: https://community.emc.com/docs/DOC-24118 and https://community.emc.com/docs/DOC-18775. You can argue that DataDomain can do the same, it opens the rman block (just 8k and 16k blocks), but it was done by reverse engineering over the sent block https://www.emc.com/collateral/white-papers/h15389-data-domain-oracle-optimized-deduplication.pdf. But if Oracle changes this pattern? Sometimes can happen that rman read the same datafile blocks (that was not touched by dbwr) but because sga/pga buffers create a different rman block.

A little representation. Think that you work in a Library and I deliver to you (every day) one box with books of an encyclopedia. You store this box but can’t open it to store the books that are inside. You can ignore the box and says that you already have this based on the size, weight, whatever you choose to define if you already have this box. As ZDLRA, you receive the same box, but you open and check if you already have this book in your collection or no. One day I come back and ask you the full encyclopedia. So, what solution do you think that will be better for Oracle backups?

The first is the normal deduplication. Works very well for a lot of blocks types (word, excel, txt) because you “maybe” have the key to open the box. But not so well for binary blocks type (like rman). ZDLRA it is Oracle, and have the key to open the box (rman block) and check the content to identify books (the datafile blocks). Clever no?

There are a lot of other details to cover, I will do in a delicate post about Virtual backups.

Real-Time Redo

The name from “zero data loss” came from this feature because its guarantee zero RPO for your database. In a simple way, ZDLRA can be archive log destination for your database. It’s work the same way that you do for DG configuration, but more simply because you just need to configure the archive_log_dest.

You can think that it is similar, or you can do the same, using DG Far Sync: https://www.oracle.com/technetwork/database/availability/8395-data-guard-far-sync-1967244.pdf. But here for ZDLRA, you don’t need standby database to be protected and every transaction still will be protected.

It is important to say that it is more than just archive log destination because can sync the redo log buffers of your database. Don’t need to wait archive be generated to sync. Can work sync or async, and if your database has an outage (by storage death as an example), ZDLRA generates for you one archivelog. And another thing it is that there is no need to do archivelog backups (if you use real-time redo) because ZDLRA detect the log switch and generate for you (inside catalog) the archivelog.

Besides that, if you use ZDLRA replicated, you can have additional protection sending to another site. I will talk later about that but using replicated ZDLRA’s, you can have RPO zero for DG databases and for non DG protected databases.

Oracle calls Virtual Backups + Real-Time Redo as Delta Push because you just sent to ZDLRA the minimum data needed to be protected. Just the delta difference. This reduces a lot the load over the system, network, avoid reading unusual/repeated data from the source database and inside ZDLRA too.

Replication

ZDLRA can work replicating backups between two ZDLRA’s. This replication is done automatically, using a dedicated network and can be:

One way: You have the ZDLRA master (called upstream) and the destination (called downstream).
Bi-Directional: Two ZDLRA’s replicate backups between each self. Both are upstream and downstream.
Hub-and-Spoke: You have one downstream ZDLRA that receive the backups from more than one upstream.

The detail here is that replication is an additional feature of ZDLRA usage, even a downstream ZDLRA can protect one site (like standby site), receiving backups from databases as a normal ZDLRA. The downstream ZDLRA does not lock in as just destination from upstream. But since it is a replicated environment, you can restore your backupset from the downstream replica (connected in the upstream ZDLRA).

Based on ZDLRA design, the rman catalog is updated too. This means that after the backup being replicated to the downstream, inside of the catalog automatically appears another copy for your backupset. And you can have different policies between ZDLRA’s. So, in the upstream you can have one policy with retention as 30 days (because maybe you need to do more backups) and the downstream have retention as 90 days.

Tape and Cloud integration

ZDLRA can sent backups to media manager (tapes), to do that the recommended way is use the pre-installed OSB (Oracle Secure backup) as media manager library. ZDLRA manage everything, used the dedicated SAN network to send the backups and manage the tape drivers. Another option is sync ZDLRA with Oracle Cloud object storage.

But you can use third part software as tape media manager driver, you just need to install the agent in database nodes (of ZDLRA) and configure it as media manger library. It is not trivial, but you can do and integrate your tape library with ZDLRA. In this case all the communication (copy from ZDLRA to tape) will be done using LAN.

Again, the rman catalog will be automatically managed by ZDLRA. When the backupset it is copy to tape, the copy appear available for restore in your rman catalog.

To manage the tape backups you use the same policies that you use for backups. You can specify the retention period, copies, and more. If you use OSB, ZDLRA will manage everything (free tapes as example).

For Cloud, the idea is the same. You have one plugin that appears in your ZDLRA as a tape library, and the jobs clones from ZDLRA to cloud (in the same way than tape). But since Oracle Cloud is fully encrypted, you have some steps to manage the key to allow you to integrate the system. This can be one way to have the third site to store your backups outside of your environment.

Rman

As told before, ZDLRA has two main parts: the software itself (that create the virtual backups) and the database. And this database contains the rman catalog that you use to connect. ZDLRA manage automatically the rman catalog database, this means that every backup is sent appears in the catalog. And everything that happens with this backup, the derived virtual full backup, clone to tape, replication, exclusion because retention expiration is reflected automatically in the rman catalog. You don’t need to manage or catalog the backupsets.

But the most important thing is that everything that you already made with rman works 100% with ZDLRA. You don’t need to learn new commands, new backups types. Everything that you need it is “backup incremental level”.

Since ZDLRA will be a central part of your environment, it is important to know how to search for data in the catalog (using SQL). I will show some examples of how to understand the RC_* tables/views that will allow you to create some reports from your backups.

Database and MAA integration

Since it is really linked with MAA family the integration between ZDLRA and databases is quite easy, covering all the points for protection. About protection, not just for backup, but raising the bar for transaction protection to reach zero data loss in any kind of scenario (DG and non DG databases).

ZDLRA it is not just a backup appliance, it allows you to reach RPO zero even not using DG. But if you are using DG you need to take attention to some details. The idea about DG for MAA team it is that everyone does your job, this means that if you need to protect a database for site outage, DG does that. ZDLRA it is not a full standby database.

Saying that, when you have a full DG environment, you can reach RTO (Recovery Time Objective) as zero too. ZDLRA alone can provide to you just RPO zero but does not help you to have zero downtime (or unavailability). You don’t not loose transactions with RPO zero, but will need some time to recover the database. The only way to have both, RTO and RPO zero is using DG. In this scenario, each database does the backup in ZDLRA that it is in the same site, and the DG does the sync between the sited (for DB with DG).

If you have more than one site and want to protect everything the best solution is have ZDLRA in both sites. You can still have just one ZDLRA and protect everything, but remember that you need to choose one site (or a third one) to put your ZDLRA.

But the good part is that you can mix everything. There are a lot of cases that some databases are not protected by DG. For these, you can use real-time redo to have RPO zero, and replicate to another ZDLRA in another site to avoid the only point of failure.

This raise the bar for MAA, because it is the highest level that you can reach in protection. Even if you cut the communication for each site you still have RPO zero because ZDLRA protection. And if you have a complete site or server failure, the RTO will be zero because DG.

This was the project that I designed and you can see this (was presented in OOW 2015: https://www.oracle.com/technetwork/database/availability/con8830-zdlradeepdive-2811109.pdf).

Conclusion

ZDLRA it is not just a backup appliance, it is more than that. It allows you to reach RPO zero for any kind of environment. And together with DG, you can reach zero RPO and RTO.

But besides that, the ZDLRA software delivers to you a lot of features that help you in daily bases activities. The virtual backups deliver to you a validated level zero backupset for each datafile. You receive a rman catalog self-managed, you don’t need to catalog (or uncatalog) pieces for a clone to tapes, deletion, replications that you do.

And there is no need to learn complex new features, you continue to do the backups in the same way that you do previously. “backup incremental level 1” and just that.

For the next post, I will try to cover feature by feature and a lot of details, virtual backups, real-time redo, dg integration, internal tasks, updates. Stay tuned.

References

All the images and base info for this post came from these links:

https://www.oracle.com/technetwork/database/availability/con8830-zdlradeepdive-2811109.pdf

https://www.oracle.com/technetwork/database/availability/oow14-con7684-zdlra-techoverview-2332009.pdf

https://www.oracle.com/technetwork/database/availability/8395-data-guard-far-sync-1967244.pdf

https://docs.oracle.com/cd/E55822_01/AMOGD/E37432-22.pdf

https://docs.oracle.com/cd/E55822_01/AMAGD/E37431-29.pdf

http://www.oracle.com/us/products/engineered-systems/recovery-appliance-twp-2313693.pdf

https://www.oracle.com/technetwork/database/availability/recoveryapplianceperformance-2636208.pdf

https://www.oracle.com/technetwork/database/availability/recovery-appliance-businesscase-2574428.pdf

https://www.oracle.com/technetwork/database/availability/recovery-appliance-data-guard-2767512.pdf

https://www.oracle.com/technetwork/database/features/availability/ow-papers-089200.html

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

INDEX_BACKUP task for ZDLRA, Percentage done

Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

INDEX_BACKUP task for ZDLRA, Percentage done

Quick post how to check and identify done for INDEX_BACKUP task in ZDLRA. In one simple way, just to contextualize, INDEX_BACKUP is one task for ZDLRA that (after you input the backup of datafile) generate an index of the blocks and create the virtual backup for you.

Here I will start a new series about ZDLRA with some hints based on my usage experience (practically since the release in 2014). The post from today is just little scratch about ZDLRA internals, I will extend this post in others (and future posts), stay tuned.

INDEX_BACKUP

As told before this task is one of the most important for ZDLRA, but it can take a lot of time to index the backup if the file is huge (a lot of TB’). Think that ZDLRA need to index all the blocks for the file, and for the first level 0 this took some time.

Unfortunately, officially there is no way to check how far you are from the end and how much was already done to create the index. The table that shows you the info for running tasks doesn’t contain this information. Because of that, I created a little SQL that you can use to verify the task and try to discover the percentage of already done.

SQL

In the SQL I will use two tables from RASYS ZDLRA schema:

RA_TASK: The table RA_TASK list the tasks processed/in wait/running. Documented here (https://docs.oracle.com/cd/E88198_01/AMAGD/amagd_views.htm#AMAGD1440)
SESSION: Internal table for ZDLRA that store all the internal sessions running tasks and other things. This is not documented table/view.

The SQL:

SELECT s.ba_session_id, s.instance_id, s.sid, s.serial#, s.job_name

       , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state, rt.waiting_on

       , rt.elapsed_seconds

       , gs.module, gs.sql_id, gs.action, gs.event

       , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

FROM sessions s

JOIN ra_task rt

ON rt.task_id = s.current_task

JOIN gv$session gs

ON gs.inst_id = s.instance_id

AND gs.sid = s.sid

AND gs.serial# = s.serial#

ORDER BY rt.LAST_EXECUTE_TIME DESC

This query is a join between RA_TASK, SESSION (from ZDLRA), and V$SESSION from DB side. As you can see, the link between RA_TASK and SESSION is the current task. And since you reach the SESSION info from ZDLRA, you can link with the sid, serial#, and inst_id running the task. Fortunately, the V$SESSION has the desired information. This query returns all running tasks, you can filter by task type if desired.

Look the result from one execution:

SQL> col JOB_NAME format a30

SQL> col STATE format a10

SQL> col DB_UNIQUE_NAME format a20

SQL> col MODULE format a30

SQL> col ACTION format a20

SQL> col EVENT format a35

SQL> col TASK_TYPE format a15

SQL> set linesize 400

SQL> l

  1  SELECT s.ba_session_id, s.instance_id, s.sid, s.serial#, s.job_name

  2         , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state, rt.waiting_on

  3         , rt.elapsed_seconds

  4         , gs.module, gs.sql_id, gs.action, gs.event

  5         , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

  6  FROM sessions s

  7  JOIN ra_task rt

  8  ON rt.task_id = s.current_task

  9  JOIN gv$session gs

 10  ON gs.inst_id = s.instance_id

 11  AND gs.sid = s.sid

 12  AND gs.serial# = s.serial#

 13* ORDER BY rt.LAST_EXECUTE_TIME DESC

SQL> /




BA_SESSION_ID INSTANCE_ID        SID    SERIAL# JOB_NAME                          TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE      WAITING_ON ELAPSED_SECONDS MODULE                         SQL_ID        ACTION               EVENT                                   BP_KEY     BS_KEY     DF_KEY     VB_KEY

------------- ----------- ---------- ---------- ------------------------------ ---------- ---------- -------------------- --------------- ---------- ---------- --------------- ------------------------------ ------------- -------------------- ----------------------------------- ---------- ---------- ---------- ----------

     93729765           2       4453      42235 RA$_EXEC_93730430                93676068     477602 SIMON_DB1            INDEX_BACKUP    RUNNING                    28558.1086 q_restore_fast                 brt6uuhzacdnu plan 12% done        cell single block physical read       41032043




SQL>

Columns import here:

ACTION: from GV$SESSION show the percentage already done for this INDEX_BACKUP task.
ELAPSED_SECONDS: from RA_TASK, show seconds that this task is running. You can use together with the percentage and try to figure out the time to reach 100%.
BP_KEY/DF_KEY/BS_KEY/VB_KEY: from RA_TASK, show base info from what is the input to task. In this case, and usually, for INDEX_BACKUP tasks the only column will be BP_KEY (that represent the backup piece key id) because you are indexing a new backup that entered in the ZDLRA.

As you can see, the INDEX_BACKUP task 93676068 for the backup piece 41032043 took 28558 seconds and it is running without a wait and processed 12% of the backup piece. We can dig a little more try to identify and validate some values from blocks processed, but I will cover in the second part.

Conclusion

As you can see, with this simple query you can check the percentage done for the INDEX_BACKUP task. Was needed to query some internal tables of database, but the desired result was easy to find. I tested this query with old version of ZDLRA (12.2.1.1.1-201805 from August of 2018) and with the last version (12.2.1.1.2-201907 from 11/July) and the result was the same, and always accurate.

I will post more about ZDLRA, from usage experience to internal details.

You can check my presentation at OOW 2015 about my MAA project that used ZDLRA to protect all environment: https://www.oracle.com/technetwork/database/availability/con8830-zdlradeepdive-2811109.pdf

ZDLRA, since 2014

Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA, since 2014

In Oracle Open World 2014 the Zero Data Loss Recovery Appliance (ZDLRA) was released and it changed MAA in many ways, but two principals: protection and backup. I watched the ZDLRA presentation and saw that matched with the needs that I had that time.

After OOW in 2014 I started the project (all phases, from conception, requirements until deployments and usage) that become (in 2015) the first ZDLRA installation in Brazil, and one of the first of the world too that use replicated ZDLRA to protect both sites (primary and standby) and many levels of databases (PRO, TST, DEV). The Oracle MAA at its finest was amazing: ZDLRA + Exadata + DG; everything integrated to protect both sites.

Because of the high design level of the project it was chosen to be one of the main presentation in Oracle Open World 2015 about ZDLRA, you can find the link of the presentation that I made together with ZDLRA dev team here. As told before, in this project was integrated two ZDLRA, two Exadata and DG to reach ZERO Recover Point Objective (RPO) and Recovery Time Objective (RTO) and beside that, reduce backup time. You can see the presentation to check the scope and other details.

I will start to write more about ZDLRA, from technical part until project details. How you use and how it impact in your MAA projects. Just to remember that ZDLRA it is not just backup oriented appliance, it is design to protect mission critical environments, where the goal is zero data loss; but also help you to improve your backup and recovery environment. Of course that you can use ZDLRA to protect all of the Oracle databases, from Sparc, RISC and x86; from DEV to PROD; from single to RAC DG environment.

ZDLRA Internals, INDEX_BACKUP task in details

I recommend that you check my previous post about ZDLRA: ZDLRA Internals, Tables and Storage, ZDLRA, Virtual Full Backup and Incremental Forever, and Understanding ZDLRA. They provide a good base to understand some aspects of ZDLRA architecture and features.

Backup

This generation is done thought INDEX_BACKUP task and occurs for every backup (level 0 or 1) that enter in the system. For this post, I already have a full backup level 0 inside ZDLRA and I made one new incremental level 1:

As you can see, it is a normal backup. And the details for this backup:

Here it is important to remember the TAG, backupset(BS)/backup piece (BP) numbers, and the SCN for the backup made.

INDEX_BACKUP

The INDEX_BACKUP appears as a single task inside ZDLRA, but have two major phases: fixup_unordered and q_restore_fast. And each one impacts differently inside the rman catalog and in the internal tables.

So, after the backup finish, inside of ZDLRA we can see the index task running:

With this SQL you can check all the INDEX_BACKUP tasks running at ZDLRA. And the SQL use the tables and report:

RA_TASK: All tasks running inside ZDLRA.

SESSIONS: Sessions running inside ZDLRA, used to get the database session running the task

GV$SESSION: Databases sessions, used to get the sql_id and others info

BP_KEY: Identify the backup piece key that it is indexing. It is the start point to identify everything here.

Above you can see that task 56242962 is running, processing the BP_KEY (backup piece key) 52141831, running the module fixup_unordered, processed 6% of this step, and running SQL_ID 1q0zr86n25pvu.

INDEX_BACKUP, fixup_unordered

Going further we can check the VBDF (virtual backup data file table – store the virtual full backups information) table and verify more details:

At VBDF table we can discover:

VB_KEY: Virtual backup key.

CKP_ID: Checkpoint and SCN for this virtual backup.

BLOCKS: Number of the blocks for this virtual backup.

VCBP_KEY: Is null at fixup_unordered because the virtual backup piece (VCBP) was not yet generated (it is doing by fixup_unordered). This will change in the future and I will show more details.

SRCBP_KEY: The source/original backup piece used to create this virtual full backup.

So, we can check that the backu is in the middle of the index the process since the VCBP_KEY is not yet available. But besides that, we can validate/check more details about the backups.

Check that does not exist a backup piece associated with this virtual backup key:

And that exists just one backup of datafile for this SCN and FILE#:

Digging more, we can see that are zero plans for this virtual backup (will be zero until fixup_unordered finish – more detail further):

During this phase of INDEX_BACKUP, the major SQL_ID is 1q0zr86n25pvu:

And the bind variables in this case are:

So, the idea here for ZDLRA index task is processing get/check all the blocks that have SCN below of the SCN from ingested backup. And since the INDEX_BACKUP result creates one virtual full backup, all the blocks for this datafile need to be processed.

If you check this query time to time, you will see the increase of scan % increasing until the :B3 and :B4 are equal (compare with the previous execution):

INDEX_BACKUP, q_restore_fast

After the fixup_unordered reach 100% of the scan, the phase q_restore_fast starts and this will create the virtual backup itself. But before let’s see what’s happened at rman catalog after the previous phase finished:

If we continue to check more details and use the same query than before, we can see that backup changed the BS_KEY and BP_KEY in the internal tables (look that old keys disappears):

And now, checking in the VBDF table we can see that info at VCBP_KEY column appears

And if you search now at backup piece table you can see that exists one listed for this virtual backup key (check in the topic before that this info does not exist):

In this phase, you can see that the details for index and virtual full backup starts to appear internally. If you check, now you have plans for this virtual full backup:

As you can see above, the plans_details for this virtual backup is increasing from time to time. This means that the q_restore_fast is running and it is reading blocks to create the virtual full backup/index (and inserting at plan_details).

We can figure out this, while the INDEX_BACKUP task is running, checking the sql_id for the session. While the task is running, we have three major sql’s (I used the same query than before – where we checked the bind for the session/task):

The idea for ZDLRA in this phase is to create the index that will be the base for the virtual full backup. And “create the index” means all the blocks that are needed to create the plan for this virtual full backup key.

INDEX_BACKUP, 100%

So, after the INDEX_BACKUP finish at ZDLRA, we have this at rman catalog:

Look that for the same TAG we have now two backups, and one it is the virtual full backup. Check that SCN and TAG for both are the same. And if we check internally, at rman catalog view, about the backup pieces linked to the virtual backup key that was created by the index task we can see:

And there is now, two backups for this datafile:

Just to show that the task took a lot of time to finish:

INDEX_BACKUP, The internals

ZDLRA Internals, Tables and Storage

Extra tables

The most import tables are listed below.

VB_KEY: The unique identifier to virtual backup.

DF_KEY: The reference to datafile key for virtual full backup.

CKP_ID and CHP_SCN: Checkpoint information (id/scn) for this virtual full backup.

SRCBP_KEY: The reference to the original backup piece (source) that was used to generate the virtual full backup. Links to table BP.

VCBP_KEY: Reference to the derived backup piece that was created during the index of the original (source) backup piece.

RASYS.PLANS: This table store the plans that link the datafile and the virtual full backup. Basically, is the reference “index” and links: virtual backup, datafile, and the number of chunks that store it. Important columns:

VB_KEY: The unique identifier to virtual backup.

DF_KEY: The reference to datafile key for virtual full backup.

NUMCHUNKS: Chunks used to store (or store) this virtual full backup.

VB_KEY: The unique identifier to virtual backup.

DF_KEY: The reference to datafile key for virtual full backup.

BLOCKNO: Block number that are part of the datafile.

NUMBLKS: Number of the blocks in sequential mode starting from BLOCKNO that are equal.

CHUNKNO: What is the chunk id that store the BLOCKNO + NUMBKS.

DF_KEY: The reference to datafile key for virtual full backup.

BLOCKNO: Block number that are part of the datafile.

SCN and CKP_ID: Checkpoint and SCN for this specific version of the block.

RASYS.CHUNK: IOT table that has the information about the chunks that was used to store the backups for the datafile. Important columns:

DF_KEY: The reference to datafile key for virtual full backup.

CHUNKNO: What is the chunk id.

FILESIZE: Size for this chunk

FILENAME: Derived file name that store the chunk (no represent the file inside ASM).

Storage

You can see more info checking tables SYS.AM* tables and views (SYS.AMAU$, SYS.AMCONT$, and SYS.AM$CONTAINER are examples). The list for containers files are not even listed as ZDLRA database (as datafile in dba_data_files) but are created directly by ZDLRA library.

You can even test the performance (before starting to use ZDLRA usage) to create these files using rastat.pl: https://docs.oracle.com/cd/E88198_01/AMAGD/amagd_monitor.htm#AMAGD1503. And this is the only info in the docs about CONTAINERS files.

Last info about ZDLRA database and storage, the database operates in NOARCHIVLOG mode. The backups for it are daily exports from the metadata information and in case of corruption/lost you can reconstruct the information reading the con files.

Conclusion

Understand the tables involved for ZDLRA are important to see what’s happening internally, how the index is stored and how works virtual full backup. The tables listed above are not used in daily bases for ZDLRA, you usually go to rman catalog view (RC_*), everything is there.

But, PLANS, PLANS_DETAILS, and VCBP are the base for a task like INDEX_BACKUP and to sustain virtual full backups. I needed to pass over these tables before explaining in more details the virtual full creation and INDEX_BACKUP task.

The storage part, containers and ASM diskgroup are needed just to help support to identify some bottleneck or problem. You don’t need to use it but I think that it is important to show you, at least superficially, how its work.

Understanding ZDLRA