Engineer System
ZDLRA Internals, Virtual Full Backup
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA Internals, Virtual Full Backup

Virtual Full Backup probably is the most know feature of Oracle Zero Data Loss Recovery Appliance (ZDLRA) and you can check here how it works. In this post I will show how virtual full backup works internally and integrate INDEX_BACKUP task with tables like PLANS, PLAN_DETAILS, CHUNKS, and BLOCKS.
About the internal tables, you can check my previous post “ZDLRA Internals, Tables and Storage” where I explained details about that. To understand the INDEX_BACKUP task, check my post “ZDLRA Internals, INDEX_BACKUP task in details”. But if you know nothing and want to start reading about ZDLRA, you can check my post “Understanding ZDLRA” and check all the features and details.
The base for this article is virtual full backup and incremental forever strategy. I explained both at “ZDLRA, Virtual Full Backup and Incremental Forever” and I included hot it’s work integrated with rman backup. Basically, after an initial backup level 0, you execute just level 1 backups and ZDLRA generated a virtual backup level 0. But here, in this post, I will show you how it works in some internal details.

 

Database Environment

In this post, I used a new datafile with 1MB (block size of 8k) and I create a small table to load some data. After every load, I took backup for the datafile (level 1). The idea is to show how ZDLRA will index the backups, generate, and store internally the virtual full backup. The database here runs over 19c, and the ZDLRA it is running 19c version too. But, this works the same for every database version supported, and for every ZDLRA version.
Creating the tablespace, checking the datafile, and creating the table:

 

SQL> create tablespace simon datafile '/u01/app/oracle/oradata/ORCL19/simon01.dbf' size 1m autoextend on next 1m maxsize 10m;




Tablespace created.




SQL>

SQL> select file_id from dba_data_files where tablespace_name = 'SIMON';




   FILE_ID

----------

         5




SQL>

SQL> create table test(c1 decimal(1,0), c2 varchar2(128)) tablespace simon;




Table created.




SQL> insert into test (c1, c2) values (0, 'SIMON');




1 row created.




SQL> commit;




Commit complete.




SQL>

 

And after that, doing the backup level 0 for the datafile:

 

RMAN> list backup of datafile 5;




specification does not match any backup in the repository




RMAN> BACKUP INCREMENTAL LEVEL 0 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 5;




Starting backup at 22-09-2019_17:54:27

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=75 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRAS1) SID=9327516A92A43E0DE053010310ACCB56

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL19/simon01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 22-09-2019_17:54:28

channel ORA_SBT_TAPE_1: finished piece 1 at 22-09-2019_17:54:29

piece handle=ORCL19_2aucdsak_1_1 tag=TAG20190922T175427 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

Finished backup at 22-09-2019_17:54:29




Starting Control File and SPFILE Autobackup at 22-09-2019_17:54:29

piece handle=c-310627084-20190922-05 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 22-09-2019_17:54:30




RMAN>




RMAN> list backup of datafile 5;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2729    Incr 0  40.00K     SBT_TAPE    00:00:02     22-09-2019_17:54:30

        BP Key: 2730   Status: AVAILABLE  Compressed: YES  Tag: TAG20190922T175427

        Handle: VB$_1887643964_2728I   Media:

  List of Datafiles in backup set 2729

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    0  Incr 2320763    22-09-2019_17:54:28              NO    /u01/app/oracle/oradata/ORCL19/simon01.dbf




RMAN>

 

So, as you can see, the tablespace SIMON was created (datafile #5). After that, the table TEST was created and some data loaded. At the end, the first backup of datafile was done and this was indexed by ZDLRA.

 

Basic information for database

 
For this database (database name ORCL19), inside internal tables of ZDLRA, we can see that have the DB_KEY as 2202 and DB_INCKEY as 2203. The DF_KEY for the datafile is 2689. This info is important to identify correctly the backups.

 

[oracle@zdlras1n1 ~]$ sqlplus rasys/change^Me2




SQL*Plus: Release 19.0.0.0.0 - Production on Sun Sep 22 17:53:21 2019

Version 19.3.0.0.0




Copyright (c) 1982, 2019, Oracle.  All rights reserved.




Last Successful login time: Sun Sep 22 2019 17:39:03 +02:00




Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0




SQL> set linesize 250

SQL> select db_key, dbinc_key from rc_database where name = 'ORCL19';




    DB_KEY  DBINC_KEY

---------- ----------

      2202       2203




SQL>

SQL> select df_key, file#, ts#, create_scn, create_time, block_size, blocks from df where dbinc_key = 2203 and file# = 5;




    DF_KEY      FILE#        TS# CREATE_SCN CREATE_TI BLOCK_SIZE     BLOCKS

---------- ---------- ---------- ---------- --------- ---------- ----------

      2689          5          6    2319183 22-SEP-19       8192        128




SQL>

 

Virtual Full Backup and PLANS

After the backup ingests, and the task INDEX_BACKUP finished at ZDLRA, we have one virtual full backup linked with the datafile. This information came from VBDF and PLANS tables:

 

SQL> select vb_key, ckp_scn, vcbp_key, srcbp_key, blocks, chunks_used from vbdf where db_key = 2202 and df_key = 2689 order by vb_key asc;




    VB_KEY    CKP_SCN   VCBP_KEY  SRCBP_KEY     BLOCKS CHUNKS_USED

---------- ---------- ---------- ---------- ---------- -----------

      2728    2320763       2730       2701        128           1




SQL>

SQL> select * from plans where db_key = 2202 and df_key = 2689 order by vb_key asc;




      TYPE     DB_KEY     VB_KEY     DF_KEY    TASK_ID        OLD   BLKSREAD    MAXRANK  NUMCHUNKS   READSIZE  NEEDCHUNK  FREECHUNK

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

         8       2202       2728       2689                               19          1          1




SQL>

 

Above we can see that this virtual full backup with VB_KEY have 128 blocks (128 * 8K = 1M) and have one plan for this backup. This virtual full backup/plan has just 19 blocks (BLKSREAD) and used 1 chunk to store it (NUMCHUNKS).
But this not show how ZDLRA see the backup, the virtual full backup store the information at PLAN_DETAILS table. Reading this table we can see the foundation of virtual full backup:

 

SQL> select * from plans_details where df_key = 2689 order by vb_key asc, blockno asc;




    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

      2689          8       2728          1          0          1          1       8192      24576

      2689          8       2728          1          2          1         17      32788       2167

      2689          8       2728          1 4294967295          1          1      34955        294




SQL>

 

Let’s explain what this means:

 

  1. The datafile block 0 (column BLOCKNO) until block 1 (BLOCKNO+NUNBLKS) are stored at chunk 1.
  2. The datafile block 2 (column BLOCKNO) until block 19 (BLOCKNO+NUNBLKS) are stored at chunk 1
  3. The datafile block 4294967295 (the last block of datafile) is stored at chunks 1.
 
To show this I created the schema below. This represents the first 13 blocks of the virtual full backup and will help to visualize the virtual full backup next. So, the virtual full backup with VB_KEY 2728 can be reconstructed following the PLANS_DETAILS as:

Subsequent backups

If we continue to modify data in this tablespace/datafile and execute new incremental backup level 1 we can see more details. So, first, add more data and do a backup:

 

SQL> BEGIN

  2    FOR i IN 1 .. 300 LOOP

  3      insert into test (c1, c2) values (2, DBMS_RANDOM.STRING('P', 128));

  4

  5      if (MOD(i, 100) = 0) then

  6         commit;

  7      end if;

  8

  9    END LOOP;

 10

 11    commit;

 12  END;

 13  /




PL/SQL procedure successfully completed.




SQL>




RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATAFILE 5;




Starting backup at 22-09-2019_18:34:30

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL19/simon01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 22-09-2019_18:34:31

channel ORA_SBT_TAPE_1: finished piece 1 at 22-09-2019_18:34:32

piece handle=ORCL19_2cucduln_1_1 tag=TAG20190922T183430 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

Finished backup at 22-09-2019_18:34:32




Starting Control File and SPFILE Autobackup at 22-09-2019_18:34:32

piece handle=c-310627084-20190922-06 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 22-09-2019_18:34:36




RMAN>

 

And now inside of ZDLRA we have two virtual full backups and two plans (for PLANS look the number of blocks read – this is normal since I added more data above):

 

SQL> select vb_key, ckp_scn, vcbp_key, srcbp_key, blocks, chunks_used from vbdf where db_key = 2202 and df_key = 2689 order by vb_key asc;




    VB_KEY    CKP_SCN   VCBP_KEY  SRCBP_KEY     BLOCKS CHUNKS_USED

---------- ---------- ---------- ---------- ---------- -----------

      2728    2320763       2730       2701        128           1

      2768    2322525       2770       2735        128           1




SQL>

SQL> select * from plans where db_key = 2202 and df_key = 2689 order by vb_key asc;




      TYPE     DB_KEY     VB_KEY     DF_KEY    TASK_ID        OLD   BLKSREAD    MAXRANK  NUMCHUNKS   READSIZE  NEEDCHUNK  FREECHUNK

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

         8       2202       2728       2689                               19          1          1

         1       2202       2768       2689                               27          1          2




SQL>

 

The virtual full backups have these PLAN_DETAILS:

 

SQL> select * from plans_details where df_key = 2689 order by vb_key asc, blockno asc;




    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

      2689          8       2728          1          0          1          1       8192      24576

      2689          8       2728          1          2          1         17      32788       2167

      2689          8       2728          1 4294967295          1          1      34955        294

      2689          1       2768          1          0       1025          1       8192      24576

      2689          1       2768          1          2       1025          2      32788        252

      2689          1       2768          1          4          1          4      33038        408

      2689          1       2768          1          8       1025         16      33040      45339

      2689          1       2768          1         71          1          3      34703        252

      2689          1       2768          1 4294967295       1025          1      78379        293




9 rows selected.




SQL>

 

Checking in details the last virtual full backup, VB_KEY 2768, we can see interesting things and start to understand how it’s work. So, to “mount” the virtual full backup 2768 we have:
  1. The datafile block 0 (column BLOCKNO) until block 1 (BLOCKNO+NUNBLKS) are stored at chunk 1025.
  2. The datafile block 2 (column BLOCKNO) until block 4 (BLOCKNO+NUNBLKS) are stored at chunk 1025.
  3. The datafile block 4 (column BLOCKNO) until block 8 (BLOCKNO+NUNBLKS) are stored at chunk 1 (and came from previous virtual full backup).
  4. The datafile block 8 (column BLOCKNO) until block 24 (BLOCKNO+NUNBLKS) are stored at chunk 1025.
So, the virtual full backup with VB_KEY 2768 for DF_KEY can be reconstructed following the PLANS_DETAILS. It is the image below represents the Delta Store and actual virtual full backups:

 

Look that ZDLRA started to save space to store the backup, being “smart” and indexing all the blocks to needed to create the virtual full backup. But the important here it is that ZDLRA understood and indexed every datafile block that was inside of backup. And the virtual full backup “does not exist”, it is basically the index for each version of the block.
If I continue to insert/delete/update some of the lines from this table I will possibly change blocks (and same blocks already created) and if I do some subsequent backups I have these plans:
SQL> select vb_key, ckp_scn, vcbp_key, srcbp_key, blocks, chunks_used from vbdf where db_key = 2202 and df_key = 2689 order by vb_key asc;

    VB_KEY    CKP_SCN   VCBP_KEY  SRCBP_KEY     BLOCKS CHUNKS_USED

———- ———- ———- ———- ———- ———–

      2728    2320763       2730       2701        128           1

      2768    2322525       2770       2735        128           1

      2818    2323607       2820       2779        128           1

      2874    2324062       2876       2829        128           1

      2936    2324792       2938       2885        128           1

SQL> select * from plans where db_key = 2202 and df_key = 2689 order by vb_key asc;

      TYPE     DB_KEY     VB_KEY     DF_KEY    TASK_ID        OLD   BLKSREAD    MAXRANK  NUMCHUNKS   READSIZE  NEEDCHUNK  FREECHUNK

———- ———- ———- ———- ———- ———- ———- ———- ———- ———- ———- ———-

         8       2202       2728       2689                               19          1          1

         1       2202       2768       2689                               27          1          2

         1       2202       2818       2689                               27          1          3

         1       2202       2874       2689                               27          1          3

         1       2202       2936       2689                               27          1          4

SQL> select * from plans_details where df_key = 2689 order by vb_key asc, blockno asc;

    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

———- ———- ———- ———- ———- ———- ———- ———- ———-

      2689          8       2728          1          0          1          1       8192      24576

      2689          8       2728          1          2          1         17      32788       2167

      2689          8       2728          1 4294967295          1          1      34955        294

      2689          1       2768          1          0       1025          1       8192      24576

      2689          1       2768          1          2       1025          2      32788        252

      2689          1       2768          1          4          1          4      33038        408

      2689          1       2768          1          8       1025         16      33040      45339

      2689          1       2768          1         71          1          3      34703        252

      2689          1       2768          1 4294967295       1025          1      78379        293

      2689          1       2818          1          0       2049          1       8192      24576

      2689          1       2818          1          2       1025          2      32788        252

      2689          1       2818          1          4          1          4      33038        408

      2689          1       2818          1          8       2049          1      32788        257

      2689          1       2818          1          9       1025          7      33293      36549

      2689          1       2818          1         16       2049          3      33045      17304

      2689          1       2818          1         19       1025          1      76821       1026

      2689          1       2818          1         20       2049          4      50349      29052

      2689          1       2818          1         71          1          3      34703        252

      2689          1       2818          1 4294967295       2049          1      79401        301

      2689          1       2874          1          0       3073          1       8192      24576

      2689          1       2874          1          2       1025          2      32788        252

      2689          1       2874          1          4          1          4      33038        408

      2689          1       2874          1          8       3073          1      32788        262

      2689          1       2874          1          9       1025          2      33293        347

      2689          1       2874          1         11       3073         13      33050      94555

      2689          1       2874          1         71          1          3      34703        252

      2689          1       2874          1 4294967295       3073          1     127605        296

      2689          1       2936          1          0       4097          1       8192      24576

      2689          1       2936          1          2       1025          2      32788        252

      2689          1       2936          1          4          1          4      33038        408

      2689          1       2936          1          8       3073          1      32788        262

      2689          1       2936          1          9       1025          2      33293        347

      2689          1       2936          1         11       3073         13      33050      94555

      2689          1       2936          1         71          1          3      34703        252

      2689          1       2936          1 4294967295       4097          1      32788        278

35 rows selected.

SQL>

As you can see above now I have 5 virtual full backups (2728, 2768,2818,2874, and 2936) stored in 5 chunks (1, 1025, 2049,3073,4097). In this case, if I want need to read the virtual full backup 2936 I read the PLAN_DETAILS and check that:
  1. The datafile block 0 until block 1 are stored at chunk 4097.
  2. The datafile block 2 until block 4 are stored at chunk 1025.
  3. The datafile block 4 until block 8 are stored at chunk 1.
  4. The datafile block 8 are stored at chunk 3073.
  5. The datafile block 9 until block 10 are stored at chunk 1025.
  6. The datafile block 11 until block 24 are stored at chunk 3073.
The image below represents this:
 

Automated Delta Pool Space Management

Since ZDLRA contains a “self-managed”  rman catalog, the backups need are managed automatically. This means that unnecessary data are deleted from time to time to avoid redundancy and to be more space-efficient. For ZDLRA this is called “automated delta pool space management”, specifically the “delta pool optimization“, to optimize the delta pool.
Internally this means that ZDLRA constantly checks the ingest backup and try to optimize the Delta Store. If I continue to change data and to do more backups we can see this in action:

 

SQL> select * from plans_details where df_key = 2689 order by vb_key asc, blockno asc;




    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

      2689          1       2768          1          0       1025          1       8192      24576

      2689          1       2768          1          2       1025          2      32788        252

      2689          1       2768          1          4          1          4      33038        408

      2689          1       2768          1          8       1025         16      33040      45339

      2689          1       2768          1         71          1          3      34703        252

      2689          1       2768          1 4294967295       1025          1      78379        293

      2689          1       2818          1          0       2049          1       8192      24576

      2689          1       2818          1          2       1025          2      32788        252

      2689          1       2818          1          4          1          4      33038        408

      2689          1       2818          1          8       2049          1      32788        257

      2689          1       2818          1          9       1025          7      33293      36549

      2689          1       2818          1         16       2049          3      33045      17304

      2689          1       2818          1         19       1025          1      76821       1026

      2689          1       2818          1         20       2049          4      50349      29052

      2689          1       2818          1         71          1          3      34703        252

      2689          1       2818          1 4294967295       2049          1      79401        301

      2689          1       2874          1          0       3073          1       8192      24576

      2689          1       2874          1          2       1025          2      32788        252

      2689          1       2874          1          4          1          4      33038        408

      2689          1       2874          1          8       3073          1      32788        262

      2689          1       2874          1          9       1025          2      33293        347

      2689          1       2874          1         11       3073         13      33050      94555

      2689          1       2874          1         71          1          3      34703        252

      2689          1       2874          1 4294967295       3073          1     127605        296

      2689          1       2936          1          0       4097          1       8192      24576

      2689          1       2936          1          2       1025          2      32788        252

      2689          1       2936          1          4          1          4      33038        408

      2689          1       2936          1          8       3073          1      32788        262

      2689          1       2936          1          9       1025          2      33293        347

      2689          1       2936          1         11       3073         13      33050      94555

      2689          1       2936          1         71          1          3      34703        252

      2689          1       2936          1 4294967295       4097          1      32788        278

      2689          1       3002          1          0       5121          1       8192      24576

      2689          1       3002          1          2       5121          2      32788        255

      2689          1       3002          1          4          1          4      33038        408

      2689          1       3002          1          8       3073          1      32788        262

      2689          1       3002          1          9       1025          1      33293        128

      2689          1       3002          1         10       5121          1      33043        215

      2689          1       3002          1         11       3073         13      33050      94555

      2689          1       3002          1         24       5121         28      33258       4986

      2689          1       3002          1         52          1          1      34703         84

      2689          1       3002          1         56       5121          8      38244       1168

      2689          1       3002          1         72          1          1      34787         84

      2689          1       3002          1        128       5121        843      39412     125460

      2689          1       3002          1 4294967295       5121          1     164872        301




45 rows selected.




SQL>

 

Here, look the plan for virtual full backup 2768 (that was one of the first and already consolidated) changed. Look that for block 0 the chunk that store changed from 1 to 1025. And this occurred for other blocks (blocks 8-13 as an example), look the plans right now:

 

 

The red arrows represent the change that we can see in the PLAN_DETAILS table for VB_KEY 2728. The back arrows represent the blocks needed (from previous backups) if I want to read the virtual full backup 3002.
At the end if I do another backup we can see more evolution for space management:

 

SQL> select * from plans_details where df_key = 2689 order by vb_key asc, blockno asc;




    DF_KEY       TYPE     VB_KEY    BLKRANK    BLOCKNO    CHUNKNO    NUMBLKS    COFFSET   NUMBYTES

---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------

      2689          1       2818          1          0       2049          1       8192      24576

      2689          1       2818          1          2       1025          2      32788        252

      2689          1       2818          1          4          1          4      33038        408

      2689          1       2818          1          8       2049          1      32788        257

      2689          1       2818          1          9       1025          7      33293      36549

      2689          1       2818          1         16       2049          3      33045      17304

      2689          1       2818          1         19       1025          1      76821       1026

      2689          1       2818          1         20       2049          4      50349      29052

      2689          1       2818          1         71          1          3      34703        252

      2689          1       2818          1 4294967295       2049          1      79401        301

      2689          1       2874          1          0       3073          1       8192      24576

      2689          1       2874          1          2       1025          2      32788        252

      2689          1       2874          1          4          1          4      33038        408

      2689          1       2874          1          8       3073          1      32788        262

      2689          1       2874          1          9       1025          2      33293        347

      2689          1       2874          1         11       3073         13      33050      94555

      2689          1       2874          1         71          1          3      34703        252

      2689          1       2874          1 4294967295       3073          1     127605        296

      2689          1       2936          1          0       4097          1       8192      24576

      2689          1       2936          1          2       1025          2      32788        252

      2689          1       2936          1          4          1          4      33038        408

      2689          1       2936          1          8       3073          1      32788        262

      2689          1       2936          1          9       1025          2      33293        347

      2689          1       2936          1         11       3073         13      33050      94555

      2689          1       2936          1         71          1          3      34703        252

      2689          1       2936          1 4294967295       4097          1      32788        278

      2689          1       3002          1          0       5121          1       8192      24576

      2689          1       3002          1          2       5121          2      32788        255

      2689          1       3002          1          4          1          4      33038        408

      2689          1       3002          1          8       3073          1      32788        262

      2689          1       3002          1          9       1025          1      33293        128

      2689          1       3002          1         10       5121          1      33043        215

      2689          1       3002          1         11       3073         13      33050      94555

      2689          1       3002          1         24       5121         28      33258       4986

      2689          1       3002          1         52          1          1      34703         84

      2689          1       3002          1         56       5121          8      38244       1168

      2689          1       3002          1         72          1          1      34787         84

      2689          1       3002          1        128       5121        843      39412     125460

      2689          1       3002          1 4294967295       5121          1     164872        301

      2689          1       3074          1          0       6145          1       8192      24576

      2689          1       3074          1          2       5121          2      32788        255

      2689          1       3074          1          4          1          4      33038        408

      2689          1       3074          1          8       3073          1      32788        262

      2689          1       3074          1          9       1025          1      33293        128

      2689          1       3074          1         10       5121          1      33043        215

      2689          1       3074          1         11       3073         13      33050      94555

      2689          1       3074          1         24       5121         28      33258       4986

      2689          1       3074          1         52          1          1      34703         84

      2689          1       3074          1         56       5121          8      38244       1168

      2689          1       3074          1         72          1          1      34787         84

      2689          1       3074          1        128       5121        843      39412     125460

      2689          1       3074          1 4294967295       6145          1      32788        281




52 rows selected.




SQL>
 
The image that represents the actual story now it is:

 

 

If you check the table and the image you can see that ZDLRA check block a block and now the block #8 for VB_KEY 2728 came from chunk 2049 (done by virtual full backup 2818). If you check with the previous report you can see that backup #8 was split from previous chunk and this information was added for VB_KEY 2728.
At the end, this means that ZDLRA optimized the delta store too, maybe, allow that chunk 1 can be deleted in the future. Probably during the INDEX_BACKUP/PLAN_DF tasks the ZDLRA checked that the block 1 it is the same that exists in others chunks and marked the existing version in chunk number 1 as obsolete.

 

Virtual Full Backup

 

This is how the virtual full backup works. Every datafile block that enters inside of ZDLRA from rman backup it is indexed and stored to create the virtual full backup. As you saw above, the idea is creating the representation for every virtual full backup of datafile (VBDF table) linking with one plan (PLANS and PLAN_DETAILS tables).
Going further, you can see that does not exist 1 to 1 relation between backup that was ingested and the virtual full backup. It is just a matrix of pointers for blocks inside chunks. This is the reason that ZDLRA it is different from other backup appliances, it can analyze block a block and index it efficiently.

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

 

ZDLRA Internals, INDEX_BACKUP task in details
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA Internals, INDEX_BACKUP task in details

For ZDLRA, the task type INDEX_BACKUP it is important (if it is not the most) because it is responsible to create the virtual full backup. This task runs for every backup that you ingest at ZDLRA and here, I will show with more details what occurs at ZDLRA: internals steps, phases, and tables involved.
I recommend that you check my previous post about ZDLRA: ZDLRA Internals, Tables and Storage, ZDLRA, Virtual Full Backup and Incremental Forever, and Understanding ZDLRA. They provide a good base to understand some aspects of ZDLRA architecture and features.
As you saw in my previous post, ZDLRA opens every backup that you sent and read every block of it to generate one new virtual full backup. And this backup is validated block a block (physically and logically) against corruption. It differs from a snapshot because it is content-aware (in this case it is proprietary Oracle datafile blocks inside another proprietary Oracle rman block) and Oracle it is the only that can do this guaranteeing that result is valid.
Backup
This generation is done thought INDEX_BACKUP task and occurs for every backup (level 0 or 1) that enter in the system. For this post, I already have a full backup level 0 inside ZDLRA and I made one new incremental level 1:
RMAN> run{

2> backup device type sbt cumulative incremental level 1 filesperset 1 datafile 29;

3> }

Starting backup at 2019-08-16_15-01-11

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=406 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRA) SID=903C95C93085DC14E0531E43B20AB30F

allocated channel: ORA_SBT_TAPE_2

channel ORA_SBT_TAPE_2: SID=451 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_2: RA Library (ZDLRA) SID=903C95D1C201DC19E0531E43B20A4C44

allocated channel: ORA_SBT_TAPE_3

channel ORA_SBT_TAPE_3: SID=502 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_3: RA Library (ZDLRA) SID=903C95E1BFD9DC38E0531E43B20A0038

allocated channel: ORA_SBT_TAPE_4

channel ORA_SBT_TAPE_4: SID=651 instance=SIMON1 device type=SBT_TAPE

channel ORA_SBT_TAPE_4: RA Library (ZDLRA) SID=903C95EAC1ACDC48E0531E43B20AC79D

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00029 name=+DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

channel ORA_SBT_TAPE_1: starting piece 1 at 2019-08-16_15-01-25

channel ORA_SBT_TAPE_1: finished piece 1 at 2019-08-16_15-03-10

piece handle=SIMON_s0u9c0a5_1_1 tag=TAG20190816T150116 comment=API Version 2.0,MMS Version 3.17.1.26

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:01:45

Finished backup at 2019-08-16_15-03-10

Starting Control File and SPFILE Autobackup at 2019-08-16_15-03-11

piece handle=c-4165931009-20190816-06 comment=API Version 2.0,MMS Version 3.17.1.26

Finished Control File and SPFILE Autobackup at 2019-08-16_15-03-18

RMAN>
As you can see, it is a normal backup. And the details for this backup:
RMAN> list backupset 52141830;

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52141830 Incr 1  30.75M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52141831   Status: AVAILABLE  Compressed: NO  Tag: TAG20190816T150116

        Handle: SIMON_s0u9c0a5_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 52141830

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

RMAN> list backup tag = 'TAG20190816T150116';

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52141830 Incr 1  30.75M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52141831   Status: AVAILABLE  Compressed: NO  Tag: TAG20190816T150116

        Handle: SIMON_s0u9c0a5_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 52141830

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>
Here it is important to remember the TAG, backupset(BS)/backup piece (BP) numbers, and the SCN for the backup made.
INDEX_BACKUP
The INDEX_BACKUP appears as a single task inside ZDLRA, but have two major phases: fixup_unordered and q_restore_fast. And each one impacts differently inside the rman catalog and in the internal tables.
So, after the backup finish, inside of ZDLRA we can see the index task running:
SQL> l

  1  SELECT s.ba_session_id, s.instance_id, s.sid, s.serial#, s.job_name

  2         , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state, rt.waiting_on

  3         , rt.elapsed_seconds

  4         , gs.module, gs.sql_id, gs.action

  5         , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

  6  FROM sessions s

  7  JOIN ra_task rt

  8  ON rt.task_id = s.current_task

  9  AND rt.task_type = 'INDEX_BACKUP'

 10  JOIN gv$session gs

 11  ON gs.inst_id = s.instance_id

 12  AND gs.sid = s.sid

 13  AND gs.serial# = s.serial#

 14* ORDER BY rt.LAST_EXECUTE_TIME DESC

SQL> /

BA_SESSION_ID INSTANCE_ID    SID    SERIAL# JOB_NAME              TASK_ID     DB_KEY DB_UNIQUE_NAME  TASK_TYPE       STATE    WAITING_ON ELAPSED_SECONDS MODULE           SQL_ID        ACTION            BP_KEY     BS_KEY     DF_KEY     VB_KEY

------------- ----------- ------ ---------- ------------------ ---------- ---------- --------------- --------------- -------- ---------- --------------- ---------------- ------------- ------------- ---------- ---------- ---------- ----------

     56215535           2   4228      30075 RA$_EXEC_56216288    56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING                  345.305895 fixup_unordered  1q0zr86n25pvu scan 6% done    52141831

SQL>
With this SQL you can check all the INDEX_BACKUP tasks running at ZDLRA. And the SQL use the tables and report:
  • RA_TASK: All tasks running inside ZDLRA.
  • SESSIONS: Sessions running inside ZDLRA, used to get the database session running the task
  • GV$SESSION: Databases sessions, used to get the sql_id and others info
  • BP_KEY: Identify the backup piece key that it is indexing. It is the start point to identify everything here.
Above you can see that task 56242962 is running, processing the BP_KEY (backup piece key) 52141831, running the module fixup_unordered, processed 6% of this step, and running SQL_ID 1q0zr86n25pvu.

 

INDEX_BACKUP, fixup_unordered
As showed, the index task is running the fixup_unordered module, and thus represents the first phase where ZDLRA it is checking all the blocks that are necessary to build the virtual full backup. Based on the report from the query above we can see that BP_KEY processed it is 52141831 and we can get more info from backup piece rman table:
SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024 from bp where bp_key = 52141831;

    BP_KEY     BS_KEY TAG                              HANDLE               BYTES/1024/1024/1024

---------- ---------- -------------------------------- -------------------- --------------------

  52141831   52141830 TAG20190816T150116               SIMON_s0u9c0a5_1_1             .030029297

SQL>

SQL> select bs_key, INCREMENTAL_LEVEL, HANDLE from rc_backup_piece where bp_key = 52141831;

    BS_KEY INCREMENTAL_LEVEL HANDLE

---------- ----------------- --------------------

  52141830                 1 SIMON_s0u9c0a5_1_1

SQL>
This tells us that the task it is processing the backup that we made before (look the result of column BP_KEY from SQL in the first topic). The BP, and rc_backup_piece are the tables from rman catalog that store info about the backup pieces, check the handle and tag to compare from rman list output.
Going further we can check the VBDF (virtual backup data file table – store the virtual full backups information) table and verify more details:
SQL> SELECT vb_key, ckp_id, df_key, db_key, blocks, vcbp_key, srcbp_key, file#, ckp_scn FROM VBDF WHERE srcbp_key = 52141831;

    VB_KEY         CKP_ID     DF_KEY     DB_KEY     BLOCKS   VCBP_KEY  SRCBP_KEY      FILE#        CKP_SCN

---------- -------------- ---------- ---------- ---------- ---------- ---------- ---------- --------------

  52141864  4932560891039     752220     752186 1655177216              52141831         29  4932560891039

SQL>
At VBDF table we can discover:
  • VB_KEY: Virtual backup key.
  • CKP_ID: Checkpoint and SCN for this virtual backup.
  • BLOCKS: Number of the blocks for this virtual backup.
  • VCBP_KEY: Is null at fixup_unordered because the virtual backup piece (VCBP) was not yet generated (it is doing by fixup_unordered). This will change in the future and I will show more details.
  • SRCBP_KEY: The source/original backup piece used to create this virtual full backup.
  •  
So, we can check that the backu is in the middle of the index the process since the VCBP_KEY is not yet available. But besides that, we can validate/check more details about the backups.
Check that does not exist a backup piece associated with this virtual backup key:
SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where vb_key = 52141864;

no rows selected

SQL>
And that exists just one backup of datafile for this SCN and FILE#:
SQL> SELECT bdf_key, bs_key, file#, ckp_scn, incr_level from bdf where CKP_SCN = 4932560891039 and file# = 29;

   BDF_KEY     BS_KEY      FILE#        CKP_SCN INCR_LEVEL

---------- ---------- ---------- -------------- ----------

  52141832   52141830         29  4932560891039          1

SQL>

SQL> SELECT db_key, bdf_key, file#, CHECKPOINT_CHANGE#, incremental_level from rc_backup_datafile where bs_key IN(52141830);

    DB_KEY    BDF_KEY      FILE# CHECKPOINT_CHANGE# INCREMENTAL_LEVEL

---------- ---------- ---------- ------------------ -----------------

    752186   52141832         29      4932560891039                 1

SQL> SELECT db_key, bdf_key, file#, CHECKPOINT_CHANGE#, incremental_level from rc_backup_datafile where CHECKPOINT_CHANGE# = 4932560891039;

    DB_KEY    BDF_KEY      FILE# CHECKPOINT_CHANGE# INCREMENTAL_LEVEL

---------- ---------- ---------- ------------------ -----------------

    752186   52141832         29      4932560891039                 1

SQL>

Digging more, we can see that are zero plans for this virtual backup (will be zero until fixup_unordered finish – more detail further):
SQL> select count(*) from plans where VB_KEY = 52141864;

  COUNT(*)

----------

         0

SQL>
During this phase of INDEX_BACKUP, the major SQL_ID is 1q0zr86n25pvu:
SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = '1q0zr86n25pvu';

SQL_FULLTEXT

----------------------------------------------------------------------------------------------------

SELECT * FROM (SELECT /*+

                     QB_NAME(c)

                     INDEX_RS_ASC(@c b@c)

                     NO_INDEX_FFS(@c b@c)

                     NO_INDEX_SS(@c b@c)

                     OPT_PARAM('optimizer_dynamic_sampling' 0)

                     OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

                     OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

                     OPT_PARAM('_optimizer_use_feedback' 'false')

                    */ BLOCKNO, SCN, CKP_ID, ENDBLK, CHUNKNO, COFFSET FROM BLOCKS B WHERE DF_KEY = :

B4 AND BLOCKNO BETWEEN :B3 AND :B2 AND SCN < :B1 ORDER BY BLOCKNO, SCN, CKP_ID ) WHERE ROWNUM < :B5

SQL>


And the bind variables in this case are:
SQL> l

  1  SELECT s.ba_session_id, s.instance_id

  2  , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state

  3  , gs.module, gs.sql_id, gs.action

  4  , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

  5  , sbc.name, sbc.value_string

  6  FROM sessions s

  7  JOIN ra_task rt

  8  ON rt.task_id = s.current_task

  9  AND RT.TASK_ID = 56242962

 10  JOIN gv$session gs

 11  ON gs.inst_id = s.instance_id

 12  AND gs.sid = s.sid

 13  AND gs.serial# = s.serial#

 14  join gv$sql_bind_capture sbc

 15  on sbc.inst_id = gs.inst_id

 16  and sbc.sql_id = gs.sql_id

 17* ORDER BY rt.LAST_EXECUTE_TIME DESC

SQL>

SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE           MODULE           SQL_ID        ACTION              BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- -------------------- --------------- --------------- ---------------- ------------- --------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B4   752220

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B3   302875642

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B2   1655177216

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B1   4932554641877

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 21% done     52141831                                  :B5   1048576

SQL>
Explaining a little. So, based on the SQL text and variables we can see that is retrieving from BLOCKS table all the blocks for the DF_KEY 752220 (you can get it in VBDF), that are between 302875642 (actual block of processing) and 1655177216 (max number of blocks for the ingested backup), and have SCN bellow 4932554641877 (from the ingested backup) and process this in package of 1048576 blocks. One information here it is that the number 302875642 will vary while it is running this phase.
So, the idea here for ZDLRA index task is processing get/check all the blocks that have SCN below of the SCN from ingested backup. And since the INDEX_BACKUP result creates one virtual full backup, all the blocks for this datafile need to be processed.
If you check this query time to time, you will see the increase of scan % increasing until the :B3 and :B4 are equal (compare with the previous execution):
SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE           MODULE           SQL_ID        ACTION             BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- -------------------- --------------- --------------- ---------------- ------------- -------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B4   752220

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B3   1641637658

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B2   1655177216

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B1   4932554641877

     56215535           2   56242962     752186 SIMON_X6S            INDEX_BACKUP    RUNNING         fixup_unordered  1q0zr86n25pvu scan 99% done    52141831                                  :B5   1048576

SQL> select rt.elapsed_seconds from ra_task rt where task_id = 56242962;

ELAPSED_SECONDS

---------------

     6618.53526

SQL>
After this hit 100% the next phase of the INDEX_BAKUP (q_restore_fast) will start. The phase fixup_unordered depends on the size of your datafile and not from the ingested backup. In this example, the backup has just 30MB, but the datafile has more than 16TB, so, the virtual full will have the same size and this takes a lot of time reading all the blocks. For you ZDLRA the DF_KEY, BS_KEY, and others will be different. Just remember that BP_KEY for the index task it is the start point for the analyses.
INDEX_BACKUP, q_restore_fast
After the fixup_unordered reach 100% of the scan, the phase q_restore_fast starts and this will create the virtual backup itself. But before let’s see what’s happened at rman catalog after the previous phase finished:
 
##########################################################

This is list took right after the backup finished:

RMAN> list backup tag = 'TAG20190816T150116';

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52141830 Incr 1  30.75M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52141831   Status: AVAILABLE  Compressed: NO  Tag: TAG20190816T150116

        Handle: SIMON_s0u9c0a5_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 52141830

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>

##########################################################

And now after the fixup_unordered finish:

RMAN> list backup tag = 'TAG20190816T150116';

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52146235 Incr 1  28.00M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52146236   Status: AVAILABLE  Compressed: YES  Tag: TAG20190816T150116

        Handle: VB$_90959062_52141864I   Media:

  List of Datafiles in backup set 52146235

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>
 
As you can see, the backupset (52141830 before, and 52146235 now) and backup piece changed (the handle changed too). But look that TAG and SCN remains the same. This means that a new backup was created because ZDLRA scanned the original backup. But as you can see, the virtual full was not created yet (there is any level 0 available in the list). If you see this in your ZDLRA, that means that INDEX_BACKUP task is in process.
If we continue to check more details and use the same query than before, we can see that backup changed the BS_KEY and BP_KEY in the internal tables (look that old keys disappears):
SQL> select bs_key, INCREMENTAL_LEVEL, HANDLE from rc_backup_piece where bp_key = 52141831;

no rows selected

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where bp_key = 52141831;

no rows selected

SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024 from bp where tag = 'TAG20190816T150116';

    BP_KEY     BS_KEY TAG                 HANDLE                  BYTES/1024/1024/1024

---------- ---------- ------------------- ----------------------- --------------------

  52146236   52146235 TAG20190816T150116  VB$_90959062_52141864I             .02734375

SQL>
And now, checking in the VBDF table we can see that info at VCBP_KEY column appears
SQL> SELECT vb_key, ckp_id, df_key, db_key, blocks, vcbp_key, srcbp_key, file#, ckp_scn FROM VBDF WHERE srcbp_key = 52141831;

    VB_KEY         CKP_ID     DF_KEY     DB_KEY     BLOCKS   VCBP_KEY  SRCBP_KEY      FILE#        CKP_SCN

---------- -------------- ---------- ---------- ---------- ---------- ---------- ---------- --------------

  52141864  4932560891039     752220     752186 1655177216   52146236   52141831         29  4932560891039

SQL>
And if you search now at backup piece table you can see that exists one listed for this virtual backup key (check in the topic before that this info does not exist):
SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where vb_key = 52141864;

    BP_KEY     BS_KEY TAG                 HANDLE                  BYTES/1024/1024/1024     VB_KEY

---------- ---------- ------------------- ----------------------- -------------------- ----------

  52146236   52146235 TAG20190816T150116  VB$_90959062_52141864I             .02734375   52141864

SQL>

And check that exists just one backup of this datafile at this SCN:

SQL> SELECT bdf_key, bs_key, file#, ckp_scn, incr_level from bdf where CKP_SCN = 4932560891039 and file# = 29;

   BDF_KEY     BS_KEY      FILE#        CKP_SCN INCR_LEVEL

---------- ---------- ---------- -------------- ----------

  52146237   52146235         29  4932560891039          1

SQL>
 
In this phase, you can see that the details for index and virtual full backup starts to appear internally. If you check, now you have plans for this virtual full backup:
SQL> select count(*) from plans where VB_KEY = 52141864;

  COUNT(*)

----------

         1

SQL>

SQL> select count(*), to_char(sysdate, 'HH24:MI') as time from plans_details where VB_KEY = 52141864;

  COUNT(*) TIME

---------- -----

   2294010 18:25

SQL> select count(*), to_char(sysdate, 'HH24:MI') as time from plans_details where VB_KEY = 52141864;

  COUNT(*) TIME

---------- -----

   5773720 18:43

SQL>
As you can see above, the plans_details for this virtual backup is increasing from time to time. This means that the q_restore_fast is running and it is reading blocks to create the virtual full backup/index (and inserting at plan_details).
We can figure out this, while the INDEX_BACKUP task is running, checking the sql_id for the session. While the task is running, we have three major sql’s (I used the same query than before – where we checked the bind for the session/task):
SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME  TASK_TYPE       STATE    MODULE          SQL_ID        ACTION                   BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- --------------- --------------- -------- --------------- ------------- -------------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B1

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B12

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B2

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B1

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B2

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B3

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B4

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B1   752220

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B3   1

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B4   1051071

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B11  0

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B10  4931849117847

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B9   4931848304742

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B8   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B7   752187

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B6   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  9ma189ab8x9c7 plan 0% done           52141831                                  :B5   4932554641877

17 rows selected.                                                                                       

SQL>                                                                                                    

SQL> /                                                                                                 

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME  TASK_TYPE       STATE    MODULE          SQL_ID        ACTION                   BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- --------------- --------------- -------- --------------- ------------- -------------------- ---------- ---------- ---------- ---------- ----- ---------------

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B1   752220

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B3   403380104

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B4   404440007

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B11  0

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B10  4931849117847

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B9   4931848304742

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B8   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B7   752187

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B6   4932554641877

     56215535           2   56242962     752186 SIMON_X6S       INDEX_BACKUP    RUNNING  q_restore_fast  brt6uuhzacdnu plan 54% done          52141831                                  :B5   4932554641877

10 rows selected.

SQL>

SQL> /

BA_SESSION_ID INSTANCE_ID    TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE      MODULE                         SQL_ID        ACTION                   BP_KEY     BS_KEY     DF_KEY     VB_KEY NAME  VALUE_STRING

------------- ----------- ---------- ---------- -------------------- --------------- ---------- ------------------------------ ------------- -------------------- ---------- ---------- ---------- ---------- ----- --------------------

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B1

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B3   752220

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B2   4389973931270

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B1   4932554641877

     56215535           2   56242962     477602 SIMON_X6S            INDEX_BACKUP    RUNNING    q_restore_fast                 8t81c0j1w9jqu plan 89% done          41032043                                  :B4   1048576

SQL>

And the SQL text for them:

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = '9ma189ab8x9c7';

SQL_FULLTEXT

--------------------------------------------------------------------------------------------------------------------------------

INSERT /*+

             QB_NAME(q_restore_fast)

             OPT_PARAM('optimizer_dynamic_sampling' 0)

             OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

             OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

             OPT_PARAM('_optimizer_use_feedback' 'false')

           */ INTO PLANS_DETAILS (DF_KEY, TYPE, VB_KEY, BLKRANK, BLOCKNO, CHUNKNO, NUMBLKS, COFFSET, NUMBYTES) SELECT :B1 , :B12

, :B2 , 1 BLKRANK, BLOCKNO, CHUNKNO, NUMBLKS, COFFSET, NUMBYTES FROM TABLE( DBMS_RA_POOL.COLLAPSE(:B1 , :B2 , :B3 , :B4 ,

 CURSOR( SELECT /*+

                     QB_NAME(q_rfast_c)

                     INDEX_RS_ASC(@q_rfast_c b@q_rfast_c)

                     NO_INDEX_FFS(@q_rfast_c b@q_rfast_c)

                     NO_INDEX_SS(@q_rfast_c b@q_rfast_c)

                     OPT_PARAM('optimizer_dynamic_sampling' 0)

                     OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

                     OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

                     OPT_PARAM('_optimizer_use_feedback' 'false')

                   */ CKP_ID, BLOCKNO, SCN, CHUNKNO, USED, COFFSET, ENDBLK FROM BLOCKS B WHERE DF_KEY = :B1 AND BLOCKNO BETWEEN

:B3 AND :B4 AND SCN BETWEEN :B11 AND :B10 AND CKP_ID >= :B9 AND (CKP_ID <= :B8 OR DBINC_KEY <> :B7 ) AND (SCN < :B6 OR CKP_ID =

:B5 ) ORDER BY BLOCKNO, SCN, CKP_ID, CHUNKNO ) ) )

SQL>

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = 'brt6uuhzacdnu';

SQL_FULLTEXT

-------------------------------------------------------------------------------------------------------------------------------

SELECT /*+ QB_NAME ("Q_RFAST_C") NO_INDEX_SS ("B") NO_INDEX_FFS ("B") INDEX_RS_ASC ("B") */ "B"."CKP_ID" "CKP_ID","B"."BLOCKNO"

"BLOCKNO","B"."SCN" "SCN","B"."CHUNKNO" "CHUNKNO","B"."USED" "USED","B"."COFFSET" "COFFSET","B"."ENDBLK" "ENDBLK" FROM "BLOCKS"

"B" WHERE "B"."DF_KEY"=:B1 AND "B"."BLOCKNO">=:B3 AND "B"."BLOCKNO"<=:B4 AND "B"."SCN">=:B11 AND "B"."SCN"<=:B10 AND "B"."CKP_ID

">=:B9 AND ("B"."CKP_ID"<=:B8 OR "B"."DBINC_KEY"<>:B7) AND ("B"."SCN"<:B6 OR "B"."CKP_ID"=:B5) ORDER BY "B"."BLOCKNO","B"."SCN",

"B"."CKP_ID","B"."CHUNKNO"

SQL>

SQL> select sql_fulltext from gv$sqlarea where inst_id = 2 and sql_id = '8t81c0j1w9jqu';

SQL_FULLTEXT

--------------------------------------------------------------------------------------------------------------------------------

SELECT NVL(MAX(BLOCKNO), :B1 ), COUNT(*) FROM (SELECT /*+

                 QB_NAME(nbr)

                 INDEX_RS_ASC(@nbr b@nbr)

                 NO_INDEX_FFS(@nbr b@nbr)

                 NO_INDEX_SS(@nbr b@nbr)

                 OPT_PARAM('optimizer_dynamic_sampling' 0)

                 OPT_PARAM('_optimizer_adaptive_cursor_sharing' 'false')

                 OPT_PARAM('_optimizer_extended_cursor_sharing_rel' 'none')

                 OPT_PARAM('_optimizer_use_feedback' 'false')

               */ BLOCKNO FROM BLOCKS B WHERE DF_KEY = :B3 AND BLOCKNO BETWEEN :B2 AND :B1 ORDER BY BLOCKNO) WHERE ROWNUM <= :B4

SQL>
As you can see, the sql_id 9ma189ab8x9c7 is responsible to insert into PLANS_DETAILS the blocks that are needed to create this virtual full backup. This is based in SCN, and means that ZDLRA create the index for all blocks that are below the SCN from the backup that was ingested originally. The sql_id brt6uuhzacdnu returns all the blocks that are needed (based on the SCN), and it is the same for sql_id 8t81c0j1w9jqu that verify if we have a block to index. These sql’s are the base for the q_restore_fast and they alternate itself until we reach 100%.
The idea for ZDLRA in this phase is to create the index that will be the base for the virtual full backup. And “create the index” means all the blocks that are needed to create the plan for this virtual full backup key.
INDEX_BACKUP, 100%
So, after the INDEX_BACKUP finish at ZDLRA, we have this at rman catalog:
RMAN> list backup of datafile 29 tag = TAG20190816T150116;

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52146235 Incr 1  28.00M     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52146236   Status: AVAILABLE  Compressed: YES  Tag: TAG20190816T150116

        Handle: VB$_90959062_52141864I   Media:

  List of Datafiles in backup set 52146235

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   1  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

52152458 Incr 0  12.67T     SBT_TAPE    00:01:44     2019-08-16_15-03-09

        BP Key: 52152459   Status: AVAILABLE  Compressed: YES  Tag: TAG20190816T150116

        Handle: VB$_90959062_52141864_29   Media:

  List of Datafiles in backup set 52152458

  File LV Type Ckp SCN    Ckp Time            Name

  ---- -- ---- ---------- ------------------- ----

  29   0  Incr 4932560891039 2019-08-16_15-01-25 +DATAC1/simon_x6s/datafile/simontsthistoric1_tbs.399.938192975

RMAN>
Look that for the same TAG we have now two backups, and one it is the virtual full backup. Check that SCN and TAG for both are the same. And if we check internally, at rman catalog view, about the backup pieces linked to the virtual backup key that was created by the index task we can see:
SQL> select bp_key, bs_key, tag, handle, bytes/1024/1024/1024, vb_key from bp where vb_key = 52141864;

    BP_KEY     BS_KEY TAG                 HANDLE                    BYTES/1024/1024/1024     VB_KEY

---------- ---------- ------------------- ------------------------- -------------------- ----------

  52152459   52152458 TAG20190816T150116  VB$_90959062_52141864_29            12975.3698   52141864

  52146236   52146235 TAG20190816T150116  VB$_90959062_52141864I               .02734375   52141864

SQL>
Look above that now we have two backups linked to the same virtual full backup (compare with the same SQL from the previous phases). The backup set key 52152458 is the virtual full backup and was generated during the phase q_restore_fast. The backup set key 52146235 was generated during the phase fixup_unordered.
And there is now, two backups for this datafile:
SQL> SELECT bdf_key, bs_key, file#, ckp_scn, incr_level from bdf where CKP_SCN = 4932560891039 and file# = 29;

   BDF_KEY     BS_KEY      FILE#         CKP_SCN INCR_LEVEL

---------- ---------- ---------- --------------- ----------

  52146237   52146235         29   4932560891039          1

  52152460   52152458         29   4932560891039          0

SQL>
Just to show that the task took a lot of time to finish:
SQL> select rt.elapsed_seconds from ra_task rt where task_id = 56242962;

ELAPSED_SECONDS

---------------

     14563.7986

SQL>

SQL> select count(*) from plans_details where VB_KEY = 52141864;

  COUNT(*)

----------

  74505579

SQL>
INDEX_BACKUP, The internals
As you can see above, the INDEX_BACKUP task it is responsible to generate the virtual full backup and “fix” the rman catalog views to represent the new backups. This is done in two major phases (fixup_unordered and q_restore_fast) to read the backup that was ingested at ZDLRA and index all the blocks. The other phases are process_allfiles and plan_blocks, but they are fast to execute.
About the blocks, it is important to hint that it is completely based in SCN/CKP number. The index creation will search for all blocks that are bellow of the SCN of ingested backup. If you check the details, the backup file does not exist, by the way, it is just a huge list of the blocks need to this SCN (information stored at plans_details table).
 
The procedures, SQL, steps, phases that I showed before will have different values in your case. The DF_KEY, BS_KEY, and others will be different. But I think that you can understand the logic of how ZDLRA index the backup and internally generate the virtual full backup. I know that it is not easy to read all the numbers and link between sql’s and tables, but the logic it is simple: verify all the blocks that belong to datafile and create one plan that points to every block needed by the virtual full backup for one scn.
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


ZDLRA, Virtual Full Backup and Incremental Forever
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA, Virtual Full Backup and Incremental Forever

One of the most knows features of ZDLRA is the virtual full backup, basically incremental forever strategy. But what this means in real life? In this post, I will show some details about that and how interesting they are, check what it is Virtual Full Backup and Incremental Forever strategy for ZDLRA.
This post is based on my previous one where I showed all the steps to configure the VPC and enroll database at ZDLRA. 

 

Virtual Full Backup

 

A virtual full backup appears as an incremental level 0 backup in the recovery catalog. From the user’s perspective, a virtual full backup is indistinguishable from a non-virtual full backup. Using virtual backups, Recovery Appliance provides the protection of frequent level 0 backups with only the cost of frequent level 1 backups.

 

 

 

This definition (and image) are in the Zero Data Loss Recovery Appliance Administrator’s Guide and I think that represents the essence of the virtual full backup. ZDLRA receive the incremental level 1 backup, index it, and generate a level 0 to you that it is indistinguishable from a normal backup level 0.
As you can see, virtual full are deeply linked with incremental forever strategy (that it is just executing incremental backups). And based in these incremental backups you receive a full backup available for usage.
To start, I open RMAN and connect in the rman catalog (ZDLRA database, using the VPC configured) and execute a level 0 backup:

 

RMAN> RUN {

2> BACKUP INCREMENTAL LEVEL 0 DEVICE TYPE SBT FILESPERSET 1 DATABASE TAG 'BKP-DB-INC0';

3> SQL 'ALTER SYSTEM ARCHIVE LOG CURRENT';

4> BACKUP DEVICE TYPE SBT FILESPERSET 100 FORMAT '%U' ARCHIVELOG ALL NOT BACKED UP TAG 'BKP-ARCH';

5> }




Starting backup at 15-08-2019_00:44:36

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=100 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRA) SID=901C8083732A1691E053010310AC46B7

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00001 name=/u01/app/oracle/oradata/ORCL18/system01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_00:44:38

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_00:47:23

piece handle=ORCL18_0tu97pnm_1_1 tag=BKP-DB-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:02:45

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00003 name=/u01/app/oracle/oradata/ORCL18/sysaux01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_00:47:24

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_00:48:59

piece handle=ORCL18_0uu97pss_1_1 tag=BKP-DB-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:01:35

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00004 name=/u01/app/oracle/oradata/ORCL18/undotbs01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_00:48:59

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_00:49:02

piece handle=ORCL18_0vu97pvr_1_1 tag=BKP-DB-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00007 name=/u01/app/oracle/oradata/ORCL18/users01.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_00:49:02

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_00:49:05

piece handle=ORCL18_10u97pvu_1_1 tag=BKP-DB-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

Finished backup at 15-08-2019_00:49:05




Starting Control File and SPFILE Autobackup at 15-08-2019_00:49:05

piece handle=c-3914023082-20190815-02 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 15-08-2019_00:49:22




sql statement: ALTER SYSTEM ARCHIVE LOG CURRENT




Starting backup at 15-08-2019_00:49:28

current log archived

using channel ORA_SBT_TAPE_1

skipping archived logs of thread 1 from sequence 4 to 9; already backed up

channel ORA_SBT_TAPE_1: starting archived log backup set

channel ORA_SBT_TAPE_1: specifying archived log(s) in backup set

input archived log thread=1 sequence=10 RECID=7 STAMP=1016326166

input archived log thread=1 sequence=11 RECID=8 STAMP=1016326170

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_00:49:32

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_00:49:35

piece handle=12u97q0s_1_1 tag=BKP-ARCH comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

Finished backup at 15-08-2019_00:49:35




Starting Control File and SPFILE Autobackup at 15-08-2019_00:49:35

piece handle=c-3914023082-20190815-03 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 15-08-2019_00:49:51




RMAN>

RMAN> list backup of datafile 1;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2222    Incr 0  748.50M    SBT_TAPE    00:02:32     15-08-2019_00:47:10

        BP Key: 2223   Status: AVAILABLE  Compressed: NO  Tag: BKP-DB-INC0

        Handle: ORCL18_0tu97pnm_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 2222

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1614233    15-08-2019_00:44:39              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




RMAN>

 

As you can see a simple incremental level 0 backup, archive log generation, and backup. ZDLRA after receive this backup start to index it to generate the base level 0. If you go to ZDLRA database you can see:

 

SQL> select TASK_ID, TASK_TYPE, STATE, WAITING_ON, DB_KEY, DB_UNIQUE_NAME, CREATION_TIME, ERROR_COUNT, DF_KEY, BS_KEY, BP_KEY, VB_KEY from rasys.ra_task where archived = 'N';




   TASK_ID TASK_TYPE            STATE           WAITING_ON     DB_KEY DB_UNIQUE_NAME                 CREATION_TIME                       ERROR_COUNT     DF_KEY     BS_KEY     BP_KEY     VB_KEY

---------- -------------------- --------------- ---------- ---------- ------------------------------ ----------------------------------- ----------- ---------- ---------- ---------- ----------

      9626 INDEX_BACKUP         RUNNING                          1527 ORCL18                         15-AUG-19 12.47.12.479004 AM +02:00           0                             2223

      9627 INDEX_BACKUP         EXECUTABLE                       1527 ORCL18                         15-AUG-19 12.48.56.229219 AM +02:00           0                             2228

      9628 INDEX_BACKUP         EXECUTABLE                       1527 ORCL18                         15-AUG-19 12.49.02.200951 AM +02:00           0                             2232

      9629 INDEX_BACKUP         EXECUTABLE                       1527 ORCL18                         15-AUG-19 12.49.04.895727 AM +02:00           0                             2236




4 rows selected.




SQL>
Above look the task INDEX_BACKUP running BS_KEY 2223, and check that it is the same backupset for datafile 1 that I showed before (I executed the list backup for datafile 1 just after the backup finish). And while the INDEX_BACKUP task still running you can see check again the list backupset for datafile 1 at rman:

 

RMAN>  list backup of datafile 1;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2222    Incr 0  748.50M    SBT_TAPE    00:02:32     15-08-2019_00:47:10

        BP Key: 2223   Status: AVAILABLE  Compressed: NO  Tag: BKP-DB-INC0

        Handle: ORCL18_0tu97pnm_1_1   Media: Recovery Appliance (ZDLRA)

  List of Datafiles in backup set 2222

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1614233    15-08-2019_00:44:39              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2336    Incr 0  329.21M    SBT_TAPE    00:02:34     15-08-2019_00:47:12

        BP Key: 2337   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC0

        Handle: VB$_4142545763_2226I   Media:

  List of Datafiles in backup set 2336

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1614233    15-08-2019_00:44:39              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




RMAN>

 

Look how interesting it is. Now you have a new backupset (BS key 2336) that it is a clone for the original. Have same SCN, same TAG, same completion time, and size. The only difference is the handle that has the ZDLRA pattern VB$_*. And after some tasks inside ZDLRA finish (like RM_INC_FILES and RUN_PURGE_DF) you have just one backup of datafile 1:

 

RMAN> list backup of datafile 1;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2336    Incr 0  329.21M    SBT_TAPE    00:02:34     15-08-2019_00:47:12

        BP Key: 2337   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC0

        Handle: VB$_4142545763_2226I   Media:

  List of Datafiles in backup set 2336

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1614233    15-08-2019_00:44:39              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




RMAN>
As I told in my previous post, ZDLRA has the ability to open your backupset and index every block for your datafile, and more important: validated every block for each datafile.

 

Incremental Forever

 

Above was just for the initial level 0 backup, and it is not “incremental forever”. But if you do an incremental level 1 backup (I cropped the output to reduce the post size but it was just: BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT FILESPERSET 1 DATABASE TAG ‘BKP-DB-INC1’;. Check the list backup after that:
 
RMAN> list backup of datafile 1;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2336    Incr 0  329.21M    SBT_TAPE    00:02:34     15-08-2019_00:47:12

        BP Key: 2337   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC0

        Handle: VB$_4142545763_2226I   Media:

  List of Datafiles in backup set 2336

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1614233    15-08-2019_00:44:39              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2589    Incr 1  48.00K     SBT_TAPE    00:00:02     15-08-2019_00:58:00

        BP Key: 2590   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2588I   Media:

  List of Datafiles in backup set 2589

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    1  Incr 1614888    15-08-2019_00:57:59              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2644    Incr 0  328.08M    SBT_TAPE    00:00:02     15-08-2019_00:58:00

        BP Key: 2645   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2588_1   Media:

  List of Datafiles in backup set 2644

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1614888    15-08-2019_00:57:59              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




RMAN>

 

Look the details above. Inside of rman catalog at ZDLRA was made available to you a new level 0 backup that it is the virtual full backup. This level 0 have the same SCN than the incremental level 1.
So, basically, ZDLRA opened the backup incremental level 1 that you sent, read all the blocks, index it and based in the previous backup level 0 that was available (virtual or not – this is important) merged the block and present to you a new level 0. And if you do this every day, after an incremental level 1 you receive a complete virtual full backup.
In real life, it is simple like that. Again, check how interesting it is, you have now one backup level 0 available after the incremental that have a new backupset (BS key 2644) that it is a merge from the incremental level 1 (backupset 2589) with the previous level 0 (backupset 2636). And have the same SCN, same TAG, same completion time that the incremental level 1.
Some details that you need to take care of this. If you use rman encryption virtual full backups will not work. For TDE, you can use it, not a problem. But you can’t use TDE with rman compression enabled. Check  here. The recommendation too is using FILESPERSET 1 while doing the backup. This is good because reduce some overhead inside of ZDLRA (to open every backup and split it), and reduce pollution inside rman catalog list backup (if you sent a backup without fileperset, ZDLRA does not remove it from the list).
If I do a backup without filersperset the ZDLRA will open the bakupset and generate a virtual full for each datafile that it is inside of the original backupset. Check below that SCN is the same than incremental level 1:

 

RMAN> list backup of database completed after "sysdate - 5/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2905    Incr 1  1.26M      SBT_TAPE    00:00:08     15-08-2019_01:04:28

        BP Key: 2906   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2827I   Media:

  List of Datafiles in backup set 2905

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    1  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf

  3    1  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/sysaux01.dbf

  4    1  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/undotbs01.dbf

  7    1  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2992    Incr 0  328.09M    SBT_TAPE    00:00:08     15-08-2019_01:04:28

        BP Key: 2993   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2827_1   Media:

  List of Datafiles in backup set 2992

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  1    0  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/system01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

2996    Incr 0  40.00K     SBT_TAPE    00:00:08     15-08-2019_01:04:28

        BP Key: 2997   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2827_7   Media:

  List of Datafiles in backup set 2996

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  7    0  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/users01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3000    Incr 0  480.00K    SBT_TAPE    00:00:08     15-08-2019_01:04:28

        BP Key: 3001   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2827_4   Media:

  List of Datafiles in backup set 3000

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  4    0  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/undotbs01.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3004    Incr 0  135.47M    SBT_TAPE    00:00:08     15-08-2019_01:04:28

        BP Key: 3005   Status: AVAILABLE  Compressed: YES  Tag: BKP-DB-INC1

        Handle: VB$_4142545763_2827_3   Media:

  List of Datafiles in backup set 3004

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  3    0  Incr 1615589    15-08-2019_01:04:20              NO    /u01/app/oracle/oradata/ORCL18/sysaux01.dbf




RMAN>
The virtual backup continue to work if you use the SECTION SIZE (it is recommended by the way, especially for huge databases). Look the example:

 

[oracle@orcloel7 ~]$ sqlplus / as sysdba




SQL*Plus: Release 18.0.0.0.0 - Production on Thu Aug 15 01:11:58 2019

Version 18.6.0.0.0




Copyright (c) 1982, 2018, Oracle.  All rights reserved.







Connected to:

Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production

Version 18.6.0.0.0




SQL> create tablespace simon datafile '/u01/app/oracle/oradata/ORCL18/simon.dbf' size 1G;




Tablespace created.




SQL> exit

Disconnected from Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production

Version 18.6.0.0.0

[oracle@orcloel7 ~]$ rman target=/ catalog=cat_zdlra/s3nhazdlra@zdlra




Recovery Manager: Release 18.0.0.0.0 - Production on Thu Aug 15 01:15:24 2019

Version 18.6.0.0.0




Copyright (c) 1982, 2018, Oracle and/or its affiliates.  All rights reserved.




connected to target database: ORCL18 (DBID=3914023082)

connected to recovery catalog database

PL/SQL package CAT_ZDLRA.DBMS_RCVCAT version 18.03.00.00. in RCVCAT database is not current

PL/SQL package CAT_ZDLRA.DBMS_RCVMAN version 18.03.00.00 in RCVCAT database is not current




RMAN> report schema;




starting full resync of recovery catalog

full resync complete

Report of database schema for database with db_unique_name ORCL18




List of Permanent Datafiles

===========================

File Size(MB) Tablespace           RB segs Datafile Name

---- -------- -------------------- ------- ------------------------

1    870      SYSTEM               YES     /u01/app/oracle/oradata/ORCL18/system01.dbf

3    510      SYSAUX               NO      /u01/app/oracle/oradata/ORCL18/sysaux01.dbf

4    320      UNDOTBS1             YES     /u01/app/oracle/oradata/ORCL18/undotbs01.dbf

5    1024     SIMON                NO      /u01/app/oracle/oradata/ORCL18/simon.dbf

7    5        USERS                NO      /u01/app/oracle/oradata/ORCL18/users01.dbf




List of Temporary Files

=======================

File Size(MB) Tablespace           Maxsize(MB) Tempfile Name

---- -------- -------------------- ----------- --------------------

1    49       TEMP                 32767       /u01/app/oracle/oradata/ORCL18/temp01.dbf




RMAN> BACKUP INCREMENTAL LEVEL 0 SECTION SIZE 250M DEVICE TYPE SBT DATAFILE 5 TAG 'BKP-DBF5-INC0';




Starting backup at 15-08-2019_01:16:35

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=68 device type=SBT_TAPE

channel ORA_SBT_TAPE_1: RA Library (ZDLRA) SID=901CF26E504F1E35E053010310AC27EC

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 1 through 32000

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_01:16:37

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_01:16:38

piece handle=ORCL18_1fu97rjl_1_1 tag=BKP-DBF5-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 32001 through 64000

channel ORA_SBT_TAPE_1: starting piece 2 at 15-08-2019_01:16:38

channel ORA_SBT_TAPE_1: finished piece 2 at 15-08-2019_01:16:39

piece handle=ORCL18_1fu97rjl_2_1 tag=BKP-DBF5-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 64001 through 96000

channel ORA_SBT_TAPE_1: starting piece 3 at 15-08-2019_01:16:39

channel ORA_SBT_TAPE_1: finished piece 3 at 15-08-2019_01:16:42

piece handle=ORCL18_1fu97rjl_3_1 tag=BKP-DBF5-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:03

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 96001 through 128000

channel ORA_SBT_TAPE_1: starting piece 4 at 15-08-2019_01:16:42

channel ORA_SBT_TAPE_1: finished piece 4 at 15-08-2019_01:16:43

piece handle=ORCL18_1fu97rjl_4_1 tag=BKP-DBF5-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

channel ORA_SBT_TAPE_1: starting incremental level 0 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 128001 through 131072

channel ORA_SBT_TAPE_1: starting piece 5 at 15-08-2019_01:16:43

channel ORA_SBT_TAPE_1: finished piece 5 at 15-08-2019_01:16:44

piece handle=ORCL18_1fu97rjl_5_1 tag=BKP-DBF5-INC0 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

Finished backup at 15-08-2019_01:16:44




Starting Control File and SPFILE Autobackup at 15-08-2019_01:16:44

piece handle=c-3914023082-20190815-08 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 15-08-2019_01:16:52




RMAN> list backup of datafile 5;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3244    Incr 0  208.00K    SBT_TAPE    00:00:08     15-08-2019_01:16:45

  List of Datafiles in backup set 3244

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    0  Incr 1616756    15-08-2019_01:16:37              NO    /u01/app/oracle/oradata/ORCL18/simon.dbf




  Backup Set Copy #1 of backup set 3244

  Device Type Elapsed Time Completion Time     Compressed Tag

  ----------- ------------ ------------------- ---------- ---

  SBT_TAPE    00:00:08     15-08-2019_01:16:45 YES        BKP-DBF5-INC0




    List of Backup Pieces for backup set 3244 Copy #1

    BP Key  Pc# Status      Media                   Piece Name

    ------- --- ----------- ----------------------- ----------

    3245    1   AVAILABLE                           VB$_4142545763_3227I

    3250    2   AVAILABLE                           VB$_4142545763_3248I

    3255    3   AVAILABLE                           VB$_4142545763_3253I

    3260    4   AVAILABLE                           VB$_4142545763_3258I

    3272    5   AVAILABLE                           VB$_4142545763_3270I




RMAN> BACKUP INCREMENTAL LEVEL 1 SECTION SIZE 250M DEVICE TYPE SBT DATAFILE 5 TAG 'BKP-DBF5-INC1';




Starting backup at 15-08-2019_01:20:01

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 1 through 32000

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_01:20:02

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_01:20:03

piece handle=ORCL18_1lu97rq2_1_1 tag=BKP-DBF5-INC1 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 32001 through 64000

channel ORA_SBT_TAPE_1: starting piece 2 at 15-08-2019_01:20:03

channel ORA_SBT_TAPE_1: finished piece 2 at 15-08-2019_01:20:10

piece handle=ORCL18_1lu97rq2_2_1 tag=BKP-DBF5-INC1 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:07

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 64001 through 96000

channel ORA_SBT_TAPE_1: starting piece 3 at 15-08-2019_01:20:10

channel ORA_SBT_TAPE_1: finished piece 3 at 15-08-2019_01:20:11

piece handle=ORCL18_1lu97rq2_3_1 tag=BKP-DBF5-INC1 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 96001 through 128000

channel ORA_SBT_TAPE_1: starting piece 4 at 15-08-2019_01:20:11

channel ORA_SBT_TAPE_1: finished piece 4 at 15-08-2019_01:20:12

piece handle=ORCL18_1lu97rq2_4_1 tag=BKP-DBF5-INC1 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

backing up blocks 128001 through 131072

channel ORA_SBT_TAPE_1: starting piece 5 at 15-08-2019_01:20:12

channel ORA_SBT_TAPE_1: finished piece 5 at 15-08-2019_01:20:13

piece handle=ORCL18_1lu97rq2_5_1 tag=BKP-DBF5-INC1 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

Finished backup at 15-08-2019_01:20:13




Starting Control File and SPFILE Autobackup at 15-08-2019_01:20:13

piece handle=c-3914023082-20190815-09 comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 15-08-2019_01:20:21




RMAN> list backup of datafile 5;







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3244    Incr 0  208.00K    SBT_TAPE    00:00:08     15-08-2019_01:16:45

  List of Datafiles in backup set 3244

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    0  Incr 1616756    15-08-2019_01:16:37              NO    /u01/app/oracle/oradata/ORCL18/simon.dbf




  Backup Set Copy #1 of backup set 3244

  Device Type Elapsed Time Completion Time     Compressed Tag

  ----------- ------------ ------------------- ---------- ---

  SBT_TAPE    00:00:08     15-08-2019_01:16:45 YES        BKP-DBF5-INC0




    List of Backup Pieces for backup set 3244 Copy #1

    BP Key  Pc# Status      Media                   Piece Name

    ------- --- ----------- ----------------------- ----------

    3245    1   AVAILABLE                           VB$_4142545763_3227I

    3250    2   AVAILABLE                           VB$_4142545763_3248I

    3255    3   AVAILABLE                           VB$_4142545763_3253I

    3260    4   AVAILABLE                           VB$_4142545763_3258I

    3272    5   AVAILABLE                           VB$_4142545763_3270I




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3355    Incr 1  200.00K    SBT_TAPE    00:00:12     15-08-2019_01:20:14

  List of Datafiles in backup set 3355

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    1  Incr 1616986    15-08-2019_01:20:02              NO    /u01/app/oracle/oradata/ORCL18/simon.dbf




  Backup Set Copy #1 of backup set 3355

  Device Type Elapsed Time Completion Time     Compressed Tag

  ----------- ------------ ------------------- ---------- ---

  SBT_TAPE    00:00:12     15-08-2019_01:20:14 YES        BKP-DBF5-INC1




    List of Backup Pieces for backup set 3355 Copy #1

    BP Key  Pc# Status      Media                   Piece Name

    ------- --- ----------- ----------------------- ----------

    3356    1   AVAILABLE                           VB$_4142545763_3354I

    3406    2   AVAILABLE                           VB$_4142545763_3404I

    3415    3   AVAILABLE                           VB$_4142545763_3413I

    3424    4   AVAILABLE                           VB$_4142545763_3422I

    3433    5   AVAILABLE                           VB$_4142545763_3431I




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3359    Incr 0  132.50K    SBT_TAPE    00:00:12     15-08-2019_01:20:14

  List of Datafiles in backup set 3359

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    0  Incr 1616986    15-08-2019_01:20:02              NO    /u01/app/oracle/oradata/ORCL18/simon.dbf




  Backup Set Copy #1 of backup set 3359

  Device Type Elapsed Time Completion Time     Compressed Tag

  ----------- ------------ ------------------- ---------- ---

  SBT_TAPE    00:00:12     15-08-2019_01:20:14 YES        BKP-DBF5-INC1




    List of Backup Pieces for backup set 3359 Copy #1

    BP Key  Pc# Status      Media                   Piece Name

    ------- --- ----------- ----------------------- ----------

    3360    1   AVAILABLE                           VB$_4142545763_3354_5

    3410    2   AVAILABLE                           VB$_4142545763_3404_5

    3419    3   AVAILABLE                           VB$_4142545763_3413_5

    3428    4   AVAILABLE                           VB$_4142545763_3422_5

    3437    5   AVAILABLE                           VB$_4142545763_3431_5




RMAN>

 

As you can see, I created a tablespace and made backup level 0 (and 1) with section size. And after the index was created a virtual full backup, based on the incremental, for each size. This speed up the restore phase because you can parallelize it (instead if you use a single backupset). ZDLRA will “fix” if you made an incremental level 1 without section size:

 

RMAN> BACKUP INCREMENTAL LEVEL 1 DEVICE TYPE SBT DATAFILE 5 TAG 'BKP-DBF5-INC1';




Starting backup at 15-08-2019_01:23:17

using channel ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set

channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set

input datafile file number=00005 name=/u01/app/oracle/oradata/ORCL18/simon.dbf

channel ORA_SBT_TAPE_1: starting piece 1 at 15-08-2019_01:23:17

channel ORA_SBT_TAPE_1: finished piece 1 at 15-08-2019_01:23:18

piece handle=ORCL18_1ru97s05_1_1 tag=BKP-DBF5-INC1 comment=API Version 2.0,MMS Version 12.2.0.2

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:01

Finished backup at 15-08-2019_01:23:18




Starting Control File and SPFILE Autobackup at 15-08-2019_01:23:19

piece handle=c-3914023082-20190815-0a comment=API Version 2.0,MMS Version 12.2.0.2

Finished Control File and SPFILE Autobackup at 15-08-2019_01:23:26




RMAN> list backup of datafile 5 completed after "sysdate - 3/1440";







List of Backup Sets

===================







BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3572    Incr 1  40.00K     SBT_TAPE    00:00:02     15-08-2019_01:23:19

        BP Key: 3573   Status: AVAILABLE  Compressed: YES  Tag: BKP-DBF5-INC1

        Handle: VB$_4142545763_3571I   Media:

  List of Datafiles in backup set 3572

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    1  Incr 1617208    15-08-2019_01:23:18              NO    /u01/app/oracle/oradata/ORCL18/simon.dbf




BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ -------------------

3581    Incr 0  132.50K    SBT_TAPE    00:00:02     15-08-2019_01:23:19

  List of Datafiles in backup set 3581

  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name

  ---- -- ---- ---------- ------------------- ----------- ------ ----

  5    0  Incr 1617208    15-08-2019_01:23:18              NO    /u01/app/oracle/oradata/ORCL18/simon.dbf




  Backup Set Copy #1 of backup set 3581

  Device Type Elapsed Time Completion Time     Compressed Tag

  ----------- ------------ ------------------- ---------- ---

  SBT_TAPE    00:00:02     15-08-2019_01:23:19 YES        BKP-DBF5-INC1




    List of Backup Pieces for backup set 3581 Copy #1

    BP Key  Pc# Status      Media                   Piece Name

    ------- --- ----------- ----------------------- ----------

    3582    1   AVAILABLE                           VB$_4142545763_3576_5

    3586    2   AVAILABLE                           VB$_4142545763_3577_5

    3590    3   AVAILABLE                           VB$_4142545763_3578_5

    3594    4   AVAILABLE                           VB$_4142545763_3579_5

    3598    5   AVAILABLE                           VB$_4142545763_3580_5




RMAN>

 

Look that ZDLRA opened the incremental level that was sent (without section size) index it and after that generated a level 0 (that originally was done with section size) and created a virtual full backup correctly
You can see here (Output-Full-Execution-Backup-Level-0-and-Backup-Level-1) a full execution for backup level 0, subsequent levels 1, to verify the generation of virtual full backups and the incremental forever strategy. Check the SCN, creation times for backups 0 that appear in the list after I create a backup level 1.

 

Virtual Full backup, Incremental forever, and Backup Strategy 

 

Virtual full backups change completely the backup strategy for your database. The incremental forever strategy starts it the key point so save you for a lot of problems.
If you start to check your backup strategy, there are a lot of details to verify. So, what it is your backup strategy? Full every weekend plus incremental every day? What it is the load for your backup appliance? And how many times do you pass doing the backup? If you have a huge database, maybe a full every weekend it is impossible, and you need to sparse in the month (1 time per month is feasible).
But the most important, the time to recover and validate the backup? You need to restore the full (form last week as an example), and all the incremental + plus archives since that. This can be really time-consuming.
But if you can do everything with a single incremental level 1 every day? You receive a full backup, that was validated by ZDLRA, and per datafile data you have. If ZDLRA detects corrupted blocks it advises you about that. The incremental forever strategy help in other places too. The overhead over the network decrease a lot (just pass incremental). The time consumption for CPU inside of database too.
Now think the restore of the database. If you need to do one restore in the middle of the week you need to use the nearest level 0 that cover your request (probably done during the weekend) and apply some days incremental and archivelogs. For ZDLRA, you have one level 0 one day before available and just need to recover just a few archives.
The base idea here is that ZDLRA it is the only appliance that delivers to you a backup full (virtual full backup, as they call) after every incremental backup that you do. This approach is called incremental forever. And even besides that, ZDLRA it is the only one that open correctly what rman sent to you (filesperset or no, section size, compressed or no) and verifies every block of datafile inside, index it, and deliver to you a validated backupset. And as you saw, in the examples before, everything related with catalog it is done automatically by ZDLRA (backup deletion and others).
In the next post, I will show you more details about what’s happens inside of ZDLRA when it receives the incremental backup. The internal tasks and where it’s store the information about the virtual backups.
 
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”

 


ZDLRA Internals, Tables and Storage
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA Internals, Tables and Storage

ZDLRA tables are owned by rasys user and are one extension of rman recovery catalog views. The internal tables that were added are just a few, but are important to understand how tasks works (mainly INDEX_BACKUP). In this post, I will show tables like plans, plans_details, blocks, and chunks. And besides that, show a little glimpse about how the files are stored.
Extra tables
If you check, officially, the rman recovery catalog already includes some columns from ZDLRA, the column VB_KEY identify the virtual backup key inside ZDLRA. It exists in RC_BACKUP_PIECE as an example. The extra tables added are linked with SBT tasks (clone to tape or cloud), replication (to other ZDLRA), storage (chunks), vbdf and plans (virtual full backup).

 

The most import tables are listed below.

 

RASYS.VBDF: This table store the reference for virtual full backups created for the datafile. Have the information about the backup piece used as the source(base) for virtual full backup and the reference to the new backup piece that was generated at virtual full creation. The most import columns are:

 

  • VB_KEY: The unique identifier to virtual backup.
  • DF_KEY: The reference to datafile key for virtual full backup.
  • CKP_ID and CHP_SCN: Checkpoint information (id/scn) for this virtual full backup.
  • SRCBP_KEY: The reference to the original backup piece (source) that was used to generate the virtual full backup. Links to table BP.
  • VCBP_KEY: Reference to the derived backup piece that was created during the index of the original (source) backup piece.

 

RASYS.PLANS: This table store the plans that link the datafile and the virtual full backup. Basically, is the reference “index” and links: virtual backup, datafile, and the number of chunks that store it. Important columns:

 

  • VB_KEY: The unique identifier to virtual backup.
  • DF_KEY: The reference to datafile key for virtual full backup.
  • NUMCHUNKS: Chunks used to store (or store) this virtual full backup.

 

RASYS.PLANS_DETAILS: IOT table that stores the detailed information for the plan for the “index”. It is partitioned over the DF_KEY column. Link the virtual full backup, datafile, block number, and chunk identification. Tells you in which chuck it is stored each block that are part of the virtual full backup. Import columns:

 

  • VB_KEY: The unique identifier to virtual backup.
  • DF_KEY: The reference to datafile key for virtual full backup.
  • BLOCKNO: Block number that are part of the datafile.
  • NUMBLKS: Number of the blocks in sequential mode starting from BLOCKNO that are equal.
  • CHUNKNO: What is the chunk id that store the BLOCKNO + NUMBKS.

 

RASYS.BLOCKS: IOT table that stores the info for each block that belongs to the datafile. It is partitioned over the DF_KEY column. There is no link to any backup, but store each version of a single block that is needed to restore any backup (the block number needed can be found in the PLAN_DETAILS and SCN come from other tables). Import columns:

 

  • DF_KEY: The reference to datafile key for virtual full backup.
  • BLOCKNO: Block number that are part of the datafile.
  • SCN and CKP_ID: Checkpoint and SCN for this specific version of the block.

 

RASYS.CHUNK: IOT table that has the information about the chunks that was used to store the backups for the datafile. Important columns:

 

  • DF_KEY: The reference to datafile key for virtual full backup.
  • CHUNKNO: What is the chunk id.
  • FILESIZE: Size for this chunk
  • FILENAME: Derived file name that store the chunk (no represent the file inside ASM).

 

These tables are the most important for ZDLRA for the internal part of ZDLRA. They are not needed in daily basis activities but I think that is useful if you want to understand how things work internally. The list above is just a glimpse for internal tables, but based on that you can easily: identify what are the datafile blocks that belongs for specific virtual backup, where the blocks are stored (which chunk), and how this virtual full backup was build (the plan that contains all the blocks).

 

 

Storage

 

As you saw above, the blocks are stored in chunks and in CHUNKS table we have filename information. But this is not related where the data are really stored, there are other places used too. All of these tables are stored at RA_POOL tablespace but don’t store the backup itself. The structure that does this is “CONTAINERS” and represent files inside ASM diskgroup.
If you see the ZDLRA doc (look the image below) this is called DELTA POOLS (CONTAINERS and CHUNKS) and is stored inside one “Delta Store” (ASM diskgroup). For ASM diskgroups you can see the CATALOG and DELTA. The CATALOG, as I wrote in this post, store the metadata information about the backups (mostly the rman catalog views) and the DELTA (the Delta Store) have the info related the backups itself.
For tablespaces, there are two important RA_POOL and RA_CATALOG. The tables listed before (in the “Extra Tables topic”) resides at RA_POOL tablespace. But the backup data itself is stored in CONTAINER files at DELTA diskgroup and you can check them in the table SYS.AMCONT$. Another option is going in ASM diskgroup and check the list at DELTA/CONTAINER files (each file have the default of 2TB):
ASMCMD> pwd

+DELTA/ZDLRA/CONTAINER

ASMCMD> ls -s

Block_Size      Blocks          Bytes          Space  Name

       512  4294959104  2199019061248  4398059094016  con.561.935405781

       512  4294959104  2199019061248  4398059094016  con.562.935405789

       512  4294959104  2199019061248  4398059094016  con.563.935405797

       512  4294959104  2199019061248  4398059094016  con.564.935405805

…

…

       512  4294959104  2199019061248  4398059094016  con.656.935406489

       512   308051968   157722607616   315692679168  con.657.935406495

ASMCMD>
You can see more info checking tables SYS.AM* tables and views (SYS.AMAU$, SYS.AMCONT$, and SYS.AM$CONTAINER are examples). The list for containers files are not even listed as ZDLRA database (as datafile in dba_data_files) but are created directly by ZDLRA library.
You can even test the performance (before starting to use ZDLRA usage) to create these files using rastat.pl: https://docs.oracle.com/cd/E88198_01/AMAGD/amagd_monitor.htm#AMAGD1503. And this is the only info in the docs about CONTAINERS files.
Last info about ZDLRA database and storage, the database operates in NOARCHIVLOG mode. The backups for it are daily exports from the metadata information and in case of corruption/lost you can reconstruct the information reading the con files.
Conclusion
Understand the tables involved for ZDLRA are important to see what’s happening internally, how the index is stored and how works virtual full backup. The tables listed above are not used in daily bases for ZDLRA, you usually go to rman catalog view (RC_*), everything is there.
But, PLANS, PLANS_DETAILS, and VCBP are the base for a task like INDEX_BACKUP and to sustain virtual full backups. I needed to pass over these tables before explaining in more details the virtual full creation and INDEX_BACKUP task.
The storage part, containers and ASM diskgroup are needed just to help support to identify some bottleneck or problem. You don’t need to use it but I think that it is important to show you, at least superficially, how its work.

 

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


Webinar – Understanding ZDLRA
Category: Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 1

Webinar - Understanding ZDRLA

Hi guys !!
On December 18th at 19:00 PM (CET) we will have our Understanding ZDRLA Webinar.

Speaker: Fernando Simon
 
You can follow it live here or via Youtube – Click Here
 
Click here for download presentation. 

André Ontalba – Board Member

Understanding ZDLRA
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Understanding ZDLRA

Zero Data Loss Recovery Appliance (ZDLRA) deliver to you the capacity to improve the reliability of your environment in more than one way. You can improve the RPO (Recovery Point Objective) for your databases until you reach zero, zero data loss. And besides that, adding a lot of new cool features (virtual backups) on the way how you do that your backups (incremental forever), and backup strategy. And again, besides that, improve the MAA at the highest level that you can hit.
But this is just marketing, right? No, really, works pretty well! My history with ZDLRA  starts with Oracle Open World 2014 when they released the ZDLRA and I watched the session/presentation. At that moment I figure out how good the solution was. In that moment, hit exactly the problem that I was suffering for databases: deduplication (bad dedup). One year later, in 2015 at OOW I made the presentation for a big project that I coordinate (from definition implementation, and usage)  with 2 Sites + 2 ZDLRA + N Exadata’s + Zero RPO and RTO + DG + Replication. And at the end of 2017 moved to a new continent, but still involved with MAA until today.
This post is just a little start point about ZDLRA, I will do a quick review about some key points but will write about each one (with examples) in several other dedicates posts. I will not cover the bureaucratic steps to build the project like that, POC, scope definition, key turn points, and budget. I will talk technically about ZDLRA.

ZDLRA

ZDLRA is one Engineered System built over Exadata. Contains Exadata Storage and Database nodes, InfiniBand network, SAS drivers (for tape connection). So, a good machine with redundancy components. Look the base config:

 

 

But as Exadata, it is not just hardware, it is software too (and a lot of in this case). ZDLRA solve problems in more than one topic, it is not just a backup appliance where you redirect your backups, it is an appliance that provides zero data loss. To do that the key features are:
  • Virtual backups: Based on the input backups (incremental) generate a virtual full backup for you.
  • Real-time Redo: ZDLRA can be an archive log destination for your database, RPO zero.
  • Replication: You can replicate data between ZDLRA’s, eliminating single point of failure.
  • Tape integration: ZDLRA used OSB to copy backups to tape.
  • Rman: totally compatible with rman commands, and deliver to you rman catalog self-managed.
  • Database and MAA integration: fully integrated with MAA environments, including DG.
Since it is built over Exadata, the updates follow the same procedure, IB, Storage, Db’s, and after that ZDLRA software. But in this case, you don’t need to concern about integration or tuning for Exadata, everything is done by Oracle. For Storage, ZDLRA uses two diskgroups: DELTA (to store the backups) and CATALOG (to store the rman catalog tables).
The integration between ZDLRA and your databases are simple. You have an SBT_TAPE library to use and the rman catalog database. The library communicates with ZDLRA appliance and sends the blocks, and the rman catalog store the metadata.
Talking about rman catalog, using ZDLRA you need to use the rman catalog. The catalog database that you connect it is the ZDLRA itself. And the catalog it is managed by ZDLRA software, this means that when you send the backup to ZDLRA tape driver, and the ZDLRA process it, the catalog reflect all the new virtual full backups that are available. You don’t need to manage the backups, everything is done by ZDLRA software (delete backup, crosscheck, validation, etc).

 

Virtual backups

Probably this is the most know feature for ZDLRA. In the most basic description, it “simple” create to you a new backup level 0 just using incremental level 1 backups. The functionality is simple:
  1. Ingest one first rman level 0 backup.
  2. Ingest subsequent rman level 1 backup.
  3. A new “virtual level 0” appears available to you.
  4.  
But how it’s work? Basically, ZDLRA open the rman block and read the datafile blocks that are inside of rman block. After that, using the already existed backup (stored inside ZDLRA) for that datafile, generate to you a new level 0 backup. This is called “incremental forever strategy”, where (after first level 0) you just need to do incremental backups.
If you are thinking about deduplication at this moment, you are almost right. But is really different of other vendors that just index the rman block (by hash, md5, whatever), ZDLRA it is the only one that opens the block and look what it is inside. It’s pick up the content of rman block, the data blocks for each datafile that is inside, and index it.
But why it is better than deduplication? Because it is sensitive to the context. And in this case, it is Oracle context. Work’s better too because can handle every kind of database blocks sizes, any kind of filesperset and chunks that you specify. You don’t need to “black magic” to minimally have good values for dedup, like DataDomain as example: https://community.emc.com/docs/DOC-24118 and https://community.emc.com/docs/DOC-18775. You can argue that DataDomain can do the same, it opens the rman block (just 8k and 16k blocks), but it was done by reverse engineering over the sent block https://www.emc.com/collateral/white-papers/h15389-data-domain-oracle-optimized-deduplication.pdf. But if Oracle changes this pattern? Sometimes can happen that rman read the same datafile blocks (that was not touched by dbwr) but because sga/pga buffers create a different rman block.
A little representation. Think that you work in a Library and I deliver to you (every day) one box with books of an encyclopedia. You store this box but can’t open it to store the books that are inside. You can ignore the box and says that you already have this based on the size, weight, whatever you choose to define if you already have this box. As ZDLRA, you receive the same box, but you open and check if you already have this book in your collection or no. One day I come back and ask you the full encyclopedia. So, what solution do you think that will be better for Oracle backups?
The first is the normal deduplication. Works very well for a lot of blocks types (word, excel, txt) because you “maybe” have the key to open the box. But not so well for binary blocks type (like rman). ZDLRA it is Oracle, and have the key to open the box (rman block) and check the content to identify books (the datafile blocks). Clever no?
There are a lot of other details to cover, I will do in a delicate post about Virtual backups.

Real-Time Redo

The name from “zero data loss” came from this feature because its guarantee zero RPO for your database. In a simple way, ZDLRA can be archive log destination for your database. It’s work the same way that you do for DG configuration, but more simply because you just need to configure the archive_log_dest.
You can think that it is similar, or you can do the same, using DG Far Sync: https://www.oracle.com/technetwork/database/availability/8395-data-guard-far-sync-1967244.pdf. But here for ZDLRA, you don’t need standby database to be protected and every transaction still will be protected.
It is important to say that it is more than just archive log destination because can sync the redo log buffers of your database. Don’t need to wait archive be generated to sync. Can work sync or async, and if your database has an outage (by storage death as an example), ZDLRA generates for you one archivelog. And another thing it is that there is no need to do archivelog backups (if you use real-time redo) because ZDLRA detect the log switch and generate for you (inside catalog) the archivelog.
Besides that, if you use ZDLRA replicated, you can have additional protection sending to another site. I will talk later about that but using replicated ZDLRA’s, you can have RPO zero for DG databases and for non DG protected databases.
Oracle calls Virtual Backups + Real-Time Redo as Delta Push because you just sent to ZDLRA the minimum data needed to be protected. Just the delta difference. This reduces a lot the load over the system, network, avoid reading unusual/repeated data from the source database and inside ZDLRA too.

 

 

Replication

ZDLRA can work replicating backups between two ZDLRA’s. This replication is done automatically, using a dedicated network and can be:
  • One way: You have the ZDLRA master (called upstream) and the destination (called downstream).
  • Bi-Directional: Two ZDLRA’s replicate backups between each self. Both are upstream and downstream.
  • Hub-and-Spoke: You have one downstream ZDLRA that receive the backups from more than one upstream.
The detail here is that replication is an additional feature of ZDLRA usage, even a downstream ZDLRA can protect one site (like standby site), receiving backups from databases as a normal ZDLRA. The downstream ZDLRA does not lock in as just destination from upstream. But since it is a replicated environment, you can restore your backupset from the downstream replica (connected in the upstream ZDLRA).
Based on ZDLRA design, the rman catalog is updated too. This means that after the backup being replicated to the downstream, inside of the catalog automatically appears another copy for your backupset. And you can have different policies between ZDLRA’s. So, in the upstream you can have one policy with retention as 30 days (because maybe you need to do more backups) and the downstream have retention as 90 days. 

 

Tape and Cloud integration

ZDLRA can sent backups to media manager (tapes), to do that the recommended way is use the pre-installed OSB (Oracle Secure backup) as media manager library. ZDLRA manage everything, used the dedicated SAN network to send the backups and manage the tape drivers. Another option is sync ZDLRA with Oracle Cloud object storage.
But you can use third part software as tape media manager driver, you just need to install the agent in database nodes (of ZDLRA) and configure it as media manger library. It is not trivial, but you can do and integrate your tape library with ZDLRA. In this case all the communication (copy from ZDLRA to tape) will be done using LAN.
Again, the rman catalog will be automatically managed by ZDLRA. When the backupset it is copy to tape, the copy appear available for restore in your rman catalog.
To manage the tape backups you use the same policies that you use for backups. You can specify the retention period, copies, and more. If you use OSB, ZDLRA will manage everything (free tapes as example).

 

 

For Cloud, the idea is the same. You have one plugin that appears in your ZDLRA as a tape library, and the jobs clones from ZDLRA to cloud (in the same way than tape). But since Oracle Cloud is fully encrypted, you have some steps to manage the key to allow you to integrate the system. This can be one way to have the third site to store your backups outside of your environment.

 

Rman

As told before, ZDLRA has two main parts: the software itself (that create the virtual backups) and the database. And this database contains the rman catalog that you use to connect. ZDLRA manage automatically the rman catalog database, this means that every backup is sent appears in the catalog. And everything that happens with this backup, the derived virtual full backup, clone to tape, replication, exclusion because retention expiration is reflected automatically in the rman catalog. You don’t need to manage or catalog the backupsets.
But the most important thing is that everything that you already made with rman works 100% with ZDLRA. You don’t need to learn new commands, new backups types. Everything that you need it is “backup incremental level”.
Since ZDLRA will be a central part of your environment, it is important to know how to search for data in the catalog (using SQL). I will show some examples of how to understand the RC_* tables/views that will allow you to create some reports from your backups.

 

Database and MAA integration

Since it is really linked with MAA family the integration between ZDLRA and databases is quite easy, covering all the points for protection. About protection, not just for backup, but raising the bar for transaction protection to reach zero data loss in any kind of scenario (DG and non DG databases).
ZDLRA it is not just a backup appliance, it allows you to reach RPO zero even not using DG. But if you are using DG you need to take attention to some details. The idea about DG for MAA team it is that everyone does your job, this means that if you need to protect a database for site outage, DG does that. ZDLRA it is not a full standby database.
Saying that, when you have a full DG environment, you can reach RTO (Recovery Time Objective) as zero too. ZDLRA alone can provide to you just RPO zero but does not help you to have zero downtime (or unavailability). You don’t not loose transactions with RPO zero, but will need some time to recover the database. The only way to have both, RTO and RPO zero is using DG. In this scenario, each database does the backup in ZDLRA that it is in the same site, and the DG does the sync between the sited (for DB with DG).
If you have more than one site and want to protect everything the best solution is have ZDLRA in both sites. You can still have just one ZDLRA and protect everything, but remember that you need to choose one site (or a third one) to put your ZDLRA.
But the good part is that you can mix everything. There are a lot of cases that some databases are not protected by DG. For these, you can use real-time redo to have RPO zero, and replicate to another ZDLRA in another site to avoid the only point of failure.
This raise the bar for MAA, because it is the highest level that you can reach in protection. Even if you cut the communication for each site you still have RPO zero because ZDLRA protection. And if you have a complete site or server failure, the RTO will be zero because DG.
This was the project that I designed and you can see this (was presented in OOW 2015: https://www.oracle.com/technetwork/database/availability/con8830-zdlradeepdive-2811109.pdf).

Conclusion

ZDLRA it is not just a backup appliance, it is more than that. It allows you to reach RPO zero for any kind of environment. And together with DG, you can reach zero RPO and RTO.
But besides that, the ZDLRA software delivers to you a lot of features that help you in daily bases activities. The virtual backups deliver to you a validated level zero backupset for each datafile. You receive a rman catalog self-managed, you don’t need to catalog (or uncatalog) pieces for a clone to tapes, deletion, replications that you do.
And there is no need to learn complex new features, you continue to do the backups in the same way that you do previously. “backup incremental level 1” and just that.
For the next post, I will try to cover feature by feature and a lot of details, virtual backups, real-time redo, dg integration, internal tasks, updates. Stay tuned.

 

References

All the images and base info for this post came from these links:
https://www.oracle.com/technetwork/database/availability/con8830-zdlradeepdive-2811109.pdf
https://www.oracle.com/technetwork/database/availability/oow14-con7684-zdlra-techoverview-2332009.pdf
https://www.oracle.com/technetwork/database/availability/8395-data-guard-far-sync-1967244.pdf
https://docs.oracle.com/cd/E55822_01/AMOGD/E37432-22.pdf
https://docs.oracle.com/cd/E55822_01/AMAGD/E37431-29.pdf
http://www.oracle.com/us/products/engineered-systems/recovery-appliance-twp-2313693.pdf
https://www.oracle.com/technetwork/database/availability/recoveryapplianceperformance-2636208.pdf
https://www.oracle.com/technetwork/database/availability/recovery-appliance-businesscase-2574428.pdf
https://www.oracle.com/technetwork/database/availability/recovery-appliance-data-guard-2767512.pdf
https://www.oracle.com/technetwork/database/features/availability/ow-papers-089200.html

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


INDEX_BACKUP task for ZDLRA, Percentage done
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

INDEX_BACKUP task for ZDLRA, Percentage done

Quick post how to check and identify done for INDEX_BACKUP task in ZDLRA. In one simple way, just to contextualize, INDEX_BACKUP is one task for ZDLRA that (after you input the backup of datafile) generate an index of the blocks and create the virtual backup for you.
Here I will start a new series about ZDLRA with some hints based on my usage experience (practically since the release in 2014). The post from today is just little scratch about ZDLRA internals, I will extend this post in others (and future posts), stay tuned.

INDEX_BACKUP

As told before this task is one of the most important for ZDLRA, but it can take a lot of time to index the backup if the file is huge (a lot of TB’). Think that ZDLRA need to index all the blocks for the file, and for the first level 0 this took some time.
Unfortunately, officially there is no way to check how far you are from the end and how much was already done to create the index. The table that shows you the info for running tasks doesn’t contain this information. Because of that, I created a little SQL that you can use to verify the task and try to discover the percentage of already done.

 

 

SQL

In the SQL I will use two tables from RASYS ZDLRA schema:
The SQL:

 

 

SELECT s.ba_session_id, s.instance_id, s.sid, s.serial#, s.job_name

       , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state, rt.waiting_on

       , rt.elapsed_seconds

       , gs.module, gs.sql_id, gs.action, gs.event

       , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

FROM sessions s

JOIN ra_task rt

ON rt.task_id = s.current_task

JOIN gv$session gs

ON gs.inst_id = s.instance_id

AND gs.sid = s.sid

AND gs.serial# = s.serial#

ORDER BY rt.LAST_EXECUTE_TIME DESC

 

This query is a join between RA_TASK, SESSION (from ZDLRA), and V$SESSION from DB side. As you can see, the link between RA_TASK and SESSION is the current task. And since you reach the SESSION info from ZDLRA, you can link with the sid, serial#, and inst_id running the task. Fortunately, the V$SESSION has the desired information. This query returns all running tasks, you can filter by task type if desired.
 Look the result from one execution:
SQL> col JOB_NAME format a30

SQL> col STATE format a10

SQL> col DB_UNIQUE_NAME format a20

SQL> col MODULE format a30

SQL> col ACTION format a20

SQL> col EVENT format a35

SQL> col TASK_TYPE format a15

SQL> set linesize 400

SQL> l

  1  SELECT s.ba_session_id, s.instance_id, s.sid, s.serial#, s.job_name

  2         , rt.task_id, rt.DB_KEY, rt.db_unique_name, rt.task_type, rt.state, rt.waiting_on

  3         , rt.elapsed_seconds

  4         , gs.module, gs.sql_id, gs.action, gs.event

  5         , rt.BP_KEY, rt.bs_key, rt.df_key, rt.vb_key

  6  FROM sessions s

  7  JOIN ra_task rt

  8  ON rt.task_id = s.current_task

  9  JOIN gv$session gs

 10  ON gs.inst_id = s.instance_id

 11  AND gs.sid = s.sid

 12  AND gs.serial# = s.serial#

 13* ORDER BY rt.LAST_EXECUTE_TIME DESC

SQL> /




BA_SESSION_ID INSTANCE_ID        SID    SERIAL# JOB_NAME                          TASK_ID     DB_KEY DB_UNIQUE_NAME       TASK_TYPE       STATE      WAITING_ON ELAPSED_SECONDS MODULE                         SQL_ID        ACTION               EVENT                                   BP_KEY     BS_KEY     DF_KEY     VB_KEY

------------- ----------- ---------- ---------- ------------------------------ ---------- ---------- -------------------- --------------- ---------- ---------- --------------- ------------------------------ ------------- -------------------- ----------------------------------- ---------- ---------- ---------- ----------

     93729765           2       4453      42235 RA$_EXEC_93730430                93676068     477602 SIMON_DB1            INDEX_BACKUP    RUNNING                    28558.1086 q_restore_fast                 brt6uuhzacdnu plan 12% done        cell single block physical read       41032043




SQL>

 

Columns import here:
  • ACTION: from GV$SESSION show the percentage already done for this INDEX_BACKUP task.
  • ELAPSED_SECONDS: from RA_TASK, show seconds that this task is running. You can use together with the percentage and try to figure out the time to reach 100%.
  • BP_KEY/DF_KEY/BS_KEY/VB_KEY: from RA_TASK, show base info from what is the input to task. In this case, and usually, for INDEX_BACKUP tasks the only column will be BP_KEY (that represent the backup piece key id) because you are indexing a new backup that entered in the ZDLRA.
As you can see, the INDEX_BACKUP task 93676068 for the backup piece 41032043 took 28558 seconds and it is running without a wait and processed 12% of the backup piece. We can dig a little more try to identify and validate some values from blocks processed, but I will cover in the second part.
Conclusion
As you can see, with this simple query you can check the percentage done for the INDEX_BACKUP task. Was needed to query some internal tables of database, but the desired result was easy to find. I tested this query with old version of ZDLRA (12.2.1.1.1-201805 from August of 2018) and with the last version (12.2.1.1.2-201907 from 11/July) and the result was the same, and always accurate.
I will post more about ZDLRA, from usage experience to internal details.
You can check my presentation at OOW 2015 about my MAA project that used ZDLRA to protect all environment: https://www.oracle.com/technetwork/database/availability/con8830-zdlradeepdive-2811109.pdf
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


1 … 3 4 5 6 7
LUXOUG – LUXEMBOURG ORACLE USERS GROUP
© 2024 LUXOUG – LUXEMBOURG ORACLE USERS GROUP