And according to documentation:

“The optional replication network connects the local Recovery Appliance (the upstream appliance) with a remote Recovery Appliance (the downstream appliance). Oracle recommends a broadband, encrypted network, instead of an insecure public network, wherever possible.”

It is optional because you can share the same “Ingest Network” to receive replication, but is not recommended. But whatever the network mode that you choose the “Replication Network” is a different subnet that not overlaps/is part of the “Ingest Network”.

ZDLRA Environment

In this post I have two ZDLRA’s to enable the replication. But this can vary in your example. So, I will show how to do in both nodes, in both ZDLRA’s (will be 4 to do).

So, use this as a guide on how to do, but remember to adapt to your environment. Ip’s, physical networks (bond or no), and routes are an example of details that will change. I will focus on how to configure the GI and ZDLRA in this post.

And the most important, I recommend opening a proactive SR at MOS informing what are you doing and asking doubts before the start. A good start point is a note “Post Install – Replication Network Configuration for ZDLRA (Doc ID 2126047.1)” at MOS.

Basic network config

The network configuration follows the same as normal Linux network configuration: you need to define the separated ip’s for both ZDLRA’s, if you will use bond, or the route table. Since this depends on every environment I will not cover here.

But basically:

1. Hostname: You need to choose at least: 1 Hostname at your replication network for each node, 1 Hostname for VIP for each node

2. Scan Hostname: 1 Hostname to be used as a scan for replication data exchange. The scan will be used for each ZDLRA that you are configuring.

3. Configure network files: Configure the ifcfg files. The files ifcfg-eth2, ifcfg-eth3 and ifcfg-bondeth1 if you will use bond. If just ifcfg-eth2 if you will use a single connection. This is done in all nodes of the ZDLRA appliance.

4. Route configuration: You need to guarantee that access made by the replication network is not routed through the normal ingest network. This depends on the way that you have your network architecture, but maybe you need to change the files “route-*” in each node.

My current system is:

ZDLRAS1: ZDLRA installed on site 1, have two nodes: zdlras1n1 and zdlras1n2. It will be upstream (who send backups).

ZDLRAS2: ZDLRA installed on site 2, have two nodes: zdlras2n1 and zdlras2n2. It will be downstream (who received backup).

What I will add:

ZDLRAS1: I will add: zdlras1n1-rvip (200.254.255.21) as replication vip for zdlras1n1, zdlras1n2-rvip (200.254.255.22) as replication vip for zdlras1n2. Scan zdlras1-rep.

ZDLRAS2: I will add: zdlras1n1-rvip (200.254.255.23) as replication vip for zdlras2n1, zdlras2n2-rvip (200.254.255.24) as replication vip for zdlras2n2. Scan zdlras2-rep

GI Configuration

The next step after the basic configuration is configure the GI to add the network, vip, and scan.

Checking interfaces

After you configure the Linux part and the network basic configuration the GI can be check if the new interface can be used by GI. So, the first step is to check this in both nodes in both ZDLRA’s with command “oifcfg iflist” (I cropped below to show just what is needed):

############## Upstream ZDLRA

[root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/oifcfg iflist -p -n

…

bondeth1  200.254.255.0  UNKNOWN  255.255.255.0

…

[root@zdlras1n1 ~]#




############## Downstream ZDLRA

[root@zdlras2n1 ~]#  /u01/app/19.0.0.0/grid/bin/oifcfg iflist -p -n

…

bondeth1  200.254.255.0  UNKNOWN  255.255.255.0

…

[root@zdlras2n1 ~]#

Add network

Since the interfaces are visible and not used, we can add the network number 2 at GI level. This is done in just one node per ZDLRA. TO do that we use the command “srvctl add network” as root user in both ZDLRA’s that we are configuring:

############## Upstream ZDLRA

[root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl config network -k 2

PRCR-1001 : Resource ora.net2.network does not exist

[root@zdlras1n1 ~]#

[root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add network -netnum 2 -subnet 200.254.255.0/255.255.255.0/bondeth1

[root@zdlras1n1 ~]#

[root@zdlras1n1 ~]# srvctl config network -k 2

Network 2 exists

Subnet IPv4: 200.254.255.0/255.255.255.0/bondeth1, static

Subnet IPv6:

Ping Targets:

Network is enabled

Network is individually enabled on nodes:

Network is individually disabled on nodes:

[root@zdlras1n1 ~]#




############## Downstream ZDLRA

[root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl config network -k 2

PRCR-1001 : Resource ora.net2.network does not exist

[root@zdlras2n1 ~]#

[root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add network -netnum 2 -subnet 200.254.255.0/255.255.255.0/bondeth1

[root@zdlras2n1 ~]#

[root@zdlras2n1 ~]# srvctl config network -k 2

Network 2 exists

Subnet IPv4: 200.254.255.0/255.255.255.0/bondeth1, static

Subnet IPv6:

Ping Targets:

Network is enabled

Network is individually enabled on nodes:

Network is individually disabled on nodes:

[root@zdlras2n1 ~]#

Look above that we are using the bondeth1 as the interface for this network (the bondeth0 if for ingest network). And I tested if the network is available or no (need to be unused).

Add VIP

After we can add the VIP for each node for each ZDLRA that we are configuring (done using “srvctl add vip”):

############## Upstream ZDLRA

[root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras1n1 -A zdlras1n1-rvip.oralocal/255.255.255.0/bondeth1 -k 2

[root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras1n2 -A zdlras1n2-rvip.oralocal/255.255.255.0/bondeth1 -k 2

[root@zdlras1n1 ~]#




############## Downstream ZDLRA

[root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras2n1 -A zdlras2n1-rvip.oralocal/255.255.255.0/bondeth1 -k 2

[root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add vip -n zdlras2n2 -A zdlras2n2-rvip.oralocal/255.255.255.0/bondeth1 -k 2

[root@zdlras2n1 ~]#

Check that I defined the hostname for each vip for each node and the parameter “k” that defined the network where this vip will be created. Be careful with the “n” parameter that defined the node name.

Add Scan

After add vip we can add scan for each ZDLRA cluster with “srvctl add scan”:

############## Upstream ZDLRA

[root@zdlras1n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add scan -netnum 2 -scanname zdlras1-rep.oralocal

[root@zdlras1n1 ~]#




############## Downstream ZDLRA

[root@zdlras2n1 ~]# /u01/app/19.0.0.0/grid/bin/srvctl add scan -netnum 2 -scanname zdlras2-rep.oralocal

[root@zdlras2n1 ~]#

Be careful to use the connect “netnum” parameter value (needs to point to the second network) and scan name.

Start Vip

After the scan we can start the vips for each node in each ZDLRA:

############## Upstream ZDLRA

[root@zdlras1n1 ~]# srvctl status vip -i zdlras1n1-rvip.oralocal

VIP 200.254.255.21 is enabled

VIP 200.254.255.21 is running on node: zdlras1n1

[root@zdlras1n1 ~]#

[root@zdlras1n1 ~]# srvctl status vip -i zdlras1n2-rvip.oralocal

VIP 200.254.255.22 is enabled

VIP 200.254.255.22 is running on node: zdlras1n2

[root@zdlras1n1 ~]#




############## Downstream ZDLRA

[root@zdlras2n1 ~]# srvctl status vip -i zdlras2n1-rvip.oralocal

VIP 200.254.255.23 is enabled

VIP 200.254.255.23 is running on node: zdlras2n1

[root@zdlras2n1 ~]#

[root@zdlras2n1 ~]# srvctl status vip -i zdlras2n2-rvip.oralocal

VIP 200.254.255.24 is enabled

VIP 200.254.255.24 is running on node: zdlras2n2

[root@zdlras2n1 ~]#

Create Listener

The last step for GI configuration is to create and start “listener” and “scan listener” in each ZDLRA cluster:

############## Upstream ZDLRA

[oracle@zdlras1n1 ~]$ export ORACLE_HOME=/u01/app/19.0.0.0/grid

[oracle@zdlras1n1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zdlras1n1 ~]$  

[oracle@zdlras1n1 ~]$ srvctl add listener -l LISTENER_REPL -p 1522 -k 2

[oracle@zdlras1n1 ~]$ srvctl start listener -l LISTENER_REPL

[oracle@zdlras1n1 ~]$ srvctl status listener -l LISTENER_REPL

Listener LISTENER_REPL is enabled

Listener LISTENER_REPL is running on node(s): zdlras1n1,zdlras1n2

[oracle@zdlras1n1 ~]$

[oracle@zdlras1n1 ~]$ srvctl add scan_listener -netnum 2 -listener LISTENER_REPL -endpoints TCP:1522

[oracle@zdlras1n1 ~]$ srvctl start scan_listener -netnum 2

[oracle@zdlras1n1 ~]$ srvctl status scan_listener -netnum 2

SCAN Listener LISTENER_REPL_SCAN1_NET2 is enabled

SCAN listener LISTENER_REPL_SCAN1_NET2 is running on node zdlras1n1

SCAN Listener LISTENER_REPL_SCAN2_NET2 is enabled

SCAN listener LISTENER_REPL_SCAN2_NET2 is running on node zdlras1n2

SCAN Listener LISTENER_REPL_SCAN3_NET2 is enabled

SCAN listener LISTENER_REPL_SCAN3_NET2 is running on node zdlras1n1

[oracle@zdlras1n1 ~]$




############## Downstream ZDLRA

[root@zdlras2n1 ~]# su - oracle

Last login: Sat Nov 23 21:28:03 CET 2019

[oracle@zdlras2n1 ~]$ export ORACLE_HOME=/u01/app/19.0.0.0/grid

[oracle@zdlras2n1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

[oracle@zdlras2n1 ~]$

[oracle@zdlras2n1 ~]$ srvctl add listener -l LISTENER_REPL -p 1522 -k 2

[oracle@zdlras2n1 ~]$ srvctl start listener -l LISTENER_REPL

[oracle@zdlras2n1 ~]$ srvctl status listener -l LISTENER_REPL

Listener LISTENER_REPL is enabled

Listener LISTENER_REPL is running on node(s): zdlras2n1,zdlras2n2

[oracle@zdlras2n1 ~]$

[oracle@zdlras2n1 ~]$

[oracle@zdlras2n1 ~]$ srvctl add scan_listener -netnum 2 -listener LISTENER_REPL -endpoints TCP:1522

[oracle@zdlras2n1 ~]$ srvctl start scan_listener -netnum 2

[oracle@zdlras2n1 ~]$ srvctl status scan_listener -netnum 2

SCAN Listener LISTENER_REPL_SCAN1_NET2 is enabled

SCAN listener LISTENER_REPL_SCAN1_NET2 is running on node zdlras2n1

SCAN Listener LISTENER_REPL_SCAN2_NET2 is enabled

SCAN listener LISTENER_REPL_SCAN2_NET2 is running on node zdlras2n2

SCAN Listener LISTENER_REPL_SCAN3_NET2 is enabled

SCAN listener LISTENER_REPL_SCAN3_NET2 is running on node zdlras2n1

[oracle@zdlras2n1 ~]$

[oracle@zdlras2n1 ~]$

The important detail here:

• Port: it is 1522. Look at the “endpoint” parameter.

• Network: Check that listener is added ate network #2 (parameter “k”).

• Listener name: we use LISTENER_REPL to follow the ZDLRA default config.

• User: All the commands are run with oracle

•  

All the commands are executed in just one node of the cluster because they affect all nodes at the same time.

With that now with have the network configured in each node of both ZDLRA’s, all the vips and scan up and running, and also the listening to this new network.

ZDLRA Configuration

Since we have all nodes with everything configured, we can add this network ad ZDLRA config. The idea for ZDLRA is to allow it to receive and send backups trough this new network, and this is incredibly easy.

We need to change just two parameters at ZDLRA config tables: REPLICATION_IP_ADDRESS and BACKUP_IP_ADDRESS. The configuration change internal ZDLRA tables, so, before doing that review and check with Oracle Support at MOS if everything is OK and you can proceed.

REPLICATION_IP_ADDRESS

This configuration resides at the intenal “rasys.host” table for ZDLRA. So, we update the table column:

############## Upstream ZDLRA

[oracle@zdlras1n1 ~]$ sqlplus rasys/change^Me2




SQL*Plus: Release 19.0.0.0.0 - Production on Sat Nov 23 23:22:09 2019

Version 19.3.0.0.0




Copyright (c) 1982, 2019, Oracle.  All rights reserved.




Last Successful login time: Sat Nov 23 2019 21:33:26 +01:00




Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0




SQL> set linesize 250

SQL>

SQL> col node_name format a50

SQL> col REPLICATION_IP_ADDRESS format a50

SQL> select node_name,replication_ip_address from host;




NODE_NAME                                          REPLICATION_IP_ADDRESS

-------------------------------------------------- --------------------------------------------------

zdlras1n1.oralocal

zdlras1n2.oralocal




SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.21' where NODE_NAME = 'zdlras1n1.oralocal';




1 row updated.




SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.22' where NODE_NAME = 'zdlras1n2.oralocal';




1 row updated.




SQL> select node_name,replication_ip_address from host;




NODE_NAME                                          REPLICATION_IP_ADDRESS

-------------------------------------------------- --------------------------------------------------

zdlras1n1.oralocal                                 200.254.255.21

zdlras1n2.oralocal                                 200.254.255.22




SQL>




############## Downstream ZDLRA

[oracle@zdlras2n1 ~]$ sqlplus rasys/change^Me2




SQL*Plus: Release 19.0.0.0.0 - Production on Sat Nov 23 23:39:13 2019

Version 19.3.0.0.0




Copyright (c) 1982, 2019, Oracle.  All rights reserved.




Last Successful login time: Sat Nov 23 2019 19:54:17 +01:00




Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0




SQL> set linesize 250

SQL> col node_name format a50

SQL> col REPLICATION_IP_ADDRESS format a50

SQL>

SQL> select node_name,replication_ip_address from host;




NODE_NAME                                          REPLICATION_IP_ADDRESS

-------------------------------------------------- --------------------------------------------------

zdlras2n2.oralocal

zdlras2n1.oralocal




SQL>

SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.23' where NODE_NAME = 'zdlras2n1.oralocal';




1 row updated.




SQL> update HOST set REPLICATION_IP_ADDRESS='200.254.255.24' where NODE_NAME = 'zdlras2n2.oralocal';




1 row updated.




SQL>

SQL> select node_name,replication_ip_address from host;




NODE_NAME                                          REPLICATION_IP_ADDRESS

-------------------------------------------------- --------------------------------------------------

zdlras2n2.oralocal                                 200.254.255.24

zdlras2n1.oralocal                                 200.254.255.23




SQL>

As you can see above, the REPLICATION_IP_ADDRESS for each node reflects the IP at the replication network.

BACKUP_IP_ADDRESS

This configuration defines with IP’s the ZDLRA allows ingesting backups. Since the replication basically is a ingest coming from other network, we need to allow it. To do that we change the table internal table rai_host. And, of course, we can commit if everything is OK.

############## Upstream ZDLRA

SQL> col ADMIN_IP_ADDRESS format a30

SQL> col BACKUP_IP_ADDRESS format a30

SQL> SELECT * FROM rai_host;




NODE_NAME                                          ADMIN_IP_ADDRESS               BACKUP_IP_ADDRESS              REPLICATION_IP_ADDRESS

-------------------------------------------------- ------------------------------ ------------------------------ --------------------------------------------------

zdlras1n1.oralocal                                 10.160.5.1                     10.160.5.2                     200.254.255.21

zdlras1n2.oralocal                                 10.160.5.3                     10.160.5.4                     200.254.255.22




SQL> UPDATE rai_host SET backup_ip_address='200.254.255.21,'|| backup_ip_address WHERE node_name = 'zdlras1n1.oralocal';




1 row updated.




SQL> UPDATE rai_host SET backup_ip_address='200.254.255.22,'|| backup_ip_address WHERE node_name = 'zdlras1n2.oralocal';




1 row updated.




SQL> SELECT * FROM rai_host;




NODE_NAME                                          ADMIN_IP_ADDRESS               BACKUP_IP_ADDRESS              REPLICATION_IP_ADDRESS

-------------------------------------------------- ------------------------------ ------------------------------ --------------------------------------------------

zdlras1n1.oralocal                                 10.160.5.1                     200.254.255.21,10.160.5.2      200.254.255.21

zdlras1n2.oralocal                                 10.160.5.3                     200.254.255.22,10.160.5.4      200.254.255.22




SQL> commit;




Commit complete.




SQL>




############## Downstream ZDLRA

SQL> col ADMIN_IP_ADDRESS format a30

SQL> col BACKUP_IP_ADDRESS format a30

SQL>

SQL> SELECT * FROM rai_host;




NODE_NAME                                          ADMIN_IP_ADDRESS               BACKUP_IP_ADDRESS              REPLICATION_IP_ADDRESS

-------------------------------------------------- ------------------------------ ------------------------------ --------------------------------------------------

zdlras2n2.oralocal                                 10.160.6.3                     10.160.6.4                     200.254.255.24

zdlras2n1.oralocal                                 10.160.6.1                     10.160.6.2                     200.254.255.23




SQL>

SQL> UPDATE rai_host SET backup_ip_address='200.254.255.23,'|| backup_ip_address WHERE node_name = 'zdlras2n1.oralocal';




1 row updated.




SQL> UPDATE rai_host SET backup_ip_address='200.254.255.24,'|| backup_ip_address WHERE node_name = 'zdlras2n2.oralocal';




1 row updated.




SQL> SELECT * FROM rai_host;




NODE_NAME                                          ADMIN_IP_ADDRESS               BACKUP_IP_ADDRESS              REPLICATION_IP_ADDRESS

-------------------------------------------------- ------------------------------ ------------------------------ --------------------------------------------------

zdlras2n2.oralocal                                 10.160.6.3                     200.254.255.24,10.160.6.4      200.254.255.24

zdlras2n1.oralocal                                 10.160.6.1                     200.254.255.23,10.160.6.2      200.254.255.23




SQL> commit;




Commit complete.




SQL>

After that, I recommend to reboot, ate least ZDLRA database to reload the configs. Again, remember that the values here (ip, networks, interfaces, and routes) will differ for you’re your environment. Use as a guide.

Replication

The replication for ZDLRA have some steps to be done. If we do this during the deployment is easier (since is done automatically by installer). But if we need to do after (because adding or preparing the environment) is not complicate, but we need to be aware of some details.

Be careful with a basic configuration like interface (bond or no), and ip’s. They will change in your scenario, but the most important is to check the route. The packet route needs to travel just inside the replication network because the ZDLRA configuration (at internal tables) expects/listens in just one ip.

And about ZDLRA internal change, they occur at internal tables. So, always review correctly the values to avoid errors (before the commit). After that, we can use the DBMS_RA.CREATE_REPLICATION_SERVER and create the replication server config, I will show this in another post.

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”