Engineer System
ZDLRA, since 2014
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ZDLRA, since 2014

In Oracle Open World 2014 the Zero Data Loss Recovery Appliance (ZDLRA) was released and it changed MAA in many ways, but two principals: protection and backup. I watched the ZDLRA presentation and saw that matched with the needs that I had that time.
After OOW in 2014 I started the project (all phases, from conception, requirements until deployments and usage) that become (in 2015) the first ZDLRA installation in Brazil, and one of the first of the world too that use replicated ZDLRA to protect both sites (primary and standby) and many levels of databases (PRO, TST, DEV). The Oracle MAA at its finest was amazing: ZDLRA + Exadata + DG; everything integrated to protect both sites.
Because of the high design level of the project it was chosen to be one of the main presentation in Oracle Open World 2015 about ZDLRA, you can find the link of the presentation that I made together with ZDLRA dev team here. As told before, in this project was integrated two ZDLRA, two Exadata and DG to reach ZERO Recover Point Objective (RPO) and Recovery Time Objective (RTO) and beside that, reduce backup time. You can see the presentation to check the scope and other details.
I will start to write more about ZDLRA, from technical part until project details. How you use and how it impact in your MAA projects. Just to remember that ZDLRA it is not just backup oriented appliance, it is design to protect mission critical environments, where the goal is zero data loss; but also help you to improve your backup and recovery environment. Of course that you can use ZDLRA to protect all of the Oracle databases, from Sparc, RISC and x86; from DEV to PROD; from single to RAC DG environment.
As usual, I always recommend to follow the MAA page to read the “Best Practices” and “Presentations”:

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


Oracle Open World 2019
Category: Cloud,Database,Engineer System Author: Andre Luiz Dutra Ontalba (Board Member) Date: 5 years ago Comments: 0

Oracle Open World 2019

 
 
First day of OOW19 and we had big announcements of new services and products.
 
Now as Larry said “Oracle Cloud is Allways Free for Everyone”
 
Oracle’s Free Tier program has two components:
Always Free services, which provide access to Oracle Cloud services for an unlimited time
Free Trial, which provides $ 300 in credits for 30 days to try additional services and larger shapes
The new Always Free program includes the essentials users need to build and test applications in the cloud: Oracle Autonomous Database, Compute VMs, Block Volumes, Object and Archive Storage, and Load Balancer. Specifications include:
2 Autonomous Databases (Autonomous Data Warehouse or Autonomous Transaction Processing), each with 1 OCPU and 20 GB storage
2 Compute VMs, each with 1/8 OCPU and 1 GB memory
2 Block Volumes, 100 GB total, with up to 5 free backups
10 GB Object Storage, 10 GB Archive Storage, and 50,000 / month API requests
1 Load Balancer, 10 Mbps bandwidth
10 TB / month Outbound Data Transfer
500 million ingestion Datapoints and 1 billion Datapoints for Monitoring Service
1 million Notification delivery options per month and 1000 emails per month
More information click here.
Another release was the Oracle Autonomous Linux “World’s First Autonomous Operating System.
Introducing Oracle OS Management Service
Oracle Autonomous Linux, in conjunction with Oracle OS Management Service, uses advanced machine learning and autonomous capabilities to deliver unprecedented cost savings, security and availability and frees up critical IT resources to tackle more strategic initiatives.
Eliminate manual OS management — World’s first autonomous operating system in the cloud to deliver automated patching, updates, and tuning without human intervention. Based on a preconfigured Oracle Linux image; automated daily package updates; enhanced OS parameter tuning and OS diagnostics gathering.
Deliver automatic, in-depth protection at all levels — 100 percent hands-off automatic security updates daily to the Linux kernel and key user space libraries. This requires no downtime along with protection from both external attacks and malicious internal users. Known Exploit Detection provides automated alerts if anyone attempts to exploit a vulnerability that has been patched by Oracle.
Provides always-on availability — Includes automated patching and upgrades while the system is running, eliminating unnecessary downtime for users and the system.
More information click here
 
A release expected by customers the new partnership with VMware.
 
For the first time, Oracle will officially support Oracle products running on VMware virtualized environments and VMware will support Oracle running VMware Workloads on Oracle Cloud WIN/WIN for customers!
 

 

 
Running VMware workloads in Oracle Cloud
With this announcement, Oracle becomes a partner in the VMware Cloud Provider Program and Oracle Cloud VMware Solution will be sold by Oracle and its partners. The solution will be based on VMware Cloud Foundation and will deliver a full software-defined data center (SDDC) stack including VMware vSphere, NSX, and vSAN. Through consistent infrastructure and operations, customers will be able to migrate and modernize applications, seamlessly moving workloads between on-premise environments and Oracle Cloud.
Customers will be able to easily use Oracle services, such as Oracle Autonomous Database, Oracle Exadata Cloud Service and Oracle Database Cloud, which run in the same cloud data centers, on the same networks, with a consistent portal and APIs. able to leverage Oracle’s rapidly expanding footprint of global regions to scale globally without needing to establish their own data centers. Oracle will provide administrative access to the underlying physical servers, enabling a level of control previously only possible on premise, and customers will be able to use VMware vCenter to manage both their on-premise clusters and Oracle Cloud-based SDDCs through a single crash of glass Oracle will also provide first line technical support for this solution.
To learn more about the offering visit: https://www.oracle.com/cloud/VMware
 
We also had the Exadata X8M release.
The Exadata X8M uses the Remote Direct Memory (RDMA) to allow database access to the storage server memory directly. And the memory, in this case, it is special, X8M uses Intel Optane DC Persistent Memory modules (DIMM / NVDIMM – Non Volatile DIMM – to provide PMEM – Persistent Memory) attached directly to server storage and these can be accessed directly from the database using RDMA trough RoCE network. Let’s check the details to see what it is.
Our contributor Fernando Simon a few hours ago posted an article about Exadata X8M where it can be viewed http://www.fernandosimon.com/blog/exadata-x8m/.
Soon more will be available here on the Portal this article and more details about the new services and products.
 
 Andre Luiz Dutra Ontalba
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”the new partnership with VMware


Exadata X8, Second look
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Exadata X8, Second look

Every year Oracle arrives and release new version of Exadata with a plenty of new things. We have the natural evolution from hardware (more memory, more cpu…) and sometimes news features from software side. The point for this post today it is not about the HW and SW things, but something is hidden in the small lines of the new X8.
In the Exadata X8 datasheet it is possible to read about the new Extended (XT) Storage Server. And for me, this was the best thing in the new release. With the new XT you can have, in the same appliance (and for your database usage), one dedicated storage that you can use for some goods and cool things.
Think about, today if you have a huge partitioned table in your database you continue to store, in the same place (sharing the same disks), the partition that you use in almost every transaction and the partition that use one time per month. The same can occurs if you have a DSS database with mix of OLTP and DW loads. Or even mix DEV/TST/PRO databases in the same appliance.
And for Exadata you “waste” features like flash, smart scan, storage server memory and the most important – money – (because the storage server need to be licensed) for this “not used data” too. Until X8 you don’t have what to do, the options to offload this data outside of Exadata are complicate and require some intervention in database side beside that you need to sustain different hardware.
So, why it is more “Efficient”, “Simple”, “Secure”, “Scalable” and “Compatible”? Because in the same appliance you can have, transparently from/for database side and using the same db features (like TDE as example), two storages. You can create one additional diskgroup in the XT storage servers and move your archived data there. Simple like that. Another option, if you use OVM, you can dedicate XT for a DEV/TST vm.
If you read the data sheet, you saw a quickly information about “storage software is optional”. But this is something that need clarification. In the docs about System Overview for 19.2 version (page 284 or here) you can see two interesting things:
As you can see, you still have the Exadata Software installed in XT server, but all the SQLOffload features (like smartscan – storage index I am not sure) will be disabled. By the way, about the Exadata Software, there is no other option, it is required to be there to share the disks thought iDB because basically it is not ODA with “JBOD/NtoN/X” connection between database servers and disks. But as told, it is cropped from offload features BUT you don’t need to pay the expensive storage license for use it.
About the hardware itself, it is the same X8-2L model for HC with just one processor, less memory, no flash and 14TB disks. And you cant upgrade the memory. The XT servers is just for “Flexible Configurations” with at least two XT servers in the config.
So, with the XT server Oracle added new devices for Exadata that increase the available options for database usage, but at same time it is complete transparent and more cheaper. You don’t need to configure fancy things to offload your data outside of Exadata, you use the same sql to read data and continued to have a good response time. Maybe you can use ASM Flex disk groups too with XT to have more flexibility. And if you use the OVM you can dedicate this XT servers for DEV/TST vms too. From maintenance part, you have the same procedure to upgrade the storage software stack.
The bad part from XT, and I believe that will occur, is that some customer will buy one appliance with a lot of XT servers and will complain that “brought Exadata and it is slow”. So, if you are a sales person, please inform correctly the customer about the usage for XT servers.
At the end, if you think about, Exadata it is there in the market since 2010 with SQLOffloading/smartscan/storage index and there is nothing in the market to beat it. Of course that exist hardware that it is more powerful than Exadata. You can have new fancy all flash storage, but if you do full stable scan over 1TB table you still load this amount from storage to database memory; for Exadata the history is different. Remember that Exadata it is not just hardware, it is software too.
 

 

I wrote one article about new option for Exadata X8
http://www.fernandosimon.com/blog/exadata-x8-second-look/
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


Exadata – Advanced Intrusion Detection Environment (AIDE)
Category: Engineer System Author: Rodrigo Mufalani (Board Member) Date: 5 years ago Comments: 0

Exadata – Advanced Intrusion Detection Environment (AIDE)

Few days ago, we have upgrade one Exadata environment to 19c version. One of new features named AIDE – Advanced Intrusion Detection Environment was introduced on this release of Exadata software, it came pre-configured. This feature is really awesome, and call my attention, because it tracks if the system files were modified and then helps the DMA (Database Machine Administrator) to find issues and security breaches on their environments. AIDE run crontab jobs daily and monitor if files changes in specific directories. If some files changes, it raise an alert and generate logs on /var/log/aide/aide.log.

 

On Oracle documentation:
exadataAIDE Syntax
The utility is located at /opt/oracle.SupportTools/exadataAIDE.
1
exadataAIDE [-s|-status] [-e|enable] [-d|disable] [-u|-update] [-h|help]
Description of syntax options:
  • -s[tatus] : Print the current status of the AIDE daily cron job
  • -e[nable] : Enable the AIDE daily cron job
  • -d[isable] : Disable the AIDE daily cron job
  • -u[pdate] : Update the AIDE database metadata and run the daily scan
  • -h[elp] : Print the command syntax and help information
  • Get the current status of the aide cron job.exadataAIDE –status
  • Disable the daily AIDE scan.exadataAIDE –disable
  • Enable the daily AIDE scan.exadataAIDE –enable
  • Update the AIDE database after making changes to the system.exadataAIDE –update
Let’s take a look on the “warning file” generated by AIDE.
 [root@myexa01 ~]# cat /var/log/aide/aide.log
 AIDE 0.15.1 found differences between database and filesystem!!
 Start timestamp: 2019-07-21 19:57:20
 Summary:
   Total number of files:        55788
   Added files:                  0
   Removed files:                0
   Changed files:                2
 
 Changed files:
 changed: /home/oracle/monitor/check_compliance
 changed: /home/oracle/monitor/check_oracle
 
 Detailed information about changes:
 File: /home/oracle/monitor/check_listener
  SHA256   : AnuG4ldrP0yB15r/3nOGN+nrnW18391+ , wqPj5hnOIl1HGK7f85/E+0tWGYvEKbH5
 File: /home/oracle/monitor/check_oracle
  SHA256   : t6zk+jufuBfia/YCNHS1WqJt/GPW8xSX , Xk3ZkzP/YpT2jbKvjORsRF3TfoCteY8Z
After that alert, for our environment we need to add an exception for a specific directory by the end of the file:
[root@myexa01 ~]# vi /etc/aide.conf

#Add one Exception for monitoring directory
!/home/oracle/monitor/
Then we need to rebuild the database for AIDE, the DB is located at /var/lib/aide/aide.db.gz
[root@myexa01 ~]# /opt/oracle.SupportTools/exadataAIDE -u
 AIDE: database update request accepted.
Then, we need to clen the alerts raised. To cleanup we use dbmcli utility
[root@myexa01 ~]# dbmcli
 DBMCLI> LIST ALERTHISTORY
          1_1     2019-07-20T18:24:29+02:00       warning         "Advanced Intrusion Detection Environment (AIDE) detected potential changes to software on this system. The changes are in /var/log/aide/aide.log "
          1_2     2019-07-22T09:14:55+02:00       clear           "Advanced Intrusion Detection Environment (AIDE) violation has been cleared."
 
DBMCLI> drop alerthistory 1_1, 1_2
 Alert 1_1 successfully dropped
 Alert 1_2 successfully dropped
To check more this secutiry framework on Exadata, please refer:
https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmsq/exadata-security-practices.html#GUID-74FF1D0C-59F2-4F02-B648-34048AC02C31

 

 

 

All the Best,
Rodrigo Mufalani

 

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited  to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


ODA REIMAGE
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

ODA REIMAGE

The idea to reimage ODA is to refresh the environment without the need to jump from one by one to reach the last available version, or even rescue the system from S.O. failure/crash. The process to do a reimage can be check in the official documentation but unfortunately can be very tricky because the information (the order and steps) are not 100% clear. The idea is to show you how to reimage using version 18 (18.3 in this example), that represents the last available.

 

In resume the process is executed in the order:
  • ILOM: Boot the ISO
  • Prepare to create the appliance
  • Upload GI and DB base version to the repository
  • Linux: Create the appliance
  • Firmware and patch
  • Create Oracle Homes and the databases
  • Finish and clean the install
The environment in this scenario is:
  • ODA HA: Oda HA version with two nodes and with SSD disk available.
  • Reimage with 18 version: 18.3 version at the moment that reimages was done, but can use any 18.x version
  • Databases: 11, 12, 18 for Oracle Home and available versions
Before even start the reimage you have two steps. The first is to verify the hardware against failures. Disks, memory, CPU, MB, whatever error that exists need to be clean before the start. If no, you can hit error during the process. To check about hw error, you can log in the ILOM in the browser and verify for some incidents/errors (this vary from version to version of ODA, but usually the errors are visible in the first page of ILOM). Or you can log in in the console for ILOM and execute the command “show faulty”. Whatever the mode, check in both nodes. If you have a connection with the system execute smartctl –a /dev/DISK and query about errors and look in the column “Total uncorrected errors”. Another option is check if the disk is in warning state using the command smartctl -q errorsonly -H -l xerror /dev/DISK; if you have the report will be similar to “SMART Health Status: WARNING: ascq=0x97 [asc=b, ascq=97]”. For the errors, open SR and wait for the maintenance from Oracle HW team.

 

The second is save/backup every config that you need. If you did not lose the system backup from /etc folder it is a good start. Another hint is execute “ifconfig” and “route” to have one output from IP’s that you have and the routing table (vlan list it is important too if used). For Oracle files, backup tnsnames.ora, listener.ora, and sqlnet.ora for every OH or GI that you have. These files are example, depend on every system and needs. Adjust if you need something additional.

 

ILOM

 

The process to reimage start putting the ISO in the ilom and booting both nodes with that. After the boot, the install runs automatically, and you have a Linux to create the appliance.
Before starting the reimage one explanation since the machine was already in use it needs to cleanup the disks. The first option is cleanup before the reimage, using the software from the currently running linux/oda/oak. Or running the cleanup after the reimage finish and before creating the appliance (the option that I will use here).

 

The steps to reimage from ILOM are done in this order:
  • Login ILOM both nodes using the browser
  • Navigate to “Host Management” menu and choose “Host Control” submenu.
    1. Select the option “CDROM” in the list and after click in SAVE.
  • Navigate to “Remote Control” menu and choose “Redirection” submenu.
    1. Click on the option “Launch Remote Control Console”
    2. You can receive some confirmation to execute some java (depend on how your machine has java configured) or ask to agree about the certificate. Always click yes and agree or continue.
  • In the newly opened window select the menu “KVMS” and after the submenu “Storage”.
  • In the “Storage Devices” do:
    1. Click on the “Add”
    2. Select the ISO file and after click in the option “Select”
    3. After, verify if the ISO appeared in the “Path” and the “Device Type” is “ISO Image” in the Storage Devices menu. Select the line with the ISO and after click in the option “Connect”.
    4. After you click in the “Connect” the information marked in red will appear in the screen and the menu now will appear as “Disconnect”. You just need to click in the option “OK”.
    5. DO NOT CLONE the Console.
  • Go back to the Browser and click in the menu “Host Management” and after in the submenu “Power Control”
    1. Select the option “Power Cycle” and after click in the “Save” option.
    2. Maybe you will receive on a question if you are sure about the reboot. Just click Ok.
  • You will see in the console:
    1. Reboot
    2. The process will boot the ISO
    3. Install the new ODA Image
In the end, you have the Linux console

 

All the steps above you can see in the Image gallery below. All the images are linked with the step mentioned.

 

 

 
Remember to do this in both nodes. You can do this both at the same time. The installation time depends more from the network because involve uploading the ISO from machine to ODA. The postscript part during the installation tends to take a lot of time.
Please does not close the Remote Console because this kills the ISO mounting device and the installation will crash. There is no problem if the browser login receives timeout, the installation will continue.
One important thing from this part is that does not update the ILOM SP or even other firmware for HW. This will be executed in the next steps.
Recently Oracle changed the ODA base page and removed the links to ISO files. But the patch 27604623 contains the Oracle Database Appliance Bare-metal ISO Image to download. Just to be clear, I am reimaging for baremetal (not OVM).

 

Prepare to create the appliance

After you “reimage” you need to configure and create the appliance. But before we need to execute some steps. Now, to connect in the Linux console we have two options: same java console opened to do the reimage, or, ssh to ilom ip and start the console (start /SP/console). I will use the last option because allow to work (copy and paste commands) more smoothly. Example (root/welcome1 are user/password that is the default for reimaged ODA):

 

Oracle(R) Integrated Lights Out Manager




Version 3.2.9.23 r116695




Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.




Warning: HTTPS certificate is set to factory default.




Hostname: odak1-ilom




-> start /SP/console

Are you sure you want to start /SP/console (y/n)? y




Serial console started.  To stop, type ESC (







Oracle Linux Server release 6.10

Kernel 4.1.12-124.18.6.el6uek.x86_64 on an x86_64




test0 login: root

[root@test0 ~]#


Cleanup

As told before, we need to clean the headers for ODA. I start checking the headers with command “cleanup.pl -checkHeader”:

 

[root@test0 ~]# /opt/oracle/oak/onecmd/cleanup.pl -checkHeader

Command may take few minutes...

OS Disk               Disk Type    OAK Header   ASM Header(p1)       ASM Header(p2)

 /dev/sda                HDD        Found           Found                Found

 /dev/sdb                HDD        Found           Found                Found

 /dev/sdaa               HDD        Found           Found                Found

 /dev/sdab               HDD        Found           Found                Found

 /dev/sdac               HDD        Found           Found                Found

 /dev/sdad               HDD        Found           Found                Found

 /dev/sdae               HDD        Found           Found                Found

 /dev/sdaf               HDD        Found           Found                Found

 /dev/sdag               HDD        Found           Found                Found

 /dev/sdah               HDD        Found          Erased               Erased

 /dev/sdai               HDD        Found           Found                Found

 /dev/sdaj               HDD        Found           Found                Found

 /dev/sdak               HDD        Found           Found                Found

 /dev/sdal               HDD        Found           Found                Found

 /dev/sdam               HDD        Found           Found                Found

 /dev/sdan               HDD        Found           Found                Found

 /dev/sdao               SSD        Found           Found       p2 not created

 /dev/sdap               SSD        Found           Found       p2 not created

 /dev/sdaq               SSD        Found           Found       p2 not created

 /dev/sdar               SSD        Found           Found       p2 not created

 /dev/sdas               SSD        Found           Found       p2 not created

 /dev/sdat               SSD        Found           Found       p2 not created

 /dev/sdau               SSD        Found           Found       p2 not created

 /dev/sdav               SSD        Found           Found       p2 not created

[root@test0 ~]#


Look above that some ASM Header was found. To clean we execute “/opt/oracle/oak/onecmd/cleanup.pl”  with option “-erasedata”. The cleanup script executes more than just erase disk headers, it cleans all the configurations (hostname, ips, users) and because of this needs to be executed in both nodes from HA env. This equalizes all nodes with the same config and avoid an error when you create the appliance. Example:

 

[root@test0 ~]# /opt/oracle/oak/onecmd/cleanup.pl -erasedata

INFO: *******************************************************************

INFO: ** Starting process to cleanup provisioned host oak1             **

INFO: *******************************************************************

WARNING: Secure Erase is an irrecoverable process. All data on the disk

WARNING: will be erased, and cannot be recovered by any means. On X3-2,

WARNING: X4-2, and X5-2 HA, the secure erase process can take more than

WARNING: 10 hours. If you need this data, then take a complete backup

WARNING: before proceeding.

Do you want to continue (yes/no) : yes

INFO:

....

....

INFO: Executing </opt/oracle/oak/bin/odaeraser.py -a -f -v >

Start erasing disks on the system

On some platforms, this will take several hours to finish, please wait

Do you want to continue (yes|no) [yes] ? yes

Number of disks are processing: 0




Disk     Vendor     Model                    Erase method         Status       Time(seconds)

e0_pd_00 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_01 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_02 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_03 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_04 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_05 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_06 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_07 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_08 HGST       H7280A520SUN8.0T         SCSI crypto erase    success               0

e0_pd_09 HGST       H7280A520SUN8.0T         SCSI cryp[  777.976350] reboot: Restarting system

...

...

After node reboot you can check the disk readers (look that the cleanup will change the hostname to “oak”):

 

[root@oak2 ~]# /opt/oracle/oak/onecmd/cleanup.pl -checkHeader

Command may take few minutes...

OS Disk               Disk Type    OAK Header   ASM Header(p1)       ASM Header(p2)

 /dev/sda                HDD       Erased          Erased              UnKnown

 /dev/sdb                HDD       Erased          Erased              UnKnown

 /dev/sdaa               HDD       Erased          Erased              UnKnown

 /dev/sdab               HDD       Erased          Erased              UnKnown

 /dev/sdac               HDD       Erased          Erased              UnKnown

 /dev/sdad               HDD       Erased          Erased              UnKnown

 /dev/sdae               HDD       Erased          Erased              UnKnown

 /dev/sdaf               HDD       Erased          Erased              UnKnown

 /dev/sdag               HDD       Erased          Erased              UnKnown

 /dev/sdah               HDD       Erased          Erased              UnKnown

 /dev/sdai               HDD       Erased          Erased              UnKnown

 /dev/sdaj               HDD       Erased          Erased              UnKnown

 /dev/sdak               HDD       Erased          Erased              UnKnown

 /dev/sdal               HDD       Erased          Erased              UnKnown

 /dev/sdam               HDD       Erased          Erased              UnKnown

 /dev/sdan               HDD       Erased          Erased              UnKnown

 /dev/sdao               SSD       Erased          Erased              UnKnown

 /dev/sdap               SSD       Erased          Erased              UnKnown

 /dev/sdaq               SSD       Erased          Erased              UnKnown

 /dev/sdar               SSD       Erased          Erased              UnKnown

 /dev/sdas               SSD       Erased          Erased              UnKnown

 /dev/sdat               SSD       Erased          Erased              UnKnown

 /dev/sdau               SSD       Erased          Erased              UnKnown

 /dev/sdav               SSD       Erased          Erased              UnKnown



First network

After the cleanup we can create the basic network. To do that (need to do in both nodes), just execute the script “/opt/oracle/dcs/bin/odacli” with parameter “configure-firstnet”. Look:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli configure-firstnet

Select the Interface to configure the network on (bond0 bond1) [bond0]:

Configure DHCP on bond0 (yes/no) [no]:

INFO: You have chosen Static configuration

Use VLAN on bond0 (yes/no) [no]:yes

Configure VLAN on bond0, input VLAN ID [2 - 4094] 2999

INFO: using network interface bond0.2999

Enter the IP address to configure : 200.200.67.100

Enter the Netmask address to configure : 255.255.255.0

Enter the Gateway address to configure[200.200.67.1] :

INFO: Restarting the network

Shutting down interface bond0:  [  OK  ]

Shutting down interface bond1:  [  OK  ]

Shutting down interface ib0:  [  OK  ]

Shutting down interface ib1:  [  OK  ]

Shutting down interface ibbond0:  [  OK  ]

Shutting down loopback interface:  [  OK  ]

Bringing up loopback interface:  [  OK  ]

Bringing up interface bond0:  [  OK  ]

Bringing up interface bond1:  [  OK  ]

Bringing up interface ibbond0:  Determining if ip address 192.168.16.24 is already in use for device ibbond0...

[  OK  ]

Bringing up interface bond0.2999:  Determining if ip address 200.200.67.100 is already in use for device bond0.2999...

[  OK  ]

INFO: Restarting the DCS agent

initdcsagent stop/waiting

initdcsagent start/running, process 23750

[root@oak1 ~]#

And in the second node:

 

[root@oak2 ~]# /opt/oracle/dcs/bin/odacli configure-firstnet

Select the Interface to configure the network on (bond0 bond1) [bond0]:

Configure DHCP on bond0 (yes/no) [no]:

INFO: You have chosen Static configuration

Use VLAN on bond0 (yes/no) [no]:yes

Configure VLAN on bond0, input VLAN ID [2 - 4094] 2999

INFO: using network interface bond0.2999

Enter the IP address to configure : 200.200.67.101

Enter the Netmask address to configure : 255.255.255.0

Enter the Gateway address to configure[200.200.67.1] :

INFO: Restarting the network

Shutting down interface bond0:  [  OK  ]

Shutting down interface bond1:  [  OK  ]

Shutting down interface ib0:  [  OK  ]

Shutting down interface ib1:  [  OK  ]

Shutting down interface ibbond0:  [  OK  ]

Shutting down loopback interface:  [  OK  ]

Bringing up loopback interface:  [  OK  ]

Bringing up interface bond0:  [  OK  ]

Bringing up interface bond1:  [  OK  ]

Bringing up interface ibbond0:  Determining if ip address 192.168.16.25 is already in use for device ibbond0...

[  OK  ]

Bringing up interface bond0.2999:  Determining if ip address 200.200.67.101 is already in use for device bond0.2999...

[  OK  ]

INFO: Restarting the DCS agent

initdcsagent stop/waiting

initdcsagent start/running, process 10775

[root@oak2 ~]#



The most important is to define correctly the vlan (if used, as in this case). You can rerun the command if needed. After that, you can exit the ILOM console and do ssh to the ODA node.

Upload GI and DB base version to the repository

After you configure the first network you need to upload the files that will fill the internal ODA repository and allow the “create appliance script” do initiate the ASM and base Oracle Home. There are some several steps, but you will execute in just one node.
Again, Oracle recently changed the ODA base page (https://support.oracle.com/epmos/faces/DocContentDisplay?id=888888.1) and removed the links to files with clone version for OH and GI. But the patch numbers are patch 27604593 for GI and patch 27604558 for RDBMS. Bot are for baremetal image, if you are using VM installation, they are others (but you can find then patch search inside Oracle MOS.
The first step is to upload the files “p27604558_183000_Linux-x86-64.zip”and “p27604593_183000_Linux-x86-64.zip” to the first node and unzip. Remember that the file name will change if you are reimage using another version. Here I used in “/tmp”

 

[root@oak1 ~]# mkdir /tmp/deploy

[root@oak1 ~]# cd /tmp/deploy

[root@oak1 deploy]# ###### EXECUTE THE SCP OR COPY FILES TO THE FOLDER########

[root@oak1 deploy]# unzip -qa p27604558_183000_Linux-x86-64.zip

[root@oak1 deploy]# unzip -qa p27604593_183000_Linux-x86-64.zip

replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

[root@oak1 deploy]#

[root@oak1 deploy]#

[root@oak1 deploy]# ls -l

total 20559868

-rw-r--r-- 1 root root 4322872013 Dec 18 08:24 odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip

-rwxr-xr-x 1 root root 6193475789 Dec 18 08:03 odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip

-rw-r--r-- 1 root root 4322873299 May 22 08:37 p27604558_183000_Linux-x86-64.zip

-rw-r--r-- 1 root root 6193477129 May 22 08:35 p27604593_183000_Linux-x86-64.zip

-rw-r--r-- 1 root root       2044 Dec 18 08:05 README.txt

[root@oak1 deploy]#


After that we can upload these file to the internal ODA repository:

 

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip

{

  "jobId" : "b66269aa-e007-4304-8074-7ba7e9c8c5c2",

  "status" : "Created",

  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 08:51:14 AM UTC",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 08:51:14 AM UTC"

}

[root@oak1 deploy]#


Look that one job will be created, and you can check the progress for the job with the “/opt/oracle/dcs/bin/odacli describe-job -i” passing as parameter the jobId:

 

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-4304-8074-7ba7e9c8c5c2

 /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-430




Job details

----------------------------------------------------------------

                     ID:  b66269aa-e007-4304-8074-7ba7e9c8c5c2

            Description:  Repository Update

                 Status:  Running

                Created:  May 22, 2019 8:51:14 AM UTC

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 8:51:14 AM UTC         May 22, 2019 8:51:14 AM UTC         Success

Setting up ssh equivalance               May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Success

Copy BundleFile                          May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Running




[root@oak1 deploy]#

[root@oak1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b66269aa-e007-4304-8074-7ba7e9c8c5c2




Job details

----------------------------------------------------------------

                     ID:  b66269aa-e007-4304-8074-7ba7e9c8c5c2

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 8:51:14 AM UTC

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-181205-GI-18.3.0.0.zip,/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-18.0.0.0.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 8:51:14 AM UTC         May 22, 2019 8:51:14 AM UTC         Success

Setting up ssh equivalance               May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:51:15 AM UTC         Success

Copy BundleFile                          May 22, 2019 8:51:15 AM UTC         May 22, 2019 8:52:21 AM UTC         Success

Validating CopiedFile                    May 22, 2019 8:52:24 AM UTC         May 22, 2019 8:52:48 AM UTC         Success

Unzip bundle                             May 22, 2019 8:52:48 AM UTC         May 22, 2019 8:54:17 AM UTC         Success

Unzip bundle                             May 22, 2019 8:54:17 AM UTC         May 22, 2019 8:55:47 AM UTC         Success

Delete PatchBundles                      May 22, 2019 8:55:47 AM UTC         May 22, 2019 8:55:48 AM UTC         Success

Removing ssh keys                        May 22, 2019 8:55:48 AM UTC         May 22, 2019 8:55:49 AM UTC         Success




[root@oak1 deploy]#


Create appliance

After you clean the disk headers, prepare the first network and upload the files in the repository you can create the appliance. This means that you will configure the operational system, groups, users, folders, GI, ASM, and Oracle database. There are two ways to create, one is using the web interface (this replace the old java app that creates the XML file), and the other is from CLI. Unfortunately, until I wrote this, if you use VLAN you can’t create the appliance with the web interface, just with CLI.
The CLI uses the script “/opt/oracle/dcs/bin/odacli create-appliance” that receive a JSON file as a parameter. There is not some much information the parameter for this JSON file, just the readme with some examples and one web page with other examples. But in both, no real good explanation to each option, look:
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxd/create-appliance-using-json-file.html#GUID-42250FD2-EA91-4457-9ED7-CA3A2A863B40

 

Fortunately, most of the options are self-explained, other parts I will explain below. First, the full JSON file that I used:

 

[root@oak1 ~]# cat odas.json

{

  "instance" : {

    "name" : "odas",

    "instanceBaseName" : "odas",

    "systemPassword" : null,

    "dbEdition" : "EE",

    "timeZone" : "Europe/Luxembourg",

    "ntpServers" : [ "200.200.13.125" ],

    "dnsServers" : [ "200.200.1.32", "200.200.1.33" ],

    "domainName" : "xxxx.xxxx.xxx",

    "isRoleSeparated" : true,

    "osUserGroup" : {

      "groups" : [ {

        "groupId" : 1001,

        "groupName" : "oinstall",

        "groupRole" : "oinstall"

      }, {

        "groupId" : 1003,

        "groupName" : "dbaoper",

        "groupRole" : "dbaoper"

      }, {

        "groupId" : 1002,

        "groupName" : "dba",

        "groupRole" : "dba"

      }, {

        "groupId" : 1006,

        "groupName" : "asmadmin",

        "groupRole" : "asmadmin"

      }, {

        "groupId" : 1005,

        "groupName" : "asmoper",

        "groupRole" : "asmoper"

      }, {

        "groupId" : 1004,

        "groupName" : "asmdba",

        "groupRole" : "asmdba"

      } ],

      "users" : [ {

        "userId" : 1001,

        "userName" : "oracle",

        "userRole" : "oracleUser"

      }, {

        "userId" : 1000,

        "userName" : "grid",

        "userRole" : "gridUser"

      } ]

    },

    "objectStoreCredentials" : null

  },

  "nodes" : [ {

    "nodeNumber" : "0",

    "nodeName" : "odas1",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.100",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  }, {

    "nodeNumber" : "1",

    "nodeName" : "odas2",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.101",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  } ],

  "grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 90

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 10

    }, {

      "diskGroupName" : "REDO",

      "redundancy" : "HIGH",

      "disks" : null,

      "diskPercentage" : null

    }, {

      "diskGroupName" : "FLASH",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : null

    } ],

    "scan" : {

      "scanName" : "odas-scan",

      "ipAddresses" : [ "200.200.67.104", "200.200.67.105" ]

    },

    "vip" : [ {

      "nodeNumber" : "0",

      "vipName" : "odas1-vip",

      "ipAddress" : "200.200.67.102"

    }, {

      "nodeNumber" : "1",

      "vipName" : "odas2-vip",

      "ipAddress" : "200.200.67.103"

    } ],

    "language" : "en",

    "enableAFD" : "TRUE",

    "gridVersion" : null

  },

  "database" : {

    "dbName" : "mfod",

    "databaseUniqueName" : "mfod_ora",

    "dbVersion" : "18.3.0.0.180717",

    "dbHomeId" : null,

    "instanceOnly" : false,

    "registerOnly" : null,

    "isCdb" : true,

    "pdBName" : "pdb1",

    "pdbAdminuserName" : "pdbuser",

    "enableTDE" : false,

    "adminPassword" : "WWeeeeee99##",

    "dbType" : "RAC",

    "dbTargetNodeNumber" : null,

    "dbClass" : "OLTP",

    "dbShape" : "odb1",

    "dbStorage" : "ASM",

    "dbRedundancy" : null,

    "dbCharacterSet" : {

      "characterSet" : "AL32UTF8",

      "nlsCharacterset" : "AL16UTF16",

      "dbTerritory" : "AMERICA",

      "dbLanguage" : "AMERICAN"

    },

    "dbConsoleEnable" : false,

    "backupConfigId" : null,

    "backupConfigName" : null,

    "dbOnFlashStorage" : true,

    "dbEdition" : "EE",

    "rmanBkupPassword" : null,

    "dbDomainName" : null,

    "enableFlashCache" : null,

    "level0BackupDay" : null

  },

  "asr" : null

}

[root@oak1 ~]#


The file starts with a basic config where you define the name parameter and the instanceBaseName that will define some basic internal identification parameters. One example is the time zone.

The first that needs attention is isRoleSeparated and osUserGroup. If you choose the first as false, you don’t need the specify grid users. Otherwise, is needed to specify it.

For nodes option, you specify the hostname, ips, and gateway. Be careful with the nickname that is the same name that you configured the first network. Some steps above. The ilom parameter, since I am not changing now (and I think that in reimage scenario you do not change at this moment), remains null.

The grid parameter defines the disk division. You can choose arbitrary values for diskPercentage and disks that you will select for each one if needed. For disk, left undefined that ODA will divide automatically and accordingly. The key point here is that even, by default, FLASH and REDO diskgroups are created with default values (check that I even defined disk percentage for each one), if you forgot to specify, they are not created and the disk will not be used. I already wrote about one problem in this particular part and you can read here: http://www.fernandosimon.com/blog/oda-json-and-flash/

The database parameter defines the information for database creation. You have the basic like database name, unique name, cdb (or no), pdb name. The most tricky here are dbVersion (that need to be the same from that you uploaded in the repository), adminPassword (that will be used for root password after create the appliance), dbShape (that is the same – you choose one – for ODA available shapes), and dbStorage (that you choose between ASM and ACFS, 11 just support ACFS). Again, for ASM and ACFS I already wrote about that here: http://www.fernandosimon.com/blog/oda-acfs-and-asm-dilemma/

Important hint, if you want to create a database in another version that 18c you need to upload the files/clone images in the repository before.

To create just execute:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli create-appliance -r odas.json

{

  "instance" : {

    "name" : "odas",

    "instanceBaseName" : "odas",

    "systemPassword" : null,

    "dbEdition" : "EE",

    "timeZone" : "Europe/Luxembourg",

    "ntpServers" : [ "200.200.13.125" ],

    "dnsServers" : [ "200.200.1.32", "200.200.1.33" ],

    "domainName" : "xxxx.xxxx.xxx",

    "isRoleSeparated" : true,

    "osUserGroup" : {

      "groups" : [ {

        "groupId" : 1001,

        "groupName" : "oinstall",

        "groupRole" : "oinstall"

      }, {

        "groupId" : 1003,

        "groupName" : "dbaoper",

        "groupRole" : "dbaoper"

      }, {

        "groupId" : 1002,

        "groupName" : "dba",

        "groupRole" : "dba"

      }, {

        "groupId" : 1006,

        "groupName" : "asmadmin",

        "groupRole" : "asmadmin"

      }, {

        "groupId" : 1005,

        "groupName" : "asmoper",

        "groupRole" : "asmoper"

      }, {

        "groupId" : 1004,

        "groupName" : "asmdba",

        "groupRole" : "asmdba"

      } ],

      "users" : [ {

        "userId" : 1001,

        "userName" : "oracle",

        "userRole" : "oracleUser"

      }, {

        "userId" : 1000,

        "userName" : "grid",

        "userRole" : "gridUser"

      } ]

    },

    "objectStoreCredentials" : null

  },

  "nodes" : [ {

    "nodeNumber" : "0",

    "nodeName" : "odas1",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.100",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  }, {

    "nodeNumber" : "1",

    "nodeName" : "odas2",

    "localDisk" : null,

    "network" : [ {

      "name" : null,

      "nicName" : "bond0.2999",

      "vlanId" : null,

      "ipAddress" : "200.200.67.101",

      "subNetMask" : "255.255.255.0",

      "gateway" : "200.200.67.1",

      "interfaceType" : null,

      "networkType" : [ "Public" ],

      "isDefaultNetwork" : true

    } ],

    "ilom" : null

  } ],

  "grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 90

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 10

    }, {

      "diskGroupName" : "REDO",

      "redundancy" : "HIGH",

      "disks" : null,

      "diskPercentage" : null

    }, {

      "diskGroupName" : "FLASH",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : null

    } ],

    "scan" : {

      "scanName" : "odas-scan",

      "ipAddresses" : [ "200.200.67.104", "200.200.67.105" ]

    },

    "vip" : [ {

      "nodeNumber" : "0",

      "vipName" : "odas1-vip",

      "ipAddress" : "200.200.67.102"

    }, {

      "nodeNumber" : "1",

      "vipName" : "odas2-vip",

      "ipAddress" : "200.200.67.103"

    } ],

    "language" : "en",

    "enableAFD" : "TRUE",

    "gridVersion" : null

  },

  "database" : {

    "dbName" : "mfod",

    "databaseUniqueName" : "mfod_ora",

    "dbVersion" : "18.3.0.0.180717",

    "dbHomeId" : null,

    "instanceOnly" : false,

    "registerOnly" : null,

    "isCdb" : true,

    "pdBName" : "pdb1",

    "pdbAdminuserName" : "pdbuser",

    "enableTDE" : false,

    "adminPassword" : "WWeeeeee99##",

    "dbType" : "RAC",

    "dbTargetNodeNumber" : null,

    "dbClass" : "OLTP",

    "dbShape" : "odb1",

    "dbStorage" : "ASM",

    "dbRedundancy" : null,

    "dbCharacterSet" : {

      "characterSet" : "AL32UTF8",

      "nlsCharacterset" : "AL16UTF16",

      "dbTerritory" : "AMERICA",

      "dbLanguage" : "AMERICAN"

    },

    "dbConsoleEnable" : false,

    "backupConfigId" : null,

    "backupConfigName" : null,

    "dbOnFlashStorage" : true,

    "dbEdition" : "EE",

    "rmanBkupPassword" : null,

    "dbDomainName" : null,

    "enableFlashCache" : null,

    "level0BackupDay" : null

  },

  "asr" : null

}

{

  "jobId" : "f07961fa-4192-489b-9a55-53165a012376",

  "status" : "Created",

  "message" : null,

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 09:03:36 AM UTC",

  "resourceList" : [ ],

  "description" : "Provisioning service creation",

  "updatedTime" : "May 22, 2019 09:03:36 AM UTC"

}

[root@oak1 ~]#


Look that you will receive one jobId if everything is fine. To follow just describe the job:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i f07961fa-4192-489b-9a55-53165a012376




Job details

----------------------------------------------------------------

                     ID:  f07961fa-4192-489b-9a55-53165a012376

            Description:  Provisioning service creation

                 Status:  Running

                Created:  May 22, 2019 11:03:36 AM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

networks updation                        May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

updating network                         May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:38 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

networks updation                        May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

updating network                         May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:43 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmdba'creation            May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmadmin'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dba'creation               May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dbaoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'oinstall'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS user 'grid'creation                   May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

OS user 'oracle'creation                 May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

SSH equivalance setup                    May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

Grid home creation                       May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Running

Creating GI home directories             May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Success

Cloning Gi home                          May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Running




[root@oak1 ~]#



And after some time:
 
[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i f07961fa-4192-489b-9a55-53165a012376




Job details

----------------------------------------------------------------

                     ID:  f07961fa-4192-489b-9a55-53165a012376

            Description:  Provisioning service creation

                 Status:  Success

                Created:  May 22, 2019 11:03:36 AM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

networks updation                        May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

updating network                         May 22, 2019 11:03:37 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:38 AM CEST       May 22, 2019 11:03:42 AM CEST       Success

networks updation                        May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

updating network                         May 22, 2019 11:03:42 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

Setting up Vlan                          May 22, 2019 11:03:43 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmdba'creation            May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'asmadmin'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dba'creation               May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'dbaoper'creation           May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS usergroup 'oinstall'creation          May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:47 AM CEST       Success

OS user 'grid'creation                   May 22, 2019 11:03:47 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

OS user 'oracle'creation                 May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

SSH equivalance setup                    May 22, 2019 11:03:48 AM CEST       May 22, 2019 11:03:48 AM CEST       Success

Grid home creation                       May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:08:00 AM CEST       Success

Creating GI home directories             May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:03:49 AM CEST       Success

Cloning Gi home                          May 22, 2019 11:03:49 AM CEST       May 22, 2019 11:06:36 AM CEST       Success

Cloning Gi home                          May 22, 2019 11:06:36 AM CEST       May 22, 2019 11:07:58 AM CEST       Success

Updating GiHome version                  May 22, 2019 11:07:58 AM CEST       May 22, 2019 11:08:00 AM CEST       Success

Updating GiHome version                  May 22, 2019 11:07:58 AM CEST       May 22, 2019 11:08:00 AM CEST       Success

Storage discovery                        May 22, 2019 11:08:00 AM CEST       May 22, 2019 11:19:03 AM CEST       Success

Grid stack creation                      May 22, 2019 11:19:03 AM CEST       May 22, 2019 11:53:45 AM CEST       Success

Configuring GI                           May 22, 2019 11:19:03 AM CEST       May 22, 2019 11:20:08 AM CEST       Success

Running GI root scripts                  May 22, 2019 11:20:08 AM CEST       May 22, 2019 11:32:18 AM CEST       Success

Running GI config assistants             May 22, 2019 11:37:22 AM CEST       May 22, 2019 11:40:04 AM CEST       Success

Setting AUDIT SYSLOG LEVEL               May 22, 2019 11:50:06 AM CEST       May 22, 2019 11:50:06 AM CEST       Success

Post cluster OAKD configuration          May 22, 2019 11:53:45 AM CEST       May 22, 2019 11:59:11 AM CEST       Success

Disk group 'RECO'creation                May 22, 2019 11:59:20 AM CEST       May 22, 2019 11:59:38 AM CEST       Success

Disk group 'REDO'creation                May 22, 2019 11:59:38 AM CEST       May 22, 2019 11:59:47 AM CEST       Success

Disk group 'FLASH'creation               May 22, 2019 11:59:47 AM CEST       May 22, 2019 11:59:58 AM CEST       Success

Volume 'commonstore'creation             May 22, 2019 11:59:58 AM CEST       May 22, 2019 12:00:48 PM CEST       Success

ACFS File system 'DATA'creation          May 22, 2019 12:00:48 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Database home creation                   May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:05:28 PM CEST       Success

Validating dbHome available space        May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Validating dbHome available space        May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Creating DbHome Directory                May 22, 2019 12:01:13 PM CEST       May 22, 2019 12:01:13 PM CEST       Success

Extract DB clones                        May 22, 2019 12:01:14 PM CEST       May 22, 2019 12:04:12 PM CEST       Success

Clone Db home                            May 22, 2019 12:04:12 PM CEST       May 22, 2019 12:05:11 PM CEST       Success

Enable DB options                        May 22, 2019 12:05:11 PM CEST       May 22, 2019 12:05:27 PM CEST       Success

Run Root DB scripts                      May 22, 2019 12:05:27 PM CEST       May 22, 2019 12:05:27 PM CEST       Success

Provisioning service creation            May 22, 2019 12:05:28 PM CEST       May 22, 2019 12:23:12 PM CEST       Success

Database Creation                        May 22, 2019 12:05:28 PM CEST       May 22, 2019 12:20:27 PM CEST       Success

Change permission for xdb wallet files   May 22, 2019 12:20:27 PM CEST       May 22, 2019 12:20:27 PM CEST       Success

Add Startup Trigger to Open all PDBS     May 22, 2019 12:20:27 PM CEST       May 22, 2019 12:20:28 PM CEST       Success

SqlPatch upgrade                         May 22, 2019 12:21:53 PM CEST       May 22, 2019 12:23:08 PM CEST       Success

Running dbms_stats init_package          May 22, 2019 12:23:08 PM CEST       May 22, 2019 12:23:11 PM CEST       Success

updating the Database version            May 22, 2019 12:23:11 PM CEST       May 22, 2019 12:23:12 PM CEST       Success

users tablespace creation                May 22, 2019 12:23:12 PM CEST       May 22, 2019 12:23:15 PM CEST       Success

Install TFA                              May 22, 2019 12:23:16 PM CEST       May 22, 2019 12:28:08 PM CEST       Success




[root@oak1 ~]#


You can follow the log too if you want to see more detailed (a lot) info. To do just go to directory /opt/oracle/dcs/log/ and check the file dcs-agent.log. Everything will be there, and it is really importing if some step fails. Here you can see one example of the output.
After the end of the job you can check the details for the appliance:

 

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-system




Appliance Information

----------------------------------------------------------------

                     ID: 6ce235cf-effc-4748-9d0f-ac246e9ee819

               Platform: X5-2-HA

        Data Disk Count: 24

         CPU Core Count: 36

                Created: May 22, 2019 11:03:36 AM CEST




System Information

----------------------------------------------------------------

                   Name: odas

            Domain Name: xxxx.xxxx.xxx

              Time Zone: Europe/Luxembourg

             DB Edition: EE

            DNS Servers: 200.200.1.32 200.200.1.33

            NTP Servers: 200.200.13.125




Disk Group Information

----------------------------------------------------------------

DG Name                   Redundancy                Percentage

------------------------- ------------------------- ------------

Data                      Normal                    90

Reco                      Normal                    10

Redo                      High                      100

Flash                     Normal                    100




[root@oak1 ~]#

[root@oak1 ~]# odaadmcli show diskgroup

DiskGroups

----------

DATA

FLASH

RECO

REDO

[root@oak1 ~]#

[root@oak1 ~]# su - grid

[grid@odas1 ~]$ asmcmd

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  108019712  108007188          6751232        50627978              0             Y  DATA/

MOUNTED  NORMAL  N         512             512   4096  4194304    1525760    1518672           381440          568616              0             N  FLASH/

MOUNTED  NORMAL  N         512             512   4096  4194304   11997184   11995736           749824         5622956              0             N  RECO/

MOUNTED  HIGH    N         512             512   4096  4194304     762880     749908           190720          186396              0             N  REDO/

ASMCMD> exit

[grid@odas1 ~]$

[grid@odas1 ~]$ logout

[root@oak1 ~]#

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli list-dbstorages




ID                                       Type   DBUnique Name        Status

---------------------------------------- ------ -------------------- ----------

d26bf34f-9379-4872-839b-ae0af6926aae     Asm    mfod_ora             Configured

[root@oak1 ~]#

[root@oak1 ~]# /opt/oracle/dcs/bin/odacli describe-dbstorage -i d26bf34f-9379-4872-839b-ae0af6926aae

Database Storage details

----------------------------------------------------------------

                     ID: d26bf34f-9379-4872-839b-ae0af6926aae

                DB Name: mfod

          DBUnique Name: mfod_ora

         DB Resource ID: 938b616a-99ed-4466-8d76-820d3489517d

           Storage Type: ASM

                   DATA:

                         Location: +FLASH/mfod_ora

                       Used Space: 3.21GB

                       Free Space: 741.53GB

                   REDO:

                         Location: +REDO/mfod_ora

                       Used Space: 4.06GB

                       Free Space: 244.11GB

                   RECO:

                         Location: +RECO/mfod_ora

                       Used Space: 334.0MB

                       Free Space: 5.72TB

                  State: ResourceState(status=Configured)

                Created: May 22, 2019 11:03:36 AM CEST

            UpdatedTime: May 22, 2019 12:01:13 PM CEST

[root@oak1 ~]#


After that you reboot both nodes to have a fresh start with all the configuration up and running:

 

[root@oak1 ~]# reboot




Broadcast message from root@odas1

        (/dev/ttyS0) at 12:37 ...




The system is going down for reboot NOW!

[root@oak1 ~]#

       

Oracle Linux Server release 6.10

Kernel 4.1.12-124.18.6.el6uek.x86_64 on an x86_64




odas1 login: root

Password:

Last login: Wed May 22 11:10:13 from odap1.xxxx.xxxx.xxx

[root@odas1 ~]#


As an example, you can add vlan to ODA if needed (after the reboot):

 

[root@odas1 ~]# odaadmcli show vlan

        NAME                     ID    INTERFACE   CONFIG_TYPE IP_ADDRESS      NETMASK         GATEWAY         NODENUM

        Public-network           2999  bond0       Public      200.200.67.100   255.255.255.0   200.200.67.1     0

        Public-network           2999  bond0       Public      200.200.67.101   255.255.255.0   200.200.67.1     1

[root@odas1 ~]#

[root@odas1 ~]# cat /proc/net/vlan/config

VLAN Dev name    | VLAN ID

Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD

bond0.2999     | 2999  | bond0

[root@odas1 ~]#

[root@odas1 ~]# odaadmcli create vlan VLAN_3001 -vlanid 3001 -if bond0 -node 0 -setuptype database -ip 200.200.131.58 -netmask 255.255.255.0 -gateway 200.200.131.1




Created Vlan : VLAN_3001

[root@odas1 ~]# odaadmcli create vlan VLAN_3010 -vlanid 3010 -if bond0 -node 0 -setuptype database -ip 200.190.3.58 -netmask 255.255.255.0 -gateway 200.190.3.1




Created Vlan : VLAN_3010

[root@odas1 ~]# odaadmcli create vlan VLAN_3030 -vlanid 3030 -if bond0 -node 0 -setuptype management -ip 200.200.192.156 -netmask 255.255.255.0 -gateway 200.200.192.1




Created Vlan : VLAN_3030

[root@odas1 ~]# odaadmcli create vlan VLAN_3002 -vlanid 3002 -if bond1 -node 0 -setuptype backup -ip 200.200.68.128 -netmask 255.255.255.0 -gateway 200.200.68.1




Created Vlan : VLAN_3002

[root@odas1 ~]# odaadmcli create vlan VLAN_3001 -vlanid 3001 -if bond0 -node 1 -setuptype database -ip 200.200.131.59 -netmask 255.255.255.0 -gateway 200.200.131.1




Created Vlan : VLAN_3001

[root@odas1 ~]# odaadmcli create vlan VLAN_3010 -vlanid 3010 -if bond0 -node 1 -setuptype database -ip 200.190.3.59 -netmask 255.255.255.0 -gateway 200.190.3.1




Created Vlan : VLAN_3010

[root@odas1 ~]# odaadmcli create vlan VLAN_3030 -vlanid 3030 -if bond0 -node 1 -setuptype management -ip 200.200.192.157 -netmask 255.255.255.0 -gateway 200.200.192.1




Created Vlan : VLAN_3030

[root@odas1 ~]# odaadmcli create vlan VLAN_3002 -vlanid 3002 -if bond1 -node 1 -setuptype backup -ip 200.200.68.129 -netmask 255.255.255.0 -gateway 200.200.68.1




Created Vlan : VLAN_3002

[root@odas1 ~]#

[root@odas1 ~]#

[root@odas1 ~]# odaadmcli show vlan

        NAME                     ID    INTERFACE   CONFIG_TYPE IP_ADDRESS      NETMASK         GATEWAY         NODENUM

        Public-network           2999  bond0       Public      200.200.67.100   255.255.255.0   200.200.67.1     0

        Public-network           2999  bond0       Public      200.200.67.101   255.255.255.0   200.200.67.1     1

        VLAN_3002                3002  bond1       backup      200.200.68.128   255.255.255.0   200.200.68.1     0

        VLAN_3002                3002  bond1       backup      200.200.68.129   255.255.255.0   200.200.68.1     1

        VLAN_3001                3001  bond0       database    200.200.131.58   255.255.255.0   200.200.131.1    0

        VLAN_3001                3001  bond0       database    200.200.131.59   255.255.255.0   200.200.131.1    1

        VLAN_3010                3010  bond0       database    200.190.3.58     255.255.255.0   200.190.3.1      0

        VLAN_3010                3010  bond0       database    200.190.3.59     255.255.255.0   200.190.3.1      1

        VLAN_3030                3030  bond0       management  200.200.192.156  255.255.255.0   200.200.192.1    0

        VLAN_3030                3030  bond0       management  200.200.192.157  255.255.255.0   200.200.192.1    1

[root@odas1 ~]#


Firmware and patch

As described in the documentation, the reimage process does not update firmware and we need to this after creating the appliance. This point ins controversy, because there is no information when need to do this, you can run before or after creating the appliance. Here I made after the create, but if you are in really old version and have a lot of old firmware running maybe you can run before the create to avoid issues.
The funny part is that the firmware’s are inside of one the default patch file. Again, you need to search in patch search page because Oracle removed from 88888.1 page these links (look the path number in the example below). So, the procedure to update firmware is the same to apply a patch in ODA and need to download the same patch version. As usual, the first step is to upload the files:

 

[root@odas1 ~]# cd /tmp/deploy/

[root@odas1 deploy]#

[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_1of3.zip

[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_2of3.zip

[root@odas1 deploy]# unzip -qa p28864490_183000_Linux-x86-64_3of3.zip

[root@odas1 deploy]#

[root@odas1 deploy]# rm -rf p*zip

[root@odas1 deploy]#

[root@odas1 deploy]# ls -l

total 15441928

-rw-r--r-- 1 root root 2767978208 Dec 13 14:50 oda-sm-18.3.0.0.0-181205-server1of3.zip

-rwxr-xr-x 1 root root 6835610213 Dec 13 15:08 oda-sm-18.3.0.0.0-181205-server2of3.zip

-rwxr-xr-x 1 root root 6193475789 Dec 13 15:22 oda-sm-18.3.0.0.0-181205-server3of3.zip

-rw-r--r-- 1 root root       1155 Dec 13 15:22 README.txt

[root@odas1 deploy]#

[root@odas1 deploy]#


If you want to check what are you running, just execute odacli describe-component (look ilom version as an example):

 

[root@odas1 ~]# odacli describe-component

System Version

---------------

18.3.0.0.0




System node Name

---------------

odas1




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      3.2.9.23.r116695      4.0.2.26.b.r125868




BIOS                                      30110000              30130500




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}




System node Name

---------------

odas2




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      3.2.9.23.r116695      4.0.2.26.b.r125868




BIOS                                      30110000              30130500




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}




root@odas1 ~]#

 

 

After that you can upload the files in the repository (look the path to the files):

 

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip

{

  "jobId" : "cdfd2e72-777b-4ccb-a670-17b88c7cc102",

  "status" : "Created",

  "message" : "/tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 15:01:03 PM CEST",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 15:01:03 PM CEST"

}

[root@odas1 deploy]#

[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i cdfd2e72-777b-4ccb-a670-17b88c7cc102




Job details

----------------------------------------------------------------

                     ID:  cdfd2e72-777b-4ccb-a670-17b88c7cc102

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 3:01:03 PM CEST

                Message:  /tmp/deploy/oda-sm-18.3.0.0.0-181205-server1of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server2of3.zip,/tmp/deploy/oda-sm-18.3.0.0.0-181205-server3of3.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:01:03 PM CEST        Success

Setting up ssh equivalance               May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:01:03 PM CEST        Success

Copy BundleFile                          May 22, 2019 3:01:03 PM CEST        May 22, 2019 3:02:47 PM CEST        Success

Validating CopiedFile                    May 22, 2019 3:02:49 PM CEST        May 22, 2019 3:03:23 PM CEST        Success

Unzip bundle                             May 22, 2019 3:03:23 PM CEST        May 22, 2019 3:05:43 PM CEST        Success

Unzip bundle                             May 22, 2019 3:05:43 PM CEST        May 22, 2019 3:08:08 PM CEST        Success

Delete PatchBundles                      May 22, 2019 3:08:08 PM CEST        May 22, 2019 3:08:11 PM CEST        Success

Removing ssh keys                        May 22, 2019 3:08:11 PM CEST        May 22, 2019 3:08:26 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#

 

After the success to upload you first update the storage using odacli update-storage command to upload to desired version (the version are in the filename or check what you downloaded from MOS):

 

[root@odas1 ~]# odacli update-storage -v 18.3.0.0.0 --rolling

{

  "jobId" : "18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd",

  "status" : "Created",

  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 15:25:24 PM CEST",

  "resourceList" : [ ],

  "description" : "Storage Firmware Patching",

  "updatedTime" : "May 22, 2019 15:25:24 PM CEST"

}

[root@odas1 ~]#

[root@odas1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i 18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd




Job details

----------------------------------------------------------------

                     ID:  18d1e5f8-7c31-4ce7-8241-f7b2ed5673cd

            Description:  Storage Firmware Patching

                 Status:  Success

                Created:  May 22, 2019 3:25:24 PM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Applying Firmware Disk Patches           May 22, 2019 3:25:24 PM CEST        May 22, 2019 3:25:45 PM CEST        Success

Applying Firmware Controller Patches     May 22, 2019 3:25:45 PM CEST        May 22, 2019 3:25:58 PM CEST        Success




[root@odas1 ~]#
After you can update the server (running odacli update-server). Here, since you are “updating” to the same version that you are running, the steps related to yum does not execute and just firmware part runs:

 

[root@odas1 ~]# odacli update-server -v 18.3.0.0.0

{

  "jobId" : "8bc7cc15-a543-4e26-ade4-8dd578ea67ba",

  "status" : "Created",

  "message" : "Success of Server Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 15:29:15 PM CEST",

  "resourceList" : [ ],

  "description" : "Server Patching",

  "updatedTime" : "May 22, 2019 15:29:15 PM CEST"

}

[root@odas1 ~]#

[root@odas1 ~]# /opt/oracle/dcs/bin/odacli describe-job -i 8bc7cc15-a543-4e26-ade4-8dd578ea67ba




Job details

----------------------------------------------------------------

                     ID:  8bc7cc15-a543-4e26-ade4-8dd578ea67ba

            Description:  Server Patching

                 Status:  Running

                Created:  May 22, 2019 3:29:15 PM CEST

                Message:




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Patch location validation                May 22, 2019 3:29:16 PM CEST        May 22, 2019 3:29:16 PM CEST        Success

Patch location validation                May 22, 2019 3:29:16 PM CEST        May 22, 2019 3:29:16 PM CEST        Success

dcs-controller upgrade                   May 22, 2019 3:29:19 PM CEST        May 22, 2019 3:29:19 PM CEST        Success

dcs-controller upgrade                   May 22, 2019 3:29:19 PM CEST        May 22, 2019 3:29:20 PM CEST        Success

Patch location validation                May 22, 2019 3:29:20 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

Patch location validation                May 22, 2019 3:29:20 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

dcs-cli upgrade                          May 22, 2019 3:29:21 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

dcs-cli upgrade                          May 22, 2019 3:29:21 PM CEST        May 22, 2019 3:29:21 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:29 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:36 PM CEST        May 22, 2019 3:29:36 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success

Updating YumPluginVersionLock rpm        May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:29:37 PM CEST        Success

Applying OS Patches                      May 22, 2019 3:29:37 PM CEST        May 22, 2019 3:31:38 PM CEST        Success

Creating repositories using yum          May 22, 2019 3:31:38 PM CEST        May 22, 2019 3:31:39 PM CEST        Success

Applying HMP Patches                     May 22, 2019 3:31:39 PM CEST        May 22, 2019 3:31:39 PM CEST        Success

Patch location validation                May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:40 PM CEST        Success

Patch location validation                May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:40 PM CEST        Success

oda-hw-mgmt upgrade                      May 22, 2019 3:31:40 PM CEST        May 22, 2019 3:31:41 PM CEST        Success

oda-hw-mgmt upgrade                      May 22, 2019 3:31:41 PM CEST        May 22, 2019 3:31:41 PM CEST        Success

OSS Patching                             May 22, 2019 3:31:41 PM CEST        May 22, 2019 3:31:43 PM CEST        Success

Applying Firmware Disk Patches           May 22, 2019 3:32:02 PM CEST        May 22, 2019 3:32:15 PM CEST        Success

Applying Firmware Expander Patches       May 22, 2019 3:32:27 PM CEST        May 22, 2019 3:32:34 PM CEST        Success

Applying Firmware Controller Patches     May 22, 2019 3:32:47 PM CEST        May 22, 2019 3:32:55 PM CEST        Success

Checking Ilom patch Version              May 22, 2019 3:32:56 PM CEST        May 22, 2019 3:32:59 PM CEST        Success

Checking Ilom patch Version              May 22, 2019 3:32:59 PM CEST        May 22, 2019 3:33:01 PM CEST        Success

Patch location validation                May 22, 2019 3:33:01 PM CEST        May 22, 2019 3:33:02 PM CEST        Success

Patch location validation                May 22, 2019 3:33:01 PM CEST        May 22, 2019 3:33:02 PM CEST        Success

Save password in Wallet                  May 22, 2019 3:33:03 PM CEST        May 22, 2019 3:33:04 PM CEST        Success

Apply Ilom patch                         May 22, 2019 3:33:04 PM CEST        May 22, 2019 3:45:07 PM CEST        Success

Apply Ilom patch                         May 22, 2019 3:45:07 PM CEST        May 22, 2019 3:56:45 PM CEST        Success

Copying Flash Bios to Temp location      May 22, 2019 3:56:45 PM CEST        May 22, 2019 3:56:45 PM CEST        Success

Copying Flash Bios to Temp location      May 22, 2019 3:56:45 PM CEST        May 22, 2019 3:56:45 PM CEST        Success

Starting the clusterware                 May 22, 2019 3:58:41 PM CEST        May 22, 2019 4:00:09 PM CEST        Success

clusterware patch verification           May 22, 2019 4:00:09 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

clusterware patch verification           May 22, 2019 4:00:09 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

Patch location validation                May 22, 2019 4:00:10 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

Patch location validation                May 22, 2019 4:00:10 PM CEST        May 22, 2019 4:00:10 PM CEST        Success

Opatch updation                          May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Opatch updation                          May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Patch conflict check                     May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Patch conflict check                     May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

clusterware upgrade                      May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

clusterware upgrade                      May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:11 PM CEST        Success

Updating GiHome version                  May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:12 PM CEST        Success

Updating GiHome version                  May 22, 2019 4:00:11 PM CEST        May 22, 2019 4:00:12 PM CEST        Success

Update System version                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Success

Update System version                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Success

preRebootNode Actions                    May 22, 2019 4:00:33 PM CEST        May 22, 2019 4:00:33 PM CEST        Running




[root@odas1 ~]#

 

 

After, reboot both nodes!
As example check that ilom was updated:

 

[root@odas1 ~]# odacli describe-component

System Version

---------------

18.3.0.0.0




System node Name

---------------

odas1




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      4.0.2.26.b.r125868    up-to-date




BIOS                                      30130500              up-to-date




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}




System node Name

---------------

odas2




Local System Version

---------------

18.3.0.0.0




Component                                Installed Version    Available Version

---------------------------------------- -------------------- --------------------

OAK                                       18.3.0.0.0            up-to-date




GI                                        18.3.0.0.180717       up-to-date




DB                                        18.3.0.0.180717       up-to-date




DCSAGENT                                  18.3.0.0.0            up-to-date




ILOM                                      4.0.2.26.b.r125868    up-to-date




BIOS                                      30130500              up-to-date




OS                                        6.10                  up-to-date




FIRMWARECONTROLLER {

[ c0 ]                                    4.650.00-7176         up-to-date

[ c1,c2 ]                                 13.00.00.00           up-to-date

}




FIRMWAREEXPANDER                          0018                  up-to-date




FIRMWAREDISK {

[ c0d0,c0d1 ]                             A7E0                  up-to-date

[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     PAG1                  up-to-date

c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,

c1d14,c1d15,c2d0,c2d1,c2d2,c2d3,c2d4,

c2d5,c2d6,c2d7,c2d8,c2d9,c2d10,c2d11,

c2d12,c2d13,c2d14,c2d15 ]

[ c1d16,c1d17,c1d18,c1d19,c1d20,c1d21,    A29A                  up-to-date

c1d22,c1d23,c2d16,c2d17,c2d18,c2d19,

c2d20,c2d21,c2d22,c2d23 ]

}







[root@odas1 ~]#

[root@odas1 ~]#
 

Create Oracle Homes and the databases

After update all the firmware’s you can upload the clone files for Oracle Homes versions that you want and create it. The procedure is the same than before, upload the files to (and just) ODA node 1 in somewhere and after that upload the repository and create the homes.
Upload the files and update the repository. Look below that I made one for each version:

 

[root@odas1 ~]# cd /tmp/deploy/

[root@odas1 deploy]#

[root@odas1 deploy]# unzip -qa p23494992_183000_Linux-x86-64.zip

[root@odas1 deploy]# rm p23494992_183000_Linux-x86-64.zip

rm: remove regular file `p23494992_183000_Linux-x86-64.zip'? y

[root@odas1 deploy]#

[root@odas1 deploy]# unzip -qa p23494997_183000_Linux-x86-64.zip

replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

[root@odas1 deploy]# rm p23494997_183000_Linux-x86-64.zip

rm: remove regular file `p23494997_183000_Linux-x86-64.zip'? y

[root@odas1 deploy]#

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip

{

  "jobId" : "b88c4922-1107-41f5-b7cd-e1648093051b",

  "status" : "Created",

  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 16:34:28 PM CEST",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 16:34:28 PM CEST"

}

[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i b88c4922-1107-41f5-b7cd-e1648093051b




Job details

----------------------------------------------------------------

                     ID:  b88c4922-1107-41f5-b7cd-e1648093051b

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 4:34:28 PM CEST

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-12.1.0.2.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 4:34:28 PM CEST        May 22, 2019 4:34:28 PM CEST        Success

Setting up ssh equivalance               May 22, 2019 4:34:28 PM CEST        May 22, 2019 4:34:29 PM CEST        Success

Copy BundleFile                          May 22, 2019 4:34:29 PM CEST        May 22, 2019 4:34:54 PM CEST        Success

Validating CopiedFile                    May 22, 2019 4:34:54 PM CEST        May 22, 2019 4:35:07 PM CEST        Success

Unzip bundle                             May 22, 2019 4:35:07 PM CEST        May 22, 2019 4:35:42 PM CEST        Success

Unzip bundle                             May 22, 2019 4:35:46 PM CEST        May 22, 2019 4:36:23 PM CEST        Success

Delete PatchBundles                      May 22, 2019 4:36:24 PM CEST        May 22, 2019 4:36:25 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:36:25 PM CEST        May 22, 2019 4:36:47 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]# odacli update-repository -f /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip

{

  "jobId" : "6698aa6d-1a44-4b1c-bda1-1e691a41a133",

  "status" : "Created",

  "message" : "/tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip",

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 16:38:53 PM CEST",

  "resourceList" : [ ],

  "description" : "Repository Update",

  "updatedTime" : "May 22, 2019 16:38:53 PM CEST"

}

[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i 6698aa6d-1a44-4b1c-bda1-1e691a41a133




Job details

----------------------------------------------------------------

                     ID:  6698aa6d-1a44-4b1c-bda1-1e691a41a133

            Description:  Repository Update

                 Status:  Success

                Created:  May 22, 2019 4:38:53 PM CEST

                Message:  /tmp/deploy/odacli-dcs-18.3.0.0.0-180905-DB-11.2.0.4.zip




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check AvailableSpace                     May 22, 2019 4:38:53 PM CEST        May 22, 2019 4:38:53 PM CEST        Success

Setting up ssh equivalance               May 22, 2019 4:38:53 PM CEST        May 22, 2019 4:38:53 PM CEST        Success

Copy BundleFile                          May 22, 2019 4:38:54 PM CEST        May 22, 2019 4:39:04 PM CEST        Success

Validating CopiedFile                    May 22, 2019 4:39:06 PM CEST        May 22, 2019 4:39:14 PM CEST        Success

Unzip bundle                             May 22, 2019 4:39:14 PM CEST        May 22, 2019 4:39:32 PM CEST        Success

Unzip bundle                             May 22, 2019 4:39:32 PM CEST        May 22, 2019 4:39:54 PM CEST        Success

Delete PatchBundles                      May 22, 2019 4:39:54 PM CEST        May 22, 2019 4:39:55 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:39:55 PM CEST        May 22, 2019 4:39:55 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#

After that, you can create the Oracle home for each version using the command odacli create-dbhome. 12c:

 

[root@odas1 deploy]#

[root@odas1 deploy]# odacli create-dbhome -v 12.1.0.2.180717




Job details

----------------------------------------------------------------

                     ID:  5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707

            Description:  Database Home OraDB12102_home1 creation with version :12.1.0.2.180717

                 Status:  Created

                Created:  May 22, 2019 4:42:58 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------




[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i 5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707




Job details

----------------------------------------------------------------

                     ID:  5ddcdd49-2d1a-404d-b4b8-7ab79c5f9707

            Description:  Database Home OraDB12102_home1 creation with version :12.1.0.2.180717

                 Status:  Success

                Created:  May 22, 2019 4:42:58 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Setting up ssh equivalance               May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Creating DbHome Directory                May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:42:59 PM CEST        Success

Extract DB clones                        May 22, 2019 4:42:59 PM CEST        May 22, 2019 4:46:22 PM CEST        Success

Clone Db home                            May 22, 2019 4:46:22 PM CEST        May 22, 2019 4:47:32 PM CEST        Success

Enable DB options                        May 22, 2019 4:47:32 PM CEST        May 22, 2019 4:47:41 PM CEST        Success

Run Root DB scripts                      May 22, 2019 4:47:41 PM CEST        May 22, 2019 4:47:42 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:47:48 PM CEST        May 22, 2019 4:47:48 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#
And 11g:

 

[root@odas1 deploy]# odacli create-dbhome -v 11.2.0.4.180717




Job details

----------------------------------------------------------------

                     ID:  d4512833-3198-4384-b0d5-3588e3ec8cdb

            Description:  Database Home OraDB11204_home1 creation with version :11.2.0.4.180717

                 Status:  Created

                Created:  May 22, 2019 4:50:06 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------




[root@odas1 deploy]#

[root@odas1 deploy]# /opt/oracle/dcs/bin/odacli describe-job -i d4512833-3198-4384-b0d5-3588e3ec8cdb




Job details

----------------------------------------------------------------

                     ID:  d4512833-3198-4384-b0d5-3588e3ec8cdb

            Description:  Database Home OraDB11204_home1 creation with version :11.2.0.4.180717

                 Status:  Success

                Created:  May 22, 2019 4:50:06 PM CEST

                Message:  Create Database Home




Task Name                                Start Time                          End Time                            Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

Setting up ssh equivalance               May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Validating dbHome available space        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Creating DbHome Directory                May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:50:07 PM CEST        Success

Extract DB clones                        May 22, 2019 4:50:07 PM CEST        May 22, 2019 4:51:59 PM CEST        Success

Clone Db home                            May 22, 2019 4:52:00 PM CEST        May 22, 2019 4:53:06 PM CEST        Success

Enable DB options                        May 22, 2019 4:53:06 PM CEST        May 22, 2019 4:53:11 PM CEST        Success

Run Root DB scripts                      May 22, 2019 4:53:11 PM CEST        May 22, 2019 4:53:11 PM CEST        Success

Removing ssh keys                        May 22, 2019 4:53:16 PM CEST        May 22, 2019 4:53:17 PM CEST        Success




[root@odas1 deploy]#

[root@odas1 deploy]#


And after that you can use the odacli list-dbhomes to see the options for database creation:

 

ID                                       Name                 DB Version                               Home Location                                 Status

---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------

886021e5-9bbf-4a14-9b50-398ddd00bfd0     OraDB18000_home1     18.3.0.0.180717                          /u01/app/oracle/product/18.0.0.0/dbhome_1     Configured

d791001f-b812-49fd-94f5-b11c5f580a30     OraDB12102_home1     12.1.0.2.180717                          /u01/app/oracle/product/12.1.0.2/dbhome_1     Configured

d162ff30-dc0d-4c4b-810e-c8f513dc9f62     OraDB11204_home1     11.2.0.4.180717                          /u01/app/oracle/product/11.2.0.4/dbhome_1     Configured




[root@odas1 deploy]#


As an example, if you want to delete the database that was created during the creation of appliance just execute odacli delete-database:

 

[root@odas1 ~]# odacli delete-database --dbName mfod

{

  "jobId" : "69adb079-e5e4-4d64-8b20-4eee73c65491",

  "status" : "Running",

  "message" : null,

  "reports" : [ ],

  "createTimestamp" : "May 22, 2019 16:17:27 PM CEST",

  "resourceList" : [ ],

  "description" : "Database service deletion with db name: mfod with id : 938b616a-99ed-4466-8d76-820d3489517d",

  "updatedTime" : "May 22, 2019 16:17:27 PM CEST"

}

[root@odas1 ~]#



Finish and clean the install

After all the steps listed until now, you have your ODA reimaged, updated, and with 18c, 12c and 11g available version for database creation. To finish you can update some files as needed. Examples of important files are /etc/fstab, /etc/sysctl.conf, /etc/sysconfig/network-scripts/route* and /etc/sysconfig/network-scripts/rules* (both related with routing tables for vlans – if needed), and add the listener for all the vlans that you created (if needed of course).
To finish, reboot both nodes and change the root password.
 
Reference that can be consulted to help you to understand the reimage and deployment procedure:
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/oracle-database-appliance-x7-2-deployment-and-users-guide.pdf         
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxd/oracle-database-appliance-x5-2-x4-2-x3-2-deployment-and-users-guide.pdf
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/odacli-comparison-chart.html#GUID-567E862F-F4A4-4E1E-A485-BD1945BEC673
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/postinstallation-tasks-oda.html#GUID-A256DFBD-9335-41D2-9BD5-A1BA6AAC4C1E
https://docs.oracle.com/cd/E80799_01/doc.121/e80521/GUID-4EAD207D-41FF-4FD7-8804-DE1B2EB7BE90.htm#CMTXG-GUID-E0CC0E23-A339-45D8-AFA2-09732DC6BC18
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.3/cmtxl/reimaging-oda.html#GUID-6DEDCA82-94B7-4155-9FBD-3B1FDC232FCC
https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/18.5/cmtxl/create-appliance-using-json-file.html#GUID-42250FD2-EA91-4457-9ED7-CA3A2A863B40
https://docs.oracle.com/cd/E75550_01/doc.121/e77147/GUID-F99D3337-E913-405D-BE3F-85C31103A26A.htm#CMTAR-GUID-CF1FECF0-722F-4B43-82AA-525EA169B3E8
 

Shrink ASM Diskgroup and Exadata Grid Disks
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Shrink ASM Diskgroup and Exadata Grid Disks

Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify if you have something in the ASM internal extent map that can block the shrink and “fix” this.
Here, in this scenario, I will reduce the size for grid disks linked with diskgroup DATAC8 (that run in a cluster for VM #08).
And before continuing, be aware of Exadata disk division:

 

ASM Extent Map and Moves

So, my actual usage space for DATAC8 is:
 

 

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  29921280  29790140           498688        14645726              0             Y  DATAC8/

MOUNTED  NORMAL  N         512             512   4096  4194304   6144000   6142020           102400         3019810              0             N  RECOC8/

ASMCMD>




SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free

  2  from v$asm_diskgroup

  3  order by 1;




NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE

------------------------------ ---------- ---------- ---------- ----------

DATAC8                           29921280   29790140     131140      99.56

RECOC8                            6144000    6142020       1980      99.97




SQL>


As you can see above, the RAW size is around 29.2 TB (14.6 after the ASM NORMAL mirror) and the actual free space is around 14.3TB. So, plenty of space for the reduction in this case. Unfortunately, even if I have 99% of free space don’t mean that I can reduce instantly to 1TB.
This occurs because, probably, I have a fragmented diskgroup and something can be allocated at the end of some disk and will deny me do reduce the disk. I will not dig in this topic but will show you how to check this. If you want to check with more details you can read these two articles: http://asmsupportguy.blogspot.com/2011/06/asm-file-extent-map.html and http://asmsupportguy.blogspot.com/2012/10/where-is-my-data.html.  
The idea is to use the ASM extend map from X$KFFXP to discover the object that has the MAX AU_KFFXP into some disk (whatever which one). So, I made:

 

SQL> select GROUP_NUMBER, name from v$asm_diskgroup;




GROUP_NUMBER NAME

------------ ------------------------------

           1 DATAC8

           2 RECOC8




SQL>

SQL> select VALUE from V$ASM_ATTRIBUTE where NAME='au_size' and GROUP_NUMBER=1;




VALUE

--------------------------------------------------------------------------------

4194304




SQL>

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;




MAX(AU_KFFXP)

-------------

       114195




SQL>

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114195;




NUMBER_KFFXP

------------

         262




SQL>

SQL> select name from v$asm_alias where FILE_NUMBER = 262 and GROUP_NUMBER = 1;




NAME

----------------------------------------------------------------------

group_1.262.983716383




SQL>



Above I discovered that the file 252 (that it is a redo – will show after), is allocated in the allocation unit 114195 from diskgroup 1 (that have – and default fore Exadata – allocation unit of 4MB). Doing a little count (114195*4), this means that this file resides in somewhere around 446GB into some disk. And if I try to reduce the disk below this value, I will receive an error.
To solve this we need to move files, in this case, the file is related to MGMTDB:

 

 

ASMCMD> ls -l +DATAC8/_MGMTDB/ONLINELOG/

Type       Redund  Striped  Time             Sys  Name

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_1.262.983716383

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_2.264.983716383

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_3.263.983716383

ASMCMD>

ASMCMD> ls -l +DATAC8

Type      Redund  Striped  Time         Sys  Name

                                        Y    ASM/

                                        Y    _MGMTDB/

                                        Y    exa-cl8/

PASSWORD  HIGH    COARSE   JAN 20 2017  N    orapwasm => +DATAC8/ASM/PASSWORD/pwdasm.256.933784119

PASSWORD  HIGH    COARSE   AUG 09 2018  N    orapwasm_backup => +DATAC8/ASM/PASSWORD/pwdasm.1471.983713235

ASMCMD> ls -l +RECOC8

ASMCMD>


To move MGMTDB we can follow the steps from note 2065175.1 and use the script mdbutil.pl. In this case, I moved to RECOC8:

 

[grid@exa01vm08 +ASM1]$ /tmp/MGMTDB/mdbutil.pl --mvmgmtdb --target=+RECOC8 -debug

mdbutil.pl version : 1.98

2019-06-21 11:43:33: D Executing: /u01/app/18.0.0/grid/bin/srvctl status diskgroup -g RECOC8

2019-06-21 11:43:34: D Exit code: 0

2019-06-21 11:43:34: D Output of last command execution:

Disk Group RECOC8 is running on exa01vm08

2019-06-21 11:43:34: D Executing: /u01/app/18.0.0/grid/bin/srvctl status mgmtdb

2019-06-21 11:43:35: D Exit code: 0

2019-06-21 11:43:35: D Output of last command execution:

…

…

2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/crsctl query crs activeversion

2019-06-21 11:53:20: D Exit code: 0

2019-06-21 11:53:20: D Output of last command execution:

Oracle Clusterware active version on the cluster is [18.0.0.0.0]

2019-06-21 11:53:20: I Starting the Cluster Health Analysis Resource

2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/srvctl start cha

2019-06-21 11:53:22: D Exit code: 0

2019-06-21 11:53:22: D Output of last command execution:

2019-06-21 11:53:22: I MGMTDB Successfully moved to +RECOC8!

[grid@exa01vm08 +ASM1]$


The output above was cropped and you can see in raw here. And after that, doing the same query as above, I checked again the extent map:

 

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;




MAX(AU_KFFXP)

-------------

       114132




SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114132;




NUMBER_KFFXP

------------

         255




SQL> select name from v$asm_alias where FILE_NUMBER = 255 and GROUP_NUMBER = 1;




NAME

----------------------------------------------------------------------

REGISTRY.255.933784121




SQL>

So, more one file to move and not it is related to OCR. To move I made:

 

[root@exa01vm08 ~]# export ORACLE_HOME=/u01/app/18.0.0/grid

[root@exa01vm08 ~]# export PATH=$ORACLE_HOME/bin:$PATH

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -add +RECOC8

[root@exa01vm08 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          4

         Total space (kbytes)     :     491684

         Used space (kbytes)      :      88624

         Available space (kbytes) :     403060

         ID                       :  354072626

         Device/File Name         :    +DATAC8

                                    Device/File integrity check succeeded

         Device/File Name         :    +RECOC8

                                    Device/File integrity check succeeded




                                    Device/File not configured




                                    Device/File not configured




                                    Device/File not configured




         Cluster registry integrity check succeeded




         Logical corruption check succeeded




[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -delete +DATAC8

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          4

         Total space (kbytes)     :     491684

         Used space (kbytes)      :      88624

         Available space (kbytes) :     403060

         ID                       :  354072626

         Device/File Name         :    +RECOC8

                                    Device/File integrity check succeeded




                                    Device/File not configured




                                    Device/File not configured




                                    Device/File not configured




                                    Device/File not configured




         Cluster registry integrity check succeeded




         Logical corruption check succeeded




[root@exa01vm08 ~]#


Above I: added a new place to OCR store the OCR files(RECOC8) and deleted the other (DATAC8). So, we still have more to move (votedisks, older OCR backups and ASM password file). To move votedisks I made:

 

[root@exa01vm08 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------
  1. ONLINE 077367b255804f0abf804a3a3ca8045d (o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04) [DATAC8]
  2. ONLINE 59ef748a7d2a4f43bffb54a90fe1b1a9 (o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05) [DATAC8]
  3. ONLINE cbaad30809f94fdcbfe5d60f6529ba63 (o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02) [DATAC8]
Located 3 voting disk(s).

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl replace votedisk +RECOC8

Successful addition of voting disk 081e9e767bc44ff2bff6067229378db5.

Successful addition of voting disk 82649e3309d34fa0bf4fd3c89c93e42f.

Successful addition of voting disk 57f3ec44b44b4fcdbf35e716e13011e9.

Successful deletion of voting disk 077367b255804f0abf804a3a3ca8045d.

Successful deletion of voting disk 59ef748a7d2a4f43bffb54a90fe1b1a9.

Successful deletion of voting disk cbaad30809f94fdcbfe5d60f6529ba63.

Successfully replaced voting disk group with +RECOC8.

CRS-4266: Voting file(s) successfully replaced

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------
  1. ONLINE 081e9e767bc44ff2bff6067229378db5 (o/200.200.10.11;200.200.10.12/RECOC8_CD_02_exaceladm04) [RECOC8]
  2. ONLINE 82649e3309d34fa0bf4fd3c89c93e42f (o/200.200.10.9;200.200.10.10/RECOC8_CD_02_exaceladm03) [RECOC8]
  3. ONLINE 57f3ec44b44b4fcdbf35e716e13011e9 (o/200.200.10.13;200.200.10.14/RECOC8_CD_03_exaceladm05) [RECOC8]
Located 3 voting disk(s).

[root@exa01vm08 ~]#


And to backups for OCR:

 

[root@exa01vm08 ~]# ocrconfig -showbackuploc

The Oracle Cluster Registry backup location is [+DATAC8].

[root@exa01vm08 ~]# ocrconfig -backuploc +RECOC8

[root@exa01vm08 ~]#

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -showbackuploc

The Oracle Cluster Registry backup location is [+RECOC8].

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -manualbackup




exa01vm08     2019/06/21 12:08:05     +RECOC8:/exa-cl8/OCRBACKUP/backup_20190621_120805.ocr.4722.1011528485     671056737




exa01vm08     2018/08/16 12:38:11     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180816_123811.ocr.1511.984314291     671056737




exa01vm08     2018/08/09 15:56:08     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180809_155608.ocr.1508.983721369     2960767134

[root@exa01vm08 ~]#



 

After one more backup, I deleted manually the old OCR backups. And to move ASM password:

 

[grid@exa01vm08 +ASM1]$ asmcmd pwget --asm

+DATAC8/orapwASM

[grid@exa01vm08 +ASM1]$ srvctl config asm -detail

ASM home: <CRS home>

Password file: +DATAC8/orapwASM

Backup of Password file: +DATAC8/orapwASM_backup

ASM listener: LISTENER

ASM is enabled.

ASM is individually enabled on nodes:

ASM is individually disabled on nodes:

ASM instance count: ALL

Cluster ASM listener: ASMNET1LSNR_ASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM +RECOC8/orapwASM -f

moving +DATAC8/orapwASM -> +RECOC8/orapwASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ srvctl config asm -detail

ASM home: <CRS home>

Password file: +RECOC8/orapwASM

Backup of Password file: +DATAC8/orapwASM_backup

ASM listener: LISTENER

ASM is enabled.

ASM is individually enabled on nodes:

ASM is individually disabled on nodes:

ASM instance count: ALL

Cluster ASM listener: ASMNET1LSNR_ASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM_backup +RECOC8/orapwASM_backup -f

moving +DATAC8/orapwASM_backup -> +RECOC8/orapwASM_backup

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$


Maybe you don’t need to move everything, I recommend that after every move you check the extent map for ASM and verify if the value less the minimum size per disk that you want.
Since I changed a lot of things I made a restart for the cluster in both nodes:

 

 

[root@exa01vm08 ~]# crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'exa01vm08'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'exa01vm08'

…

CRS-2673: Attempting to stop 'ora.diskmon' on 'exa01vm08'

CRS-2677: Stop of 'ora.diskmon' on 'exa01vm08' succeeded

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exa01vm08'

…

…

CRS-2672: Attempting to start 'ora.crsd' on 'exa01vm08'

CRS-2676: Start of 'ora.crsd' on 'exa01vm08' succeeded

[root@exa01vm08 ~]#

And after all of these moves I got the extent map with:

 

 

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;




MAX(AU_KFFXP)

-------------

        15149




SQL>

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 15149;




NUMBER_KFFXP

------------

           1




SQL>
Basically, this means that something is around 60Gb of some disk. But the link is for a file #1. So, to check what is this you can query inside X$KFFXP:

 

SQL> select NUMBER_KFFXP "ASM file number",

  2  DECODE (NUMBER_KFFXP,

  3          1, 'File directory'

  4          , 2, 'Disk directory'

  5          , 3, 'Active change directory'

  6          , 4, 'Continuing operations directory'

  7          , 5, 'Template directory'

  8          , 6, 'Alias directory'

  9          , 7, 'ADVM file directory'

 10          , 8, 'Disk free space directory'

 11          , 9, 'Attributes directory'

 12          , 10, 'ASM User directory'

 13          , 11, 'ASM user group directory'

 14          , 12, 'Staleness directory'

 15          , 253, 'spfile for ASM instance'

 16          , 254, 'Stale bit map space registry '

 17          , 255, 'Oracle Cluster Repository registry') "ASM metadata file name",

 18  count(AU_KFFXP) "Allocation units"

 19  from X$KFFXP

 20  where GROUP_KFFXP=1

 21  group by NUMBER_KFFXP

 22  order by 1;




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

              1 File directory                                   15

              2 Disk directory                                    3

              3 Active change directory                          69

              4 Continuing operations directory                   6

              5 Template directory                                3

              6 Alias directory                                   3

              8 Disk free space directory                         3

              9 Attributes directory                              3

             12 Staleness directory                               3

             13                                                   3

             16                                                   3




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

            120                                                   3

            121                                                 180

            253 spfile for ASM instance                           2

            254 Stale bit map space registry                     18

            255 Oracle Cluster Repository registry               83

            262                                                  99

            321                                                 185

            428                                                 221

            492                                                 305

            551                                                 129

            584                                                 705




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

            740                                                1865

           1003                                                 105

           1417                                                   2

           4708                                                   6

           4710                                                   6

           4713                                                   6

           4716                                                   6

           4719                                                   6

           4722                                                   6

           4731                                                  52

           4733                                                  52




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

           4736                                                  52

           4739                                                 379

           4742                                                 379

           4744                                                  52

           4747                                                 379

           4750                                                  24

           4753                                                 129

           4756                                                5125

           4759                                                7685

           4762                                                1029

           4765                                                 517




ASM file number ASM metadata file name             Allocation units


--------------- ---------------------------------- ----------------

           4768                                                  52

           4771                                                   4

           4773                                               11819




47 rows selected.




SQL>



In this case, it is a file directory ASM, but I will not try to move because it is even impossible in my case because does not appears in ASMCMD command. Basically, can be internal file/directory that we can’t see. Since I know that the diskgroup is empty and the allocation is related to internal ASM, I continued.
Remember at the end of the procedure, after shrinking in grid disk (that I will show later) to move again these files to original diskgroup. I will not cover this new move here in the post but the steps are the same that I showed before.

 

Shrink in ASM

The shrinking part start in ASM it is simple and you just need to define the value that you want per disk and reduce them. The most critical part here is to define one size for the disk that it is aligned with 16 MB of the grid disk. Different from when you are increasing the space, here, for shrink, you start to define the size in ASM side and after go to grid disk. And since the grid disk in storage cell round down the value to the nearest 16MB value, you can misalign and have ASM disk with a different value than grid disk (and this will be REALLY BAD). A trick secret is to think directly in GB to avoid the error.
For 16 Mb explanation you can check in the Exadata docs https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-42DA2512-667D-443C-93C5-6E5110DFAE21:
Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.
Since I defined to reduce the DATAC8 to around 1.8TB I defined the disks with 30GB:

 

SQL> select ((30*12)*5) as sizeGB FROM dual;




    SIZEGB

----------

      1800




SQL>


Remember in the beginning that you need to know your environment? Here I have 12 disks per cell and 5 cells:

 

SQL> set linesize 250

SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC8')

  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status

  6  order by 1,2,3;




NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS

------ ------------------------------ -------- ------------ ------- ------- ----------

DATAC8 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12




SQL>



After choosing the size that you want, you need to check if all the grid disk as online. It is not recommended to do the shrink with missing disks or even faulty failgroups. To check both at the same time you can do:

 

SQL> col path format a100

SQL> select dg.name, d.failgroup, d.path

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC8') and state = 'NORMAL'

  5  order by 1,2,3;




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_00_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_01_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_02_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_03_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_04_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_05_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_06_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_07_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_08_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_09_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_10_exaceladm01




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_11_exaceladm01

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_00_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_01_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_03_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_04_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_05_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_06_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_07_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_08_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_09_exaceladm02




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_10_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_11_exaceladm02

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_00_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_01_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_02_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_03_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_04_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_05_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_06_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_07_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_08_exaceladm03




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_09_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_10_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_11_exaceladm03

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_00_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_01_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_03_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_04_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_05_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_06_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_07_exaceladm04




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_08_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_09_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_10_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_11_exaceladm04

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_00_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_01_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_03_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_04_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_05_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_06_exaceladm05




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_07_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_08_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_09_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_10_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_11_exaceladm05




60 rows selected.




SQL>


After all the checks 16MB align, missing disks, and online failgroups you can do the resize in ASM:

 

SQL> alter diskgroup DATAC8 resize all size 30720M rebalance power 1024;




Diskgroup altered.



As you can see above, I specified in MB and I made this to show you where the error can occurs. If I defined (as example) disk size as 30700M it will not be aligned to 16MB (30700/16 = 1918,75) and the griddisk will be 30688M and if ASM allocate something in the end of the disk you will corrupt something.
After executing the change in ASM you just continue after having nothing in v$asm_operation. And for ASM now you see the size for your diskgroup:

 

SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free

  2  from v$asm_diskgroup

  3  order by 1;




NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE

------------------------------ ---------- ---------- ---------- ----------

DATAC8                            1843200    1837896       5304      99.71

RECOC8                            6144000    6016048     127952      97.92




SQL>



Shrink for GRID DISK

After shrinking in ASM side, you need to reduce in storage side the grid disk to release the space to celldisk. The procedure is the same for the increase and you use ALTER GRIDDISK to specify the new value.
Just to show, that before the shrink in storage cell I have disk defined for grid disk (487GB) and celldisk (346GB free):

 

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;

         name:                   DATAC8_CD_00_exaceladm01

         asmDiskGroupName:       DATAC8

         asmDiskName:            DATAC8_CD_00_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_00_exaceladm01

         comment:                "Cluster exa-cl8 diskgroup DATAC8"

         creationTime:           2017-01-20T16:19:29+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2

         size:                   487G

         status:                 active




CellCLI>




CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;

         name:                   CD_00_exaceladm01

         comment:

         creationTime:           2016-11-29T10:44:55+01:00

         deviceName:             /dev/sda

         devicePartition:        /dev/sda3

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              346.0625G

         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55

         physicalDisk:           PUUK3V

         size:                   7.1192474365234375T

         status:                 normal




CellCLI>



As before, you have two option: execute manually disk by disk or using the script with dcli. Below you see that I created and called the script (I cropped the output but you can see the RAW execution here):

 

[DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM01 size=30720M;

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM02 size=30720M;

…

…

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM02 size=30720M;

dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM03 size=30720M;

dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM04 size=30720M;

dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM05 size=30720M;

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$




[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

exaceladm01: GridDisk DATAC8_CD_00_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC8_CD_00_exaceladm02 successfully altered

…

…

exaceladm02: GridDisk DATAC8_CD_11_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC8_CD_11_exaceladm03 successfully altered

exaceladm04: GridDisk DATAC8_CD_11_exaceladm04 successfully altered

exaceladm05: GridDisk DATAC8_CD_11_exaceladm05 successfully altered

[DOM0 - root@exadbadm01 tmp]$

 

Check again that I used the value defined in MB, 30720MB in this case. Again, be careful with 16MB align. After the change I have in storage cell:

 

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;

         name:                   DATAC8_CD_00_exaceladm01

         asmDiskGroupName:       DATAC8

         asmDiskName:            DATAC8_CD_00_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_00_exaceladm01

         comment:                "Cluster exa-cl8 diskgroup DATAC8"

         creationTime:           2017-01-20T16:19:29+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2

         size:                   30G

         status:                 active




CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;

         name:                   CD_00_exaceladm01

         comment:

         creationTime:           2016-11-29T10:44:55+01:00

         deviceName:             /dev/sda

         devicePartition:        /dev/sda3

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              803.0625G

         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55

         physicalDisk:           PUUK3V

         size:                   7.1192474365234375T

         status:                 normal




CellCLI>


Conclusion

Execute the shrink for Exadata is something that you don’t realize every day in your daily tasks. I suppose that not in a quarterly task because today is always “add, add, add”. Besides the increase of space, the shrink is trickier, and you really need to take care for more steps. Maybe you need to move some data, restart the cluster, move data again.
But, the most critical part is the 16MB alignment, more than when you add space. This is important because for shrink you define the value in two places: for ASM and for grid disk. If you choose a bad value, the size for grid disk can differ ASM and you will corrupt something. As I told, if you think directly in GB instead of MB for disk sizes, you are safer because GB is always compatible with 16MB.
 
This post was published in my personal blog too: http://www.fernandosimon.com/blog/shrink-asm-diskgroup-and-exadata-grid-disks/
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


Increase Size For Exadata Grid Disks
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Increase Size For Exadata Grid Disks

A quick article about a maintenance task for Oracle Exadata when you are using OVM and you divided your storage cell disks for every VM. Here I will show you how to extend your Grid Disks to add more space in your ASM diskgroup.
The first thing is being aware of your environment, before everything you need to know the points below because, they are important to calculate the new space, and to avoid do something wrong:
  • Number of cells in your appliance.
  • Number of disks for each cell.
  • Mirroring for your ASM.
  • The VM that you want to add the space.
The “normal” Exadata storage cell has 12 disks, the Extreme Flash version uses 8 disks per storage. If you have doubt about how many disks you have per storage cell, you can connect in each one and check the number of celldisks you have. And before continuing, be aware of Exadata disk division:

 

 

To do this change we execute three major steps: ASM, Exadata Storage, and ASM again.

 

For ASM

 

Inside ASM we can use this query to collect some information about the diskgroups:

 

SQL> col name format a12 head 'Disk Group';

SQL> col total_mb format 999999999 head 'Total GB|Raw';

SQL> col free_mb format 999999999 head 'Free GB|Raw';

SQL> col avail_mb format 999999999 head 'Total GB|Usable';

SQL> col usable_mb format 999999999 head 'Free GB|Usable';

SQL> col usable_mb format 999999999 head 'Free GB|Usable';

SQL> col cdisks format 99999 head 'Cell|Disksl';

SQL>

SQL> select a.name,a.total_mb,a.free_mb,a.type,

  2  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) avail_mb,

  3  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024) usable_mb,

  4  count(b.path) cdisks

  5  from v$asm_diskgroup a, v$asm_disk b

  6  where a.group_number=b.group_number

  7  group by a.name,a.total_mb,a.free_mb,a.type,

  8  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) ,

  9  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024)

 10  order by 2,1

 11  /




               Total GB    Free GB          Total GB    Free GB   Cell

Disk Group          Raw        Raw TYPE       Usable     Usable Disksl

------------ ---------- ---------- ------ ---------- ---------- ------

RECOC3          4239360    2465540 NORMAL       2070       1204     60

DATAC3         15790080    2253048 NORMAL       7710       1100     60




SQL>

SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC3')

  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status

  6  order by 1,2,3;




NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS

------ ------------------------------ -------- ------------ ------- ------- ----------

DATAC3 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12




SQL>

 

 

With that, I have three important information: number the disks (60), redundancy type (NORMAL), total actual size (RAW value – 15790080). To discover the size for each disk in ASM (here I do manually and not check in v$asm_disk just to show you the steps and to be more didact) you can divide the raw space/#disks:

 

SQL> SELECT (15790080/1024)/60 as gbDISK FROM dual; 

GBDISK
----------       
257 

SQL>

 

So, each disk has 257GB of space size (in raw). Since the actual free space is 1100GB (1.07TB) and we want to add more 2TB we need to increase the value for each disk.

 

The formula is simple: NewValue(inGB)*#OfDisksPerCell*#NumberofCells. Here I choose 330GB per disk, so, the new size for diskgroup will be:

 

SQL> SELECT (330*12*5) AS newsizeGB FROM dual;  

NEWSIZEGB
----------     
19800 

SQL>

 

But this value is not correct because does not consider the mirror type, so, we need to divide this value. If it NORMAL, divide by 2, if HIGH, divide by 3. To compare, the old and new expected value:

 

SQL> SELECT (257*12*5)/2 as actualsizeGBUsable, (330*12*5)/2 AS newsizeGBUSable FROM dual;

ACTUALSIZEGBUSABLE NEWSIZEGBUSABLE
------------------ ---------------    
          7710            9900 

SQL>


 

So, the new total space for diskgroup will be around 9.6 TB (9900 GB). And we will add (as free space) around 2.1 TB. Probably you need to execute these formulas more than one time to find the desired size per disk.

 

I start to calculate by disk (and after discovering the final diskgroup size) instead of starting with diskgroup size (and dividing to discover the disk size) because doing this way, the size for disk will be always correct and align with storage cell grid disk. Remember that grid disks are aligned in 16MB and, if you start to choose one arbitrary value to the max size for the ASM diskgroup, you can reach a value per grid disk that is not 16MB aligned. As an example, if I start choosing 20TB for diskgroup, the size per disk will be (20*1024)/60 = 341.33GB and this is not aligned with 16MB.

 

For 16 Mb explanation, you can check in the Exadata docs: 

 

Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.

 

For Storage Cell

 

In the Exadata side, first, check some info about the actual state for grid disks. Here I connect in one cell (if you want you can use dcli to call every/all cells) and check some info for the grid disk:

 

 

[root@exaceladm01 ~]# cellcliCellCLI

Release 18.1.6.0.0 - Production on Fri Jun 21 16:57:49 CEST 2019 Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved. 

CellCLI> list celldisk

CD_00_exaceladm01       normal  
CD_01_exaceladm01       normal         
CD_02_exaceladm01       normal         
CD_03_exaceladm01       normal         
CD_04_exaceladm01       normal         
CD_05_exaceladm01       normal         
CD_06_exaceladm01       normal         
CD_07_exaceladm01       normal         
CD_08_exaceladm01       normal         
CD_09_exaceladm01       normal         
CD_10_exaceladm01       normal         
CD_11_exaceladm01       normal         
FD_00_exaceladm01       normal         
FD_01_exaceladm01       normal         
FD_02_exaceladm01       normal
FD_03_exaceladm01       normal

CellCLI> list celldisk 
CD_02_exaceladm01 detail

name:                   CD_02_exaceladm01         
comment:         
creationTime:           2016-11-29T10:23:35+01:00         
deviceName:             /dev/sdc         
devicePartition:        /dev/sdc         
diskType:               HardDisk         
errorCount:             0         
freeSpace:              2.6688079833984375T         
id:                     d57b31bb-6043-4cea-b992-ef8075f42e77         
physicalDisk:           PUT81V         
size:                   7.152252197265625T         
status:                 normal


CellCLI> 

CellCLI> list griddisk where name like 'DATAC3.*';

DATAC3_CD_00_exaceladm01        active         
DATAC3_CD_01_exaceladm01        active         
DATAC3_CD_02_exaceladm01        active         
DATAC3_CD_03_exaceladm01        active         
DATAC3_CD_04_exaceladm01        active         
DATAC3_CD_05_exaceladm01        active         
DATAC3_CD_06_exaceladm01        active         
DATAC3_CD_07_exaceladm01        active         
DATAC3_CD_08_exaceladm01        active         
DATAC3_CD_09_exaceladm01        active         
DATAC3_CD_10_exaceladm01        active         
DATAC3_CD_11_exaceladm01        active

CellCLI>

CellCLI> list griddisk where name = 'DATAC3_CD_04_exaceladm01' detail;

name:                   DATAC3_CD_04_exaceladm01         
asmDiskGroupName:       DATAC3         
asmDiskName:            DATAC3_CD_04_EXACELADM01         
asmFailGroupName:       EXACELADM01         
availableTo:         
cachedBy:               FD_00_exaceladm01         
cachingPolicy:          default         
cellDisk:               CD_04_exaceladm01         
comment:                "Cluster exa-cl3 diskgroup DATAC3"         
creationTime:           2017-01-20T17:23:21+01:00         
diskType:               HardDisk         
errorCount:             0         
id:                     2cb2aecb-cfa1-4282-b90d-3a08ed079778         
size:                   257G         
status:                 active

CellCLI>


Here you can see that I checked:
  •  The celldisks info for this cell.
  •  Detail for one celldisk (look the freeSpaceattribute to verify if you have free space).
  •  The grid disks for this cell.
  •  Details for the griddisk (look that the size is the same value that I calculated manually).
This part was just to check and show you how to verify some info, with time, you don’t need to check this in every maintenance (because you will be familiar with the environment). Be careful that, if you have different grid disk space division per storage cells, you need to check if you have available space in all your storage celldisks.

 

To expand the grid disks you have two options, enter in each cell and expand manually one by one, or create one script and call by dcli (the option that I choose). So, create one script that executes the ALTER GRIDDISK command for the new desired size. Just remember to be careful and choose the correct grid disks (here is for VM 03, that means DATAC3):

 

DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC3_CD_00_EXACELADM01 size=330G;

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC3_CD_00_EXACELADM02 size=330G;

…

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM02 size=330G;

dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM03 size=330G;

dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM04 size=330G;

dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM05 size=330G;

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

exaceladm01: GridDisk DATAC3_CD_00_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC3_CD_00_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC3_CD_00_exaceladm03 successfully altered

…

…

exaceladm01: GridDisk DATAC3_CD_11_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC3_CD_11_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC3_CD_11_exaceladm03 successfully altered

exaceladm04: GridDisk DATAC3_CD_11_exaceladm04 successfully altered

exaceladm05: GridDisk DATAC3_CD_11_exaceladm05 successfully altered

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$




Above I cropped the output to reduce the size or post, but you can check the raw output here. After the change you can check the info for the grid disk:

 

[root@exaceladm01 ~]# cellcli

CellCLI: Release 18.1.6.0.0 - Production on Fri Jun 21 17:24:03 CEST 2019




Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved.




CellCLI> list griddisk where name = 'DATAC3_CD_04_exaceladm01' detail;

         name:                   DATAC3_CD_04_exaceladm01

         asmDiskGroupName:       DATAC3

         asmDiskName:            DATAC3_CD_04_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_04_exaceladm01

         comment:                "Cluster exa-cl3 diskgroup DATAC3"

         creationTime:           2017-01-20T17:23:21+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     2cb2aecb-cfa1-4282-b90d-3a08ed079778

         size:                   330G

         status:                 active




CellCLI> list celldisk CD_02_exaceladm01 detail

         name:                   CD_02_exaceladm01

         comment:

         creationTime:           2016-11-29T10:23:35+01:00

         deviceName:             /dev/sdc

         devicePartition:        /dev/sdc

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              2.5975189208984375T

         id:                     d57b31bb-6043-4cea-b992-ef8075f42e77

         physicalDisk:           PUT81V

         size:                   7.152252197265625T

         status:                 normal




CellCLI> exit

quitting




[root@exaceladm01 ~]#


For ASM – Part #2
After you change the grid disks in the storage side, you can go back to ASM and extend the diskgroup:

 

SQL> ALTER DISKGROUP DATAC3 RESIZE ALL;




Diskgroup altered.




SQL>


And you can check that the size was already added (look that values hit what we calculated before):

 

SQL> select a.name,a.total_mb,a.free_mb,a.type,

  2  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) avail_mb,

  3  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024) usable_mb,

  4  count(b.path) cdisks

  5  from v$asm_diskgroup a, v$asm_disk b

  6  where a.group_number=b.group_number

  7  group by a.name,a.total_mb,a.free_mb,a.type,

  8  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) ,

  9  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024)

 10  order by 2,1

 11  /




               Total GB    Free GB          Total GB    Free GB   Cell

Disk Group          Raw        Raw TYPE       Usable     Usable Disksl

------------ ---------- ---------- ------ ---------- ---------- ------

RECOC3          4239360    2465540 NORMAL       2070       1204     60

DATAC3         20275200    6738128 NORMAL       9900       3290     60




SQL>


And you can check the v$asm_operation to check the rebalance progress:

 

SQL> select operation, EST_MINUTES, EST_RATE, EST_WORK, sofar from v$asm_operation;




OPERA EST_MINUTES   EST_RATE   EST_WORK      SOFAR

----- ----------- ---------- ---------- ----------

REBAL           0          0          0          0

REBAL           0      46158      18490       5761

REBAL           0          0          0          0

REBAL           0          0          0          0

REBAL           0          0          0          0




SQL>




Conclusion
As you can see, the steps to do that are simple and not complex, you just need to take care about some details of your environment: Number of the disks per cell, number the cells and the VM where you want to add the space is critical to do the correct change. Remember to align with 16MB the size of your grid disk, when you are adding it is not a big deal, but if you want to shrink this can break your ASM diskgroup.
Check that the only change effectively is the size of the grid disk, all the others occur automatically because of the grid disk. ASM diskgroup will increase to the max value that it is available and space is available just after the command.
The steps above are more detailed that you will do in daily maintenance, but help you to understand most of the datils for this kind of change.
 
Reference:
How to Resize Grid Disks in Exadata (Doc ID 2176737.1) – https://support.oracle.com/epmos/faces/DocContentDisplay?id=2176737.1
Resizing Grid Disks – https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-570A0C37-907C-4417-BC93-AC4ABAF7E3AD 

 

This post is published in my personal blog too: http://www.fernandosimon.com/blog/increase-size-for-exadata-grid-disks/
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific


1 … 4 5 6 7
LUXOUG – LUXEMBOURG ORACLE USERS GROUP
© 2024 LUXOUG – LUXEMBOURG ORACLE USERS GROUP