Blog
Shrink ASM Diskgroup and Exadata Grid Disks
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Shrink ASM Diskgroup and Exadata Grid Disks

Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify if you have something in the ASM internal extent map that can block the shrink and “fix” this.
Here, in this scenario, I will reduce the size for grid disks linked with diskgroup DATAC8 (that run in a cluster for VM #08).
And before continuing, be aware of Exadata disk division:

 

ASM Extent Map and Moves

So, my actual usage space for DATAC8 is:
 

 

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  29921280  29790140           498688        14645726              0             Y  DATAC8/

MOUNTED  NORMAL  N         512             512   4096  4194304   6144000   6142020           102400         3019810              0             N  RECOC8/

ASMCMD>




SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free

  2  from v$asm_diskgroup

  3  order by 1;




NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE

------------------------------ ---------- ---------- ---------- ----------

DATAC8                           29921280   29790140     131140      99.56

RECOC8                            6144000    6142020       1980      99.97




SQL>


As you can see above, the RAW size is around 29.2 TB (14.6 after the ASM NORMAL mirror) and the actual free space is around 14.3TB. So, plenty of space for the reduction in this case. Unfortunately, even if I have 99% of free space don’t mean that I can reduce instantly to 1TB.
This occurs because, probably, I have a fragmented diskgroup and something can be allocated at the end of some disk and will deny me do reduce the disk. I will not dig in this topic but will show you how to check this. If you want to check with more details you can read these two articles: http://asmsupportguy.blogspot.com/2011/06/asm-file-extent-map.html and http://asmsupportguy.blogspot.com/2012/10/where-is-my-data.html.  
The idea is to use the ASM extend map from X$KFFXP to discover the object that has the MAX AU_KFFXP into some disk (whatever which one). So, I made:

 

SQL> select GROUP_NUMBER, name from v$asm_diskgroup;




GROUP_NUMBER NAME

------------ ------------------------------

           1 DATAC8

           2 RECOC8




SQL>

SQL> select VALUE from V$ASM_ATTRIBUTE where NAME='au_size' and GROUP_NUMBER=1;




VALUE

--------------------------------------------------------------------------------

4194304




SQL>

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;




MAX(AU_KFFXP)

-------------

       114195




SQL>

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114195;




NUMBER_KFFXP

------------

         262




SQL>

SQL> select name from v$asm_alias where FILE_NUMBER = 262 and GROUP_NUMBER = 1;




NAME

----------------------------------------------------------------------

group_1.262.983716383




SQL>



Above I discovered that the file 252 (that it is a redo – will show after), is allocated in the allocation unit 114195 from diskgroup 1 (that have – and default fore Exadata – allocation unit of 4MB). Doing a little count (114195*4), this means that this file resides in somewhere around 446GB into some disk. And if I try to reduce the disk below this value, I will receive an error.
To solve this we need to move files, in this case, the file is related to MGMTDB:

 

 

ASMCMD> ls -l +DATAC8/_MGMTDB/ONLINELOG/

Type       Redund  Striped  Time             Sys  Name

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_1.262.983716383

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_2.264.983716383

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_3.263.983716383

ASMCMD>

ASMCMD> ls -l +DATAC8

Type      Redund  Striped  Time         Sys  Name

                                        Y    ASM/

                                        Y    _MGMTDB/

                                        Y    exa-cl8/

PASSWORD  HIGH    COARSE   JAN 20 2017  N    orapwasm => +DATAC8/ASM/PASSWORD/pwdasm.256.933784119

PASSWORD  HIGH    COARSE   AUG 09 2018  N    orapwasm_backup => +DATAC8/ASM/PASSWORD/pwdasm.1471.983713235

ASMCMD> ls -l +RECOC8

ASMCMD>


To move MGMTDB we can follow the steps from note 2065175.1 and use the script mdbutil.pl. In this case, I moved to RECOC8:

 

[grid@exa01vm08 +ASM1]$ /tmp/MGMTDB/mdbutil.pl --mvmgmtdb --target=+RECOC8 -debug

mdbutil.pl version : 1.98

2019-06-21 11:43:33: D Executing: /u01/app/18.0.0/grid/bin/srvctl status diskgroup -g RECOC8

2019-06-21 11:43:34: D Exit code: 0

2019-06-21 11:43:34: D Output of last command execution:

Disk Group RECOC8 is running on exa01vm08

2019-06-21 11:43:34: D Executing: /u01/app/18.0.0/grid/bin/srvctl status mgmtdb

2019-06-21 11:43:35: D Exit code: 0

2019-06-21 11:43:35: D Output of last command execution:

…

…

2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/crsctl query crs activeversion

2019-06-21 11:53:20: D Exit code: 0

2019-06-21 11:53:20: D Output of last command execution:

Oracle Clusterware active version on the cluster is [18.0.0.0.0]

2019-06-21 11:53:20: I Starting the Cluster Health Analysis Resource

2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/srvctl start cha

2019-06-21 11:53:22: D Exit code: 0

2019-06-21 11:53:22: D Output of last command execution:

2019-06-21 11:53:22: I MGMTDB Successfully moved to +RECOC8!

[grid@exa01vm08 +ASM1]$


The output above was cropped and you can see in raw here. And after that, doing the same query as above, I checked again the extent map:

 

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;




MAX(AU_KFFXP)

-------------

       114132




SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114132;




NUMBER_KFFXP

------------

         255




SQL> select name from v$asm_alias where FILE_NUMBER = 255 and GROUP_NUMBER = 1;




NAME

----------------------------------------------------------------------

REGISTRY.255.933784121




SQL>

So, more one file to move and not it is related to OCR. To move I made:

 

[root@exa01vm08 ~]# export ORACLE_HOME=/u01/app/18.0.0/grid

[root@exa01vm08 ~]# export PATH=$ORACLE_HOME/bin:$PATH

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -add +RECOC8

[root@exa01vm08 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          4

         Total space (kbytes)     :     491684

         Used space (kbytes)      :      88624

         Available space (kbytes) :     403060

         ID                       :  354072626

         Device/File Name         :    +DATAC8

                                    Device/File integrity check succeeded

         Device/File Name         :    +RECOC8

                                    Device/File integrity check succeeded




                                    Device/File not configured




                                    Device/File not configured




                                    Device/File not configured




         Cluster registry integrity check succeeded




         Logical corruption check succeeded




[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -delete +DATAC8

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          4

         Total space (kbytes)     :     491684

         Used space (kbytes)      :      88624

         Available space (kbytes) :     403060

         ID                       :  354072626

         Device/File Name         :    +RECOC8

                                    Device/File integrity check succeeded




                                    Device/File not configured




                                    Device/File not configured




                                    Device/File not configured




                                    Device/File not configured




         Cluster registry integrity check succeeded




         Logical corruption check succeeded




[root@exa01vm08 ~]#


Above I: added a new place to OCR store the OCR files(RECOC8) and deleted the other (DATAC8). So, we still have more to move (votedisks, older OCR backups and ASM password file). To move votedisks I made:

 

[root@exa01vm08 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------
  1. ONLINE 077367b255804f0abf804a3a3ca8045d (o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04) [DATAC8]
  2. ONLINE 59ef748a7d2a4f43bffb54a90fe1b1a9 (o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05) [DATAC8]
  3. ONLINE cbaad30809f94fdcbfe5d60f6529ba63 (o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02) [DATAC8]
Located 3 voting disk(s).

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl replace votedisk +RECOC8

Successful addition of voting disk 081e9e767bc44ff2bff6067229378db5.

Successful addition of voting disk 82649e3309d34fa0bf4fd3c89c93e42f.

Successful addition of voting disk 57f3ec44b44b4fcdbf35e716e13011e9.

Successful deletion of voting disk 077367b255804f0abf804a3a3ca8045d.

Successful deletion of voting disk 59ef748a7d2a4f43bffb54a90fe1b1a9.

Successful deletion of voting disk cbaad30809f94fdcbfe5d60f6529ba63.

Successfully replaced voting disk group with +RECOC8.

CRS-4266: Voting file(s) successfully replaced

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------
  1. ONLINE 081e9e767bc44ff2bff6067229378db5 (o/200.200.10.11;200.200.10.12/RECOC8_CD_02_exaceladm04) [RECOC8]
  2. ONLINE 82649e3309d34fa0bf4fd3c89c93e42f (o/200.200.10.9;200.200.10.10/RECOC8_CD_02_exaceladm03) [RECOC8]
  3. ONLINE 57f3ec44b44b4fcdbf35e716e13011e9 (o/200.200.10.13;200.200.10.14/RECOC8_CD_03_exaceladm05) [RECOC8]
Located 3 voting disk(s).

[root@exa01vm08 ~]#


And to backups for OCR:

 

[root@exa01vm08 ~]# ocrconfig -showbackuploc

The Oracle Cluster Registry backup location is [+DATAC8].

[root@exa01vm08 ~]# ocrconfig -backuploc +RECOC8

[root@exa01vm08 ~]#

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -showbackuploc

The Oracle Cluster Registry backup location is [+RECOC8].

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -manualbackup




exa01vm08     2019/06/21 12:08:05     +RECOC8:/exa-cl8/OCRBACKUP/backup_20190621_120805.ocr.4722.1011528485     671056737




exa01vm08     2018/08/16 12:38:11     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180816_123811.ocr.1511.984314291     671056737




exa01vm08     2018/08/09 15:56:08     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180809_155608.ocr.1508.983721369     2960767134

[root@exa01vm08 ~]#



 

After one more backup, I deleted manually the old OCR backups. And to move ASM password:

 

[grid@exa01vm08 +ASM1]$ asmcmd pwget --asm

+DATAC8/orapwASM

[grid@exa01vm08 +ASM1]$ srvctl config asm -detail

ASM home: <CRS home>

Password file: +DATAC8/orapwASM

Backup of Password file: +DATAC8/orapwASM_backup

ASM listener: LISTENER

ASM is enabled.

ASM is individually enabled on nodes:

ASM is individually disabled on nodes:

ASM instance count: ALL

Cluster ASM listener: ASMNET1LSNR_ASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM +RECOC8/orapwASM -f

moving +DATAC8/orapwASM -> +RECOC8/orapwASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ srvctl config asm -detail

ASM home: <CRS home>

Password file: +RECOC8/orapwASM

Backup of Password file: +DATAC8/orapwASM_backup

ASM listener: LISTENER

ASM is enabled.

ASM is individually enabled on nodes:

ASM is individually disabled on nodes:

ASM instance count: ALL

Cluster ASM listener: ASMNET1LSNR_ASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM_backup +RECOC8/orapwASM_backup -f

moving +DATAC8/orapwASM_backup -> +RECOC8/orapwASM_backup

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$


Maybe you don’t need to move everything, I recommend that after every move you check the extent map for ASM and verify if the value less the minimum size per disk that you want.
Since I changed a lot of things I made a restart for the cluster in both nodes:

 

 

[root@exa01vm08 ~]# crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'exa01vm08'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'exa01vm08'

…

CRS-2673: Attempting to stop 'ora.diskmon' on 'exa01vm08'

CRS-2677: Stop of 'ora.diskmon' on 'exa01vm08' succeeded

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exa01vm08'

…

…

CRS-2672: Attempting to start 'ora.crsd' on 'exa01vm08'

CRS-2676: Start of 'ora.crsd' on 'exa01vm08' succeeded

[root@exa01vm08 ~]#

And after all of these moves I got the extent map with:

 

 

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;




MAX(AU_KFFXP)

-------------

        15149




SQL>

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 15149;




NUMBER_KFFXP

------------

           1




SQL>
Basically, this means that something is around 60Gb of some disk. But the link is for a file #1. So, to check what is this you can query inside X$KFFXP:

 

SQL> select NUMBER_KFFXP "ASM file number",

  2  DECODE (NUMBER_KFFXP,

  3          1, 'File directory'

  4          , 2, 'Disk directory'

  5          , 3, 'Active change directory'

  6          , 4, 'Continuing operations directory'

  7          , 5, 'Template directory'

  8          , 6, 'Alias directory'

  9          , 7, 'ADVM file directory'

 10          , 8, 'Disk free space directory'

 11          , 9, 'Attributes directory'

 12          , 10, 'ASM User directory'

 13          , 11, 'ASM user group directory'

 14          , 12, 'Staleness directory'

 15          , 253, 'spfile for ASM instance'

 16          , 254, 'Stale bit map space registry '

 17          , 255, 'Oracle Cluster Repository registry') "ASM metadata file name",

 18  count(AU_KFFXP) "Allocation units"

 19  from X$KFFXP

 20  where GROUP_KFFXP=1

 21  group by NUMBER_KFFXP

 22  order by 1;




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

              1 File directory                                   15

              2 Disk directory                                    3

              3 Active change directory                          69

              4 Continuing operations directory                   6

              5 Template directory                                3

              6 Alias directory                                   3

              8 Disk free space directory                         3

              9 Attributes directory                              3

             12 Staleness directory                               3

             13                                                   3

             16                                                   3




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

            120                                                   3

            121                                                 180

            253 spfile for ASM instance                           2

            254 Stale bit map space registry                     18

            255 Oracle Cluster Repository registry               83

            262                                                  99

            321                                                 185

            428                                                 221

            492                                                 305

            551                                                 129

            584                                                 705




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

            740                                                1865

           1003                                                 105

           1417                                                   2

           4708                                                   6

           4710                                                   6

           4713                                                   6

           4716                                                   6

           4719                                                   6

           4722                                                   6

           4731                                                  52

           4733                                                  52




ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

           4736                                                  52

           4739                                                 379

           4742                                                 379

           4744                                                  52

           4747                                                 379

           4750                                                  24

           4753                                                 129

           4756                                                5125

           4759                                                7685

           4762                                                1029

           4765                                                 517




ASM file number ASM metadata file name             Allocation units


--------------- ---------------------------------- ----------------

           4768                                                  52

           4771                                                   4

           4773                                               11819




47 rows selected.




SQL>



In this case, it is a file directory ASM, but I will not try to move because it is even impossible in my case because does not appears in ASMCMD command. Basically, can be internal file/directory that we can’t see. Since I know that the diskgroup is empty and the allocation is related to internal ASM, I continued.
Remember at the end of the procedure, after shrinking in grid disk (that I will show later) to move again these files to original diskgroup. I will not cover this new move here in the post but the steps are the same that I showed before.

 

Shrink in ASM

The shrinking part start in ASM it is simple and you just need to define the value that you want per disk and reduce them. The most critical part here is to define one size for the disk that it is aligned with 16 MB of the grid disk. Different from when you are increasing the space, here, for shrink, you start to define the size in ASM side and after go to grid disk. And since the grid disk in storage cell round down the value to the nearest 16MB value, you can misalign and have ASM disk with a different value than grid disk (and this will be REALLY BAD). A trick secret is to think directly in GB to avoid the error.
For 16 Mb explanation you can check in the Exadata docs https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-42DA2512-667D-443C-93C5-6E5110DFAE21:
Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.
Since I defined to reduce the DATAC8 to around 1.8TB I defined the disks with 30GB:

 

SQL> select ((30*12)*5) as sizeGB FROM dual;




    SIZEGB

----------

      1800




SQL>


Remember in the beginning that you need to know your environment? Here I have 12 disks per cell and 5 cells:

 

SQL> set linesize 250

SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC8')

  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status

  6  order by 1,2,3;




NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS

------ ------------------------------ -------- ------------ ------- ------- ----------

DATAC8 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12




SQL>



After choosing the size that you want, you need to check if all the grid disk as online. It is not recommended to do the shrink with missing disks or even faulty failgroups. To check both at the same time you can do:

 

SQL> col path format a100

SQL> select dg.name, d.failgroup, d.path

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC8') and state = 'NORMAL'

  5  order by 1,2,3;




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_00_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_01_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_02_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_03_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_04_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_05_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_06_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_07_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_08_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_09_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_10_exaceladm01




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_11_exaceladm01

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_00_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_01_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_03_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_04_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_05_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_06_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_07_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_08_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_09_exaceladm02




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_10_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_11_exaceladm02

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_00_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_01_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_02_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_03_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_04_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_05_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_06_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_07_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_08_exaceladm03




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_09_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_10_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_11_exaceladm03

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_00_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_01_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_03_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_04_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_05_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_06_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_07_exaceladm04




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_08_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_09_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_10_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_11_exaceladm04

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_00_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_01_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_03_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_04_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_05_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_06_exaceladm05




NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_07_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_08_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_09_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_10_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_11_exaceladm05




60 rows selected.




SQL>


After all the checks 16MB align, missing disks, and online failgroups you can do the resize in ASM:

 

SQL> alter diskgroup DATAC8 resize all size 30720M rebalance power 1024;




Diskgroup altered.



As you can see above, I specified in MB and I made this to show you where the error can occurs. If I defined (as example) disk size as 30700M it will not be aligned to 16MB (30700/16 = 1918,75) and the griddisk will be 30688M and if ASM allocate something in the end of the disk you will corrupt something.
After executing the change in ASM you just continue after having nothing in v$asm_operation. And for ASM now you see the size for your diskgroup:

 

SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free

  2  from v$asm_diskgroup

  3  order by 1;




NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE

------------------------------ ---------- ---------- ---------- ----------

DATAC8                            1843200    1837896       5304      99.71

RECOC8                            6144000    6016048     127952      97.92




SQL>



Shrink for GRID DISK

After shrinking in ASM side, you need to reduce in storage side the grid disk to release the space to celldisk. The procedure is the same for the increase and you use ALTER GRIDDISK to specify the new value.
Just to show, that before the shrink in storage cell I have disk defined for grid disk (487GB) and celldisk (346GB free):

 

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;

         name:                   DATAC8_CD_00_exaceladm01

         asmDiskGroupName:       DATAC8

         asmDiskName:            DATAC8_CD_00_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_00_exaceladm01

         comment:                "Cluster exa-cl8 diskgroup DATAC8"

         creationTime:           2017-01-20T16:19:29+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2

         size:                   487G

         status:                 active




CellCLI>




CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;

         name:                   CD_00_exaceladm01

         comment:

         creationTime:           2016-11-29T10:44:55+01:00

         deviceName:             /dev/sda

         devicePartition:        /dev/sda3

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              346.0625G

         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55

         physicalDisk:           PUUK3V

         size:                   7.1192474365234375T

         status:                 normal




CellCLI>



As before, you have two option: execute manually disk by disk or using the script with dcli. Below you see that I created and called the script (I cropped the output but you can see the RAW execution here):

 

[DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM01 size=30720M;

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM02 size=30720M;

…

…

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM02 size=30720M;

dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM03 size=30720M;

dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM04 size=30720M;

dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM05 size=30720M;

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$




[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

exaceladm01: GridDisk DATAC8_CD_00_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC8_CD_00_exaceladm02 successfully altered

…

…

exaceladm02: GridDisk DATAC8_CD_11_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC8_CD_11_exaceladm03 successfully altered

exaceladm04: GridDisk DATAC8_CD_11_exaceladm04 successfully altered

exaceladm05: GridDisk DATAC8_CD_11_exaceladm05 successfully altered

[DOM0 - root@exadbadm01 tmp]$

 

Check again that I used the value defined in MB, 30720MB in this case. Again, be careful with 16MB align. After the change I have in storage cell:

 

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;

         name:                   DATAC8_CD_00_exaceladm01

         asmDiskGroupName:       DATAC8

         asmDiskName:            DATAC8_CD_00_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_00_exaceladm01

         comment:                "Cluster exa-cl8 diskgroup DATAC8"

         creationTime:           2017-01-20T16:19:29+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2

         size:                   30G

         status:                 active




CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;

         name:                   CD_00_exaceladm01

         comment:

         creationTime:           2016-11-29T10:44:55+01:00

         deviceName:             /dev/sda

         devicePartition:        /dev/sda3

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              803.0625G

         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55

         physicalDisk:           PUUK3V

         size:                   7.1192474365234375T

         status:                 normal




CellCLI>


Conclusion

Execute the shrink for Exadata is something that you don’t realize every day in your daily tasks. I suppose that not in a quarterly task because today is always “add, add, add”. Besides the increase of space, the shrink is trickier, and you really need to take care for more steps. Maybe you need to move some data, restart the cluster, move data again.
But, the most critical part is the 16MB alignment, more than when you add space. This is important because for shrink you define the value in two places: for ASM and for grid disk. If you choose a bad value, the size for grid disk can differ ASM and you will corrupt something. As I told, if you think directly in GB instead of MB for disk sizes, you are safer because GB is always compatible with 16MB.
 
This post was published in my personal blog too: http://www.fernandosimon.com/blog/shrink-asm-diskgroup-and-exadata-grid-disks/
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications were removed to allow reach the generic audience and to be useful for the community.”


Increase Size For Exadata Grid Disks
Category: Engineer System Author: Fernando Simon (Board Member) Date: 5 years ago Comments: 0

Increase Size For Exadata Grid Disks

A quick article about a maintenance task for Oracle Exadata when you are using OVM and you divided your storage cell disks for every VM. Here I will show you how to extend your Grid Disks to add more space in your ASM diskgroup.
The first thing is being aware of your environment, before everything you need to know the points below because, they are important to calculate the new space, and to avoid do something wrong:
  • Number of cells in your appliance.
  • Number of disks for each cell.
  • Mirroring for your ASM.
  • The VM that you want to add the space.
The “normal” Exadata storage cell has 12 disks, the Extreme Flash version uses 8 disks per storage. If you have doubt about how many disks you have per storage cell, you can connect in each one and check the number of celldisks you have. And before continuing, be aware of Exadata disk division:

 

 

To do this change we execute three major steps: ASM, Exadata Storage, and ASM again.

 

For ASM

 

Inside ASM we can use this query to collect some information about the diskgroups:

 

SQL> col name format a12 head 'Disk Group';

SQL> col total_mb format 999999999 head 'Total GB|Raw';

SQL> col free_mb format 999999999 head 'Free GB|Raw';

SQL> col avail_mb format 999999999 head 'Total GB|Usable';

SQL> col usable_mb format 999999999 head 'Free GB|Usable';

SQL> col usable_mb format 999999999 head 'Free GB|Usable';

SQL> col cdisks format 99999 head 'Cell|Disksl';

SQL>

SQL> select a.name,a.total_mb,a.free_mb,a.type,

  2  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) avail_mb,

  3  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024) usable_mb,

  4  count(b.path) cdisks

  5  from v$asm_diskgroup a, v$asm_disk b

  6  where a.group_number=b.group_number

  7  group by a.name,a.total_mb,a.free_mb,a.type,

  8  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) ,

  9  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024)

 10  order by 2,1

 11  /




               Total GB    Free GB          Total GB    Free GB   Cell

Disk Group          Raw        Raw TYPE       Usable     Usable Disksl

------------ ---------- ---------- ------ ---------- ---------- ------

RECOC3          4239360    2465540 NORMAL       2070       1204     60

DATAC3         15790080    2253048 NORMAL       7710       1100     60




SQL>

SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC3')

  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status

  6  order by 1,2,3;




NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS

------ ------------------------------ -------- ------------ ------- ------- ----------

DATAC3 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC3 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12




SQL>

 

 

With that, I have three important information: number the disks (60), redundancy type (NORMAL), total actual size (RAW value – 15790080). To discover the size for each disk in ASM (here I do manually and not check in v$asm_disk just to show you the steps and to be more didact) you can divide the raw space/#disks:

 

SQL> SELECT (15790080/1024)/60 as gbDISK FROM dual; 

GBDISK
----------       
257 

SQL>

 

So, each disk has 257GB of space size (in raw). Since the actual free space is 1100GB (1.07TB) and we want to add more 2TB we need to increase the value for each disk.

 

The formula is simple: NewValue(inGB)*#OfDisksPerCell*#NumberofCells. Here I choose 330GB per disk, so, the new size for diskgroup will be:

 

SQL> SELECT (330*12*5) AS newsizeGB FROM dual;  

NEWSIZEGB
----------     
19800 

SQL>

 

But this value is not correct because does not consider the mirror type, so, we need to divide this value. If it NORMAL, divide by 2, if HIGH, divide by 3. To compare, the old and new expected value:

 

SQL> SELECT (257*12*5)/2 as actualsizeGBUsable, (330*12*5)/2 AS newsizeGBUSable FROM dual;

ACTUALSIZEGBUSABLE NEWSIZEGBUSABLE
------------------ ---------------    
          7710            9900 

SQL>


 

So, the new total space for diskgroup will be around 9.6 TB (9900 GB). And we will add (as free space) around 2.1 TB. Probably you need to execute these formulas more than one time to find the desired size per disk.

 

I start to calculate by disk (and after discovering the final diskgroup size) instead of starting with diskgroup size (and dividing to discover the disk size) because doing this way, the size for disk will be always correct and align with storage cell grid disk. Remember that grid disks are aligned in 16MB and, if you start to choose one arbitrary value to the max size for the ASM diskgroup, you can reach a value per grid disk that is not 16MB aligned. As an example, if I start choosing 20TB for diskgroup, the size per disk will be (20*1024)/60 = 341.33GB and this is not aligned with 16MB.

 

For 16 Mb explanation, you can check in the Exadata docs: 

 

Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.

 

For Storage Cell

 

In the Exadata side, first, check some info about the actual state for grid disks. Here I connect in one cell (if you want you can use dcli to call every/all cells) and check some info for the grid disk:

 

 

[root@exaceladm01 ~]# cellcliCellCLI

Release 18.1.6.0.0 - Production on Fri Jun 21 16:57:49 CEST 2019 Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved. 

CellCLI> list celldisk

CD_00_exaceladm01       normal  
CD_01_exaceladm01       normal         
CD_02_exaceladm01       normal         
CD_03_exaceladm01       normal         
CD_04_exaceladm01       normal         
CD_05_exaceladm01       normal         
CD_06_exaceladm01       normal         
CD_07_exaceladm01       normal         
CD_08_exaceladm01       normal         
CD_09_exaceladm01       normal         
CD_10_exaceladm01       normal         
CD_11_exaceladm01       normal         
FD_00_exaceladm01       normal         
FD_01_exaceladm01       normal         
FD_02_exaceladm01       normal
FD_03_exaceladm01       normal

CellCLI> list celldisk 
CD_02_exaceladm01 detail

name:                   CD_02_exaceladm01         
comment:         
creationTime:           2016-11-29T10:23:35+01:00         
deviceName:             /dev/sdc         
devicePartition:        /dev/sdc         
diskType:               HardDisk         
errorCount:             0         
freeSpace:              2.6688079833984375T         
id:                     d57b31bb-6043-4cea-b992-ef8075f42e77         
physicalDisk:           PUT81V         
size:                   7.152252197265625T         
status:                 normal


CellCLI> 

CellCLI> list griddisk where name like 'DATAC3.*';

DATAC3_CD_00_exaceladm01        active         
DATAC3_CD_01_exaceladm01        active         
DATAC3_CD_02_exaceladm01        active         
DATAC3_CD_03_exaceladm01        active         
DATAC3_CD_04_exaceladm01        active         
DATAC3_CD_05_exaceladm01        active         
DATAC3_CD_06_exaceladm01        active         
DATAC3_CD_07_exaceladm01        active         
DATAC3_CD_08_exaceladm01        active         
DATAC3_CD_09_exaceladm01        active         
DATAC3_CD_10_exaceladm01        active         
DATAC3_CD_11_exaceladm01        active

CellCLI>

CellCLI> list griddisk where name = 'DATAC3_CD_04_exaceladm01' detail;

name:                   DATAC3_CD_04_exaceladm01         
asmDiskGroupName:       DATAC3         
asmDiskName:            DATAC3_CD_04_EXACELADM01         
asmFailGroupName:       EXACELADM01         
availableTo:         
cachedBy:               FD_00_exaceladm01         
cachingPolicy:          default         
cellDisk:               CD_04_exaceladm01         
comment:                "Cluster exa-cl3 diskgroup DATAC3"         
creationTime:           2017-01-20T17:23:21+01:00         
diskType:               HardDisk         
errorCount:             0         
id:                     2cb2aecb-cfa1-4282-b90d-3a08ed079778         
size:                   257G         
status:                 active

CellCLI>


Here you can see that I checked:
  •  The celldisks info for this cell.
  •  Detail for one celldisk (look the freeSpaceattribute to verify if you have free space).
  •  The grid disks for this cell.
  •  Details for the griddisk (look that the size is the same value that I calculated manually).
This part was just to check and show you how to verify some info, with time, you don’t need to check this in every maintenance (because you will be familiar with the environment). Be careful that, if you have different grid disk space division per storage cells, you need to check if you have available space in all your storage celldisks.

 

To expand the grid disks you have two options, enter in each cell and expand manually one by one, or create one script and call by dcli (the option that I choose). So, create one script that executes the ALTER GRIDDISK command for the new desired size. Just remember to be careful and choose the correct grid disks (here is for VM 03, that means DATAC3):

 

DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC3_CD_00_EXACELADM01 size=330G;

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC3_CD_00_EXACELADM02 size=330G;

…

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM02 size=330G;

dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM03 size=330G;

dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM04 size=330G;

dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC3_CD_11_EXACELADM05 size=330G;

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC3_Cluster_To_330G.sh

exaceladm01: GridDisk DATAC3_CD_00_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC3_CD_00_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC3_CD_00_exaceladm03 successfully altered

…

…

exaceladm01: GridDisk DATAC3_CD_11_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC3_CD_11_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC3_CD_11_exaceladm03 successfully altered

exaceladm04: GridDisk DATAC3_CD_11_exaceladm04 successfully altered

exaceladm05: GridDisk DATAC3_CD_11_exaceladm05 successfully altered

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$




Above I cropped the output to reduce the size or post, but you can check the raw output here. After the change you can check the info for the grid disk:

 

[root@exaceladm01 ~]# cellcli

CellCLI: Release 18.1.6.0.0 - Production on Fri Jun 21 17:24:03 CEST 2019




Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved.




CellCLI> list griddisk where name = 'DATAC3_CD_04_exaceladm01' detail;

         name:                   DATAC3_CD_04_exaceladm01

         asmDiskGroupName:       DATAC3

         asmDiskName:            DATAC3_CD_04_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_04_exaceladm01

         comment:                "Cluster exa-cl3 diskgroup DATAC3"

         creationTime:           2017-01-20T17:23:21+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     2cb2aecb-cfa1-4282-b90d-3a08ed079778

         size:                   330G

         status:                 active




CellCLI> list celldisk CD_02_exaceladm01 detail

         name:                   CD_02_exaceladm01

         comment:

         creationTime:           2016-11-29T10:23:35+01:00

         deviceName:             /dev/sdc

         devicePartition:        /dev/sdc

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              2.5975189208984375T

         id:                     d57b31bb-6043-4cea-b992-ef8075f42e77

         physicalDisk:           PUT81V

         size:                   7.152252197265625T

         status:                 normal




CellCLI> exit

quitting




[root@exaceladm01 ~]#


For ASM – Part #2
After you change the grid disks in the storage side, you can go back to ASM and extend the diskgroup:

 

SQL> ALTER DISKGROUP DATAC3 RESIZE ALL;




Diskgroup altered.




SQL>


And you can check that the size was already added (look that values hit what we calculated before):

 

SQL> select a.name,a.total_mb,a.free_mb,a.type,

  2  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) avail_mb,

  3  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024) usable_mb,

  4  count(b.path) cdisks

  5  from v$asm_diskgroup a, v$asm_disk b

  6  where a.group_number=b.group_number

  7  group by a.name,a.total_mb,a.free_mb,a.type,

  8  decode(a.type,'NORMAL',a.total_mb/2/1024,'HIGH',a.total_mb/3/1024) ,

  9  decode(a.type,'NORMAL',a.free_mb/2/1024,'HIGH',a.free_mb/3/1024)

 10  order by 2,1

 11  /




               Total GB    Free GB          Total GB    Free GB   Cell

Disk Group          Raw        Raw TYPE       Usable     Usable Disksl

------------ ---------- ---------- ------ ---------- ---------- ------

RECOC3          4239360    2465540 NORMAL       2070       1204     60

DATAC3         20275200    6738128 NORMAL       9900       3290     60




SQL>


And you can check the v$asm_operation to check the rebalance progress:

 

SQL> select operation, EST_MINUTES, EST_RATE, EST_WORK, sofar from v$asm_operation;




OPERA EST_MINUTES   EST_RATE   EST_WORK      SOFAR

----- ----------- ---------- ---------- ----------

REBAL           0          0          0          0

REBAL           0      46158      18490       5761

REBAL           0          0          0          0

REBAL           0          0          0          0

REBAL           0          0          0          0




SQL>




Conclusion
As you can see, the steps to do that are simple and not complex, you just need to take care about some details of your environment: Number of the disks per cell, number the cells and the VM where you want to add the space is critical to do the correct change. Remember to align with 16MB the size of your grid disk, when you are adding it is not a big deal, but if you want to shrink this can break your ASM diskgroup.
Check that the only change effectively is the size of the grid disk, all the others occur automatically because of the grid disk. ASM diskgroup will increase to the max value that it is available and space is available just after the command.
The steps above are more detailed that you will do in daily maintenance, but help you to understand most of the datils for this kind of change.
 
Reference:
How to Resize Grid Disks in Exadata (Doc ID 2176737.1) – https://support.oracle.com/epmos/faces/DocContentDisplay?id=2176737.1
Resizing Grid Disks – https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-570A0C37-907C-4417-BC93-AC4ABAF7E3AD 

 

This post is published in my personal blog too: http://www.fernandosimon.com/blog/increase-size-for-exadata-grid-disks/
 

Disclaimer: “The postings on this site are my own and don’t necessarily represent my actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific


Login failed due to ORA-28030: ORA-28030: Server encountered problems accessing LDAP directory service
Category: Database Author: Rodrigo Mufalani (Board Member) Date: 5 years ago Comments: 0

Login failed due to ORA-28030: ORA-28030: Server encountered problems accessing LDAP directory service

 
 
Today I noticed in my environment a lot of messages ORA-28030 in the alert.log of the one database version 18c, previously configured to authenticate on OID (Oracle Internet Directory), event with removal of the configuration, one parameter still configured and that was the reason of the messages. 
Also, I checked on listener.log and no failures occurred.
 
I try to find useful information regargind this error, but just found issues on the oposite direction, people configuring authentication on OID, our issue was cause because of removal, not completely, the configuration of OID.
 
So, I decided to write this small post about the issue and how to solve.
 
Before, one of the steps to configure OID is setting the parameter LDAP_DIRECTORY_ACCESS = PASSWORD, so one of my coleagues have done that:
 
alter system set LDAP_DIRECTORY_ACCESS = 'PASSWORD' scope=both sid='*';
— Note that this parameter is dynamic
 
To solve the issue, I issue the parameter as default:
alter system set LDAP_DIRECTORY_ACCESS = 'NONE' scope=both sid='*';
Immediately, after the command, no more messages “Login failed due to ORA-28030: ORA-28030: Server encountered problems accessing LDAP directory service”. 
 
I hope this helps you to avoid this messages when you are removing OID authentication of one database.
 
You can also find more useful information on
 
Deinstall
https://docs.oracle.com/en/middleware/lifecycle/12.2.1.3/inoim/uninstalling-or-reinstalling-product.html#GUID-3FC6A4E1-4684-4BAC-9760-DA308A114FE7
Install
https://docs.oracle.com/en/middleware/lifecycle/12.2.1.3/inoim/installing-product-software.html#GUID-5D47117E-3983-4BED-BADB-CB7DDD63DB34
 
All the Best,
Rodrigo Mufalani

Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited 
to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”

 


How to apply patch (PSU) in Oracle Enterprise Manager 13c
Category: Database Author: Leonardo Santos Lopes Date: 6 years ago Comments: 0

How to apply patch (PSU) in Oracle Enterprise Manager 13c

Today in this article I will show you how to apply patch (PSU) in Oracle Enterprise Manager 13c (OEM13c).

 

Before start:
export ORACLE_HOME=/u01/software/em/middleware

export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/OMSPatcher
 
1 – Download the latest PSU in the MOS (My Oracle Support) and upload the .zip file to the OMS server:

 

– Enterprise Manager 13.2 Master Bundle Patch List (Includes Plugins: 13.2.1, 13.2.2, 13.2.3) (Doc ID 2219797.1)
 
2 – Download the required OPatch and OMSPatcher version as describe in the README of the PSU: (13.9.0.0.0)
 
OPatch:
– https://updates.oracle.com/download/6880880.html

 

OMSPatcher:
– Patch 19999993: OMSPatcher patch of version 13.8.0.0.2 for Enterprise Manager Cloud Control 13.2.0.0.0
 
3 – Unzip the patch for the OPatch and apply:
unzip p6880880_139000_Generic.zip -d /home/oracle/6880880
 
cd /home/oracle/6880880
 
$ORACLE_HOME/oracle_common/jdk/bin/java -jar /home/oracle/6880880/opatch_generic.jar -silent oracle_home=$ORACLE_HOME

 

4 – Unzip the new OMSPatcher version:
unzip p19999993_132000_Generic.zip -d /home/oracle/OMSPatcher/
 
mv /home/oracle/OMSPatcher/ /u01/software/em/middleware/
 
mv /u01/software/em/middleware/OMSPatcher/
 
/u01/software/em/middleware/OMSPatcher_old20180427

 

5 – Validate the OPatch and OMSPatcher version:
  $ORACLE_HOME/OPatch/opatch version



e.g.:
OPatch Version: 13.9.3.2.0

OPatch succeeded.
 
$ORACLE_HOME/OMSPatcher/omspatcher version



e.g.:
OMSPatcher Version: 13.8.0.0.2

 OPlan Version: 12.2.0.1.6

 OsysModel build: Wed Mar 21 18:20:48 PDT 2018

 OMSPatcher succeeded.

 

6 – Apply patch in the OPatch agent:
export ORACLE_HOME=/u01/software/em/agent/agent_13.2.0.0.0
 
$ORACLE_HOME/jdk/bin/java -jar /home/oracle/6880880/opatch_generic.jar -silent oracle_home=$ORACLE_HOME
 
$ORACLE_HOME/OPatch/opatch version



e.g.:
OPatch Version: 13.9.3.2.0

OPatch succeeded.
7 – Generating the property file:
export ORACLE_HOME=/u01/software/em/middleware
 
cd $ORACLE_HOME/OMSPatcher/wlskeys
 
$ORACLE_HOME/OMSPatcher/wlskeys/createkeys.sh -oh $ORACLE_HOME -location
 
$ORACLE_HOME/OMSPatcher/wlskeys

 

PS: You will be prompted to enter the credentials for the user: weblogic (Admin Server)

 

8 – Unzip the file and check prerequisites, as oracle:
unzip p27612395_132000_Generic.zip -d /home/oracle/27612395
 
cd /home/oracle/27612395
 
omspatcher apply -analyze -property_file /u01/software/em/middleware/OMSPatcher/wlskeys/properties OMSPatcher.OMS_DISABLE_HOST_CHECK=true

 

PS: DO NOT copy and paste the “omspatcher” command, it does not support. You must type each command manually.

 

9 – Stopping and Applying the patch in the OMS Server:
$ORACLE_HOME/bin/emctl stop oms
 
omspatcher apply -property_file /u01/software/em/middleware/OMSPatcher/wlskeys/properties OMSPatcher.OMS_DISABLE_HOST_CHECK=true

 

10 – Start OMS Server after apply patch:
$ORACLE_HOME/bin/emctl start oms

 

 

 

 

See you in the next post!
 

 

Leonardo Santos Lopes – https://leonardosantoslopes.wordpress.com/
 

 

   

 Disclaimer: “The postings on this site are my own and don’t necessarily represent may actual employer positions, strategies or opinions. The information here was edited to be useful for general purpose, specific data and identifications was removed to allow reach generic audience and to be useful.”


1 … 25 26 27 28 29 … 32
LUXOUG – LUXEMBOURG ORACLE USERS GROUP
© 2024 LUXOUG – LUXEMBOURG ORACLE USERS GROUP