We had to migrate some databases to a new Cluster, which involved moving all the disks associated to ASM.
Everything was going well: Oracle recognized the Disk Groups, we proceeded to mount them and open the databases, but it was also necessary to mount an ACFS filesystem, so as a first step I proceeded to try to register it and that’s when the adventure begins.
The problem
In order to use an ACFS filesystem, the first step is to create an ADVM volume in a Disk Group, so we start by verifying its existence.
ASMCMD> volinfo --all
ASMCMD-9470: ASM proxy instance unavailable
ASMCMD-9473: volume STATE will show as REMOTE
Diskgroup Name: ACFS
Volume Name: BACKUP
Volume Device: /dev/asm/backup-166
State: REMOTE
Size (MB): 18874368
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath:Other than messages ASMCMD-9470 and ASMCMD-9473, everything seemed to be in order, we have the BACKUP volume in the Disk Group +ACFS, so we proceed to register the ACFS filesystem associated with the found volume.
# /sbin/acfsutil registry -f -a /dev/asm/backup-166 /mnt/backup
PRCA-1051 : No volume resource registered with CRS for specified diskgroup or volume name or volume device
acfsutil registry: ACFS-09173: The srvctl command line "/u01/app/grid/19.0.0/grid_1/bin/srvctl add filesystem -device /dev/asm/backup-166 -path /mnt/backup" failed to execute. (0)
acfsutil registry: ACFS-03111: unable to add ACFS mount /mnt/backup within Oracle Registry
Oracle is unable to complete our request, pointing out that the required ADVM volume is not registered with the CRS, which we verified.
# crsctl stat res ora.ACFS.BACKUP.advm -t
CRS-2613: Could not find resource 'ora.ACFS.BACKUP.advm'. The solution
First we must resolve the error messages ASMCMD-9470 and ASMCMD-9473. To do this we check the status of the ASM proxy, and as we find it OFFLINE we will have to enable it.
# crsctl stat res ora.proxy_advm -t
-------------------------------------------------------
Name Target State Server State details
-------------------------------------------------------
Local Resources
-------------------------------------------------------
ora.proxy_advm
OFFLINE OFFLINE node1 STABLE
OFFLINE OFFLINE node2 STABLE
OFFLINE OFFLINE node3 STABLE
-------------------------------------------------------
# srvctl enable asm -proxy
# srvctl start asm -proxy
# srvctl status asm -proxy
ADVM proxy is running on node node1,node2,node3
# crsctl stat res ora.proxy_advm -t
-----------------------------------------------------
Name Target State Server State details
-----------------------------------------------------
Local Resources
-----------------------------------------------------
ora.proxy_advm
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ONLINE ONLINE node3 STABLE
-----------------------------------------------------Then we check that the error messages ASMCMD-9470 and ASMCMD-9473 have disappeared.
ASMCMD> volinfo --all
Diskgroup Name: ACFS
Volume Name: BACKUP
Volume Device: /dev/asm/backup-166
State: ENABLED
Size (MB): 18874368
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath:Now we must register the ADVM volume with the CRS.
# srvctl add volume -volume BACKUP -diskgroup ACFS -device /dev/asm/backup-166
# crsctl stat res ora.ACFS.BACKUP.advm -t
-----------------------------------------------------
Name Target State Server State details
-----------------------------------------------------
Local Resources
-----------------------------------------------------
ora.ACFS.BACKUP.advm
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ONLINE ONLINE node3 STABLE
------------------------------ ------ ---------------All we have left is to register the ACFS filesystem.
# /sbin/acfsutil registry -f -a /dev/asm/backup-166 /mnt/backup
acfsutil registry: mount point /mnt/backup successfully added to Oracle Registry
# crsctl stat res ora.acfs.backup.acfs -t
----------------------------------------------------------
Name Target State Server State details
----------------------------------------------------------
Local Resources
----------------------------------------------------------
ora.acfs.backup.acfs
ONLINE ONLINE node1 mounted on /mnt/back
up,STABLE
ONLINE ONLINE node2 mounted on /mnt/back
up,STABLE
ONLINE ONLINE node2 mounted on /mnt/back
up,STABLE
----------------------------------------------------------With this, the incident is resolved: the ACFS filesystem is now mounted and its contents are available for use by the applications.
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 252G 0 252G 0% /dev
tmpfs 252G 1.6G 251G 1% /dev/shm
tmpfs 252G 1.4G 251G 1% /run
tmpfs 252G 0 252G 0% /sys/fs/cgroup
/dev/mapper/rootvg-root 50G 4.9G 42G 11% /
/dev/sda2 1014M 178M 837M 18% /boot
/dev/sda1 200M 8.3M 192M 5% /boot/efi
/dev/mapper/rootvg-home 50G 1.1G 46G 3% /home
/dev/mapper/rootvg-var 99G 2.9G 91G 4% /var
/dev/mapper/rootvg-u01 193G 76G 108G 42% /u01
tmpfs 51G 0 51G 0% /run/user/0
tmpfs 51G 0 51G 0% /run/user/10000
tmpfs 51G 0 51G 0% /run/user/1029
/dev/asm/backup-166 18T 13T 5.5T 70% /mnt/backup
To complement the above, I recommend reading the notes:
| 2473506.1 | ASM-Proxy Offline [ora.proxy_advm (OFFLINE OFFLINE)] After Upgrade To 12.2/19c |
| 3010189.1 | Unable To Mount ACFS Filesystems Post Migration To A New RAC Cluster |
