We had to migrate some databases to a new Cluster, which involved moving all the disks associated with ASM.
Everything was going well: Oracle recognized the Disk Groups, we proceeded to mount them and open the databases, but it was also necessary to mount an ACFS filesystem, so as a first step I proceeded to try to register it and that’s when the adventure begins.
The problem
To use an ACFS filesystem, the first step is to create an ADVM volume in a Disk Group, so we start by verifying its existence.
ASMCMD> volinfo --all
ASMCMD-9470: ASM proxy instance unavailable
ASMCMD-9473: volume STATE will show as REMOTE
Diskgroup Name: ACFS
Volume Name: BACKUP
Volume Device: /dev/asm/backup-166
State: REMOTE
Size (MB): 18874368
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath:Other than messages ASMCMD-9470 and ASMCMD-9473, everything seemed to be in order, we have the BACKUP volume in the Disk Group +ACFS, so we proceed to register the ACFS filesystem associated with the found volume.
# /sbin/acfsutil registry -f -a /dev/asm/backup-166 /mnt/backup
PRCA-1051 : No volume resource registered with CRS for specified diskgroup or volume name or volume device
acfsutil registry: ACFS-09173: The srvctl command line "/u01/app/grid/19.0.0/grid_1/bin/srvctl add filesystem -device /dev/asm/backup-166 -path /mnt/backup" failed to execute. (0)
acfsutil registry: ACFS-03111: unable to add ACFS mount /mnt/backup within Oracle Registry
Oracle is unable to complete our request, pointing out that the required ADVM volume is not registered with the CRS, which we verified.
# crsctl stat res ora.ACFS.BACKUP.advm -t
CRS-2613: Could not find resource 'ora.ACFS.BACKUP.advm'. The solution
First, we must resolve the error messages ASMCMD-9470 and ASMCMD-9473. To do this, we check the status of the ASM proxy, and as we find it OFFLINE, we will have to enable it.
# crsctl stat res ora.proxy_advm -t
-------------------------------------------------------
Name Target State Server State details
-------------------------------------------------------
Local Resources
-------------------------------------------------------
ora.proxy_advm
OFFLINE OFFLINE node1 STABLE
OFFLINE OFFLINE node2 STABLE
OFFLINE OFFLINE node3 STABLE
-------------------------------------------------------
# srvctl enable asm -proxy
# srvctl start asm -proxy
# srvctl status asm -proxy
ADVM proxy is running on node node1,node2,node3
# crsctl stat res ora.proxy_advm -t
-----------------------------------------------------
Name Target State Server State details
-----------------------------------------------------
Local Resources
-----------------------------------------------------
ora.proxy_advm
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ONLINE ONLINE node3 STABLE
-----------------------------------------------------Then we check that the error messages ASMCMD-9470 and ASMCMD-9473 have disappeared.
ASMCMD> volinfo --all
Diskgroup Name: ACFS
Volume Name: BACKUP
Volume Device: /dev/asm/backup-166
State: ENABLED
Size (MB): 18874368
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath:Now we must register the ADVM volume with the CRS.
# srvctl add volume -volume BACKUP -diskgroup ACFS -device /dev/asm/backup-166
# crsctl stat res ora.ACFS.BACKUP.advm -t
-----------------------------------------------------
Name Target State Server State details
-----------------------------------------------------
Local Resources
-----------------------------------------------------
ora.ACFS.BACKUP.advm
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ONLINE ONLINE node3 STABLE
------------------------------ ------ ---------------All we have left is to register the ACFS filesystem.
# /sbin/acfsutil registry -f -a /dev/asm/backup-166 /mnt/backup
acfsutil registry: mount point /mnt/backup successfully added to Oracle Registry
# crsctl stat res ora.acfs.backup.acfs -t
----------------------------------------------------------
Name Target State Server State details
----------------------------------------------------------
Local Resources
----------------------------------------------------------
ora.acfs.backup.acfs
ONLINE ONLINE node1 mounted on /mnt/back
up,STABLE
ONLINE ONLINE node2 mounted on /mnt/back
up,STABLE
ONLINE ONLINE node2 mounted on /mnt/back
up,STABLE
----------------------------------------------------------With this, the incident is resolved: the ACFS filesystem is now mounted and its contents are available for use by the applications.
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 252G 0 252G 0% /dev
tmpfs 252G 1.6G 251G 1% /dev/shm
tmpfs 252G 1.4G 251G 1% /run
tmpfs 252G 0 252G 0% /sys/fs/cgroup
/dev/mapper/rootvg-root 50G 4.9G 42G 11% /
/dev/sda2 1014M 178M 837M 18% /boot
/dev/sda1 200M 8.3M 192M 5% /boot/efi
/dev/mapper/rootvg-home 50G 1.1G 46G 3% /home
/dev/mapper/rootvg-var 99G 2.9G 91G 4% /var
/dev/mapper/rootvg-u01 193G 76G 108G 42% /u01
tmpfs 51G 0 51G 0% /run/user/0
tmpfs 51G 0 51G 0% /run/user/10000
tmpfs 51G 0 51G 0% /run/user/1029
/dev/asm/backup-166 18T 13T 5.5T 70% /mnt/backup
To complement the above, I recommend reading the notes:
| 2473506.1 | ASM-Proxy Offline [ora.proxy_advm (OFFLINE OFFLINE)] After Upgrade To 12.2/19c |
| 3010189.1 | Unable To Mount ACFS Filesystems Post Migration To A New RAC Cluster |
