Not long ago, one of my customers started the migration process of their disk storage system, from EMC CX4 to EMC VNX. They have their databases operating with Oracle RAC 11.2.0.3, so they followed the procedure of adding LUNs from the new EMC VNX in the data diskgroups, and then removing the LUNs from the old EMC CX4, taking advantage of the automatic rebalancing that ASM does when adding and removing disks.
So far so good, but there was still the work on the diskgroup that contains the ocr, voting disks and spfile of ASM, called in this case +OCR, for which the procedure is not as simple as adding and removing disks.
Next I will show you in detail the procedure that I followed to conclude with this last step in the migration of disk storage system for this diskgroup, fulfilling the express requirement of the client: not to suspend the service.
The diskgroup +OCR was created with normal redundancy, so it is a mandatory requirement to use 3 disks. The goal is that its contents are now moved to 3 disks in the new disk array, but this cannot be done directly as in the case of a data diskgroup, basically because of the voting disks.
The ocr and the spfile of ASM are files that are distributed among the disks that constitute the diskgroup on which they reside, but Oracle creates an identical voting disk in each disk of the diskgroup. Thus, when adding more disks to the diskgroup, the ocr and spfile are candidates to be redistributed among all the disks, but the voting disks are not, they stay on their original disks, so they require a special treatment.
The note 428681.1, OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE), has the details of what to do in different scenarios, from which it is clear that it is not possible to add more voting disks, but only to move them to another diskgroup. For this reason, the creation of 3 new LUNs was requested, on which the diskgroup that would temporarily house the voting disks would be created, so that it would look something like this:
After the LUNs were provisioned and duly assigned to all the RAC nodes, we proceeded to the creation of the diskgroup VDISK, using the utility ASMCA.
With this we are now ready for the initial step: the movement of voting disks.
[oracle@rac1]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1e56372cc94a4f92bfd9952beef36a5b (/dev/rhdiskpower3) [OCR]
2. ONLINE 570dd695f7ea4f08bf86d92adc2a2343 (/dev/rhdiskpower1) [OCR]
3. ONLINE 027f719d79914f72bfc1fdfb3098209b (/dev/rhdiskpower2) [OCR]
Located 3 voting disk(s).
[oracle@rac1]$ crsctl replace votedisk +VDISK
Successful addition of voting disk 42cd3a1355714febbf8b8be58da5d277.
Successful addition of voting disk 95b0b904a7844f6bbf35881a7fd545f6.
Successful addition of voting disk 189d6f4cba754fc6bf1d8f94fd70fe89.
Successful deletion of voting disk 1e56372cc94a4f92bfd9952beef36a5b.
Successful deletion of voting disk 570dd695f7ea4f08bf86d92adc2a2343.
Successful deletion of voting disk 027f719d79914f72bfc1fdfb3098209b.
Successfully replaced voting disk group with +VDISK.
CRS-4266: Voting file(s) successfully replaced
[oracle@rac1]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 42cd3a1355714febbf8b8be58da5d277 (/dev/rhdiskpower57) [VDISK]
2. ONLINE 95b0b904a7844f6bbf35881a7fd545f6 (/dev/rhdiskpower130) [VDISK]
3. ONLINE 189d6f4cba754fc6bf1d8f94fd70fe89 (/dev/rhdiskpower90) [VDISK]
Located 3 voting disk(s).Supported by the crsctl utility, we have moved the voting disks from diskgroup +OCR to the diskgroup +VDISK, being the current scenario:
Now that the +OCR diskgroup contains only the ocr and the ASM spfile, it is possible to rebalance ASM, so we proceed to add the LUNs provisioned in the new disk array, using again the ASMCA utility.
After adding the 3 LUNs of the new disk array, the configuration is:
It is now possible to remove the disks from the old disk array, always supported with ASMCA.
With the old LUNs already removed, the new situation is:
Now that the contents of the +OCR diskgroup reside exclusively on LUNs of the new disk array, it is time to return the voting disks to their original diskgroup.
[oracle@rac1]$ crsctl replace votedisk +OCR
Successful addition of voting disk 5385fdac3eb44f96bf206fc7cce7fe48.
Successful addition of voting disk a4a5d6afbd064fa2bffe306654b0f3ac.
Successful addition of voting disk 2cbc13a0ff444fd7bf79670b75fe42c4.
Successful deletion of voting disk 42cd3a1355714febbf8b8be58da5d277.
Successful deletion of voting disk 95b0b904a7844f6bbf35881a7fd545f6.
Successful deletion of voting disk 189d6f4cba754fc6bf1d8f94fd70fe89.
Successfully replaced voting disk group with +OCR.
CRS-4266: Voting file(s) successfully replaced
[oracle@rac1]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5385fdac3eb44f96bf206fc7cce7fe48 (/dev/rhdiskpower1002) [OCR]
2. ONLINE a4a5d6afbd064fa2bffe306654b0f3ac (/dev/rhdiskpower1001) [OCR]
3. ONLINE 2cbc13a0ff444fd7bf79670b75fe42c4 (/dev/rhdiskpower1000) [OCR]
Located 3 voting disk(s).We have finally arrived at the following scenario:
At this point we already have both the ocr, the spfile and the voting disks in the diskgroup +OCR, using only LUNs from the new EMC VNX disk storage system, so it is possible to remove the diskgroup +VDISK, we don’t need it anymore.
With this we can conclude the task of moving the contents of the diskgroup +OCR (ocr, voting disks, ASM spfile), to the new EMC VNX disk array, without any suspension of service and in a few minutes.
Of course, the procedure was previously tested in a virtualized environment, because one thing is what the documentation says and another what happens in real life, so it is highly recommended that you also do it, remember that each installation has its particularities, so it is better not to take unnecessary risks. See you next time friends!