Troubleshoot: CRS-2304 GPnP profile signature verification failed.
In the previous article I told you that I am developing a project to upgrade Oracle Database Server 11.2 to 19c, and after correcting the permissions problem, everything seemed to be in order.
The first step was to have the databases running Oracle 11.2, and it was at this point that an error message in the alert.log file caught my attention.
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 2
Number of processor cores in the system is 2
Number of processor sockets in the system is 1
2024-10-06 09:50:16.306:
[USER(21480)]CRS-2304:GPnP profile signature verification failed. get-profile request aborted.
WARNING: No cluster interconnect has been specified. Depending on
the communication driver configured Oracle cluster traffic
may be directed to the public interface of this machine.
Oracle recommends that RAC clustered databases be configured
with a private interconnect for enhanced security and
performance.
Picked latch-free SCN scheme 3
. . .
pga_aggregate_target = 128M
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
192.168.0.61
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2When starting the instance, it was not possible to obtain the network data to use for the interconnect, so Oracle opted to use the public network for this purpose.
This did not prevent the database from starting up, but it was definitely an irregular situation that I had not encountered in previous projects.
The problem
Let’s start by remembering that in every Oracle Cluster there are 2 networks: a public and a private (interconnect) one, with different uses and properties.
The alert file records the use of the address 192.168.1.61 for interconnect purposes, when it should be 169.254.13.3, which can be verified by consulting v$cluster_interconnects.
SQL> select * from v$cluster_interconnects;
NAME IP_ADDRESS IS_PUBLIC SOURCE
--------------- -------------------- --------- ----------------------------------------
eth1 192.168.0.61 OS dependent software
First attempt
| 3004140.1 | Oracle Server Processes Terminated By Fatal Error ORA-27504 In 11.2 DB Managed By 19c GI Cluster |
It indicates that this problem occurs when an Oracle RAC 11.2 database operates under Oracle Grid 19c, since under this combination the Oracle 11.2 instance fails to verify the GPnP 19c (Grid Plug and Play) profile, making it necessary to apply the patch 32967645.
Unfortunately Upgrade Support (formerly Market Driven Support) is required to download it.
It is also mentioned that there is a workaround, which is to manually set the addresses using the cluster_interconnects parameter, for each instance.
While it sounds tempting, it does not seem to be the ultimate solution, after all this problem has not been presented to me before in similar projects.
What is different about this installation?
Second attempt
| 2985244.1 | GPNP reports gpnp call get-profile got invalid profile error |
It states that due to “internal requirements of the security team”, the algorithm used to verify the GPnP profile has been updated from Oracle 19.16.
This data solves our question: this installation was born directly with Oracle 19.24, therefore it is subject to the new algorithm, and this prevents the 11.2 instances from being able to obtain the address data for interconnect.
In the same note there is a procedure to follow to manually update the algorithm, since for installations prior to 19.16 the old and insecure algorithm is in use and it is recommended to update it to the new and secure algorithm.
But our case is different, we already have the new algorithm in use and we want to go back to the old algorithm. Is it possible? Let’s try.
As indicated in the note, the new algorithm is sha512, and checking an old installation shows that the previous algorithm is sha1. That’s the discrepancy that has brought us here!
We will follow the procedure detailed in the note, but instead of updating the validation algorithm from sha1 to sha512, we will update from sha512 to sha1.
export GI_HOME=/u01/app/grid/19.0.0/grid_1
-- get the current profile sequence number
[oracle@node1 ~]$ gpnptool getpval -prf_sq \
-p=${GI_HOME}/gpnp/node1/profiles/peer/profile.xml
5
-- increment the sequence number by 5
[oracle@node1 ~]$ gpnptool edit -prf_sq=10 \
-p=${GI_HOME}/gpnp/node1/profiles/peer/profile.xml \
-o=/tmp/profile_sqenum.xml
-- sign the profile with the old antiguo (sha1)
[oracle@node1 ~]$ gpnptool sign -a=sha1 \
-p=/tmp/profile_sqenum.xml -o=/tmp/profile_signed.xml
Warning: sha1 algorithm is deprecated.
-- push the profile to all cluster nodes
[oracle@node1 ~]$ gpnptool put -p=/tmp/profile_signed.xmlAlthough when signing the profile we are given a warning that a deprecated algorithm is being used, it does complete the command.
We proceed to restart the 11.2 instance and check that there are no more error messages in the alert file.
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 2
Number of processor cores in the system is 2
Number of processor sockets in the system is 1
Private Interface 'eth2:1' configured from GPnP for use as a private interconnect.
[name='eth2:1', type=1, ip=169.254.13.3, mac=08-00-27-25-2a-76, net=169.254.0.0/19, mask=255.255.224.0, use=haip:cluster_interconnect/62]
Public Interface 'eth1' configured from GPnP for use as a public interface.
[name='eth1', type=1, ip=192.168.0.61, mac=08-00-27-7f-71-2e, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth1:1' configured from GPnP for use as a public interface.
[name='eth1:1', type=1, ip=192.168.0.66, mac=08-00-27-7f-71-2e, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth1:2' configured from GPnP for use as a public interface.
[name='eth1:2', type=1, ip=192.168.0.65, mac=08-00-27-7f-71-2e, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth1:3' configured from GPnP for use as a public interface.
[name='eth1:3', type=1, ip=192.168.0.67, mac=08-00-27-7f-71-2e, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth1:4' configured from GPnP for use as a public interface.
[name='eth1:4', type=1, ip=192.168.0.63, mac=08-00-27-7f-71-2e, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth1:5' configured from GPnP for use as a public interface.
[name='eth1:5', type=1, ip=192.168.0.64, mac=08-00-27-7f-71-2e, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Picked latch-free SCN scheme 3
. . .
db_unique_name = "db112"
open_cursors = 300
pga_aggregate_target = 128M
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
169.254.13.3
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2It is observed that the addresses of the public network and the private network are correctly recognized, and we also verify this in v$cluster_interconnects.
SQL> select * from v$cluster_interconnects;
NAME IP_ADDRESS IS_PUBLIC SOURCE
--------------- -------------------- --------- ------------------------------
eth2:1 169.254.13.3 NOThe case is finally solved, fortunately a solution was found that saved us from having to pay additional support to download patches, nor to set instance parameters!