Environment: Current Database: Oracle RAC 19.28 Operating System : Redhat Linux 8 Error: CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /grid19/app/gridbase/diag/crs/testdbnode1/crs/trace/ocssd.trc CRS-2674: Start of 'ora.cssd' on 'testdbnode1' failed .... CRS-4000: Command Start failed, or completed with errors. 2025/12/17 18:19:13 CLSRSC-119: Start of the exclusive mode cluster failed Died at /grid19/app/grid/19.0/crs/install/crsinstall.pm line 2583. Issue Description: This is a fresh Oracle 19c two-node RAC installation on the Red Hat Enterprise Linux 8 operating system. During the CRS reconfiguration phase, the root.sh script failed on Node 1 with the error message mentioned above. Cause: During CRS reconfiguration, the root.sh script failed at Step 16 with the following error messages: [root@testdbnode1 install]#sh /grid19/app/grid/19.0/root.sh The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /grid19/app/grid/19.0 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Relinking oracle with rac_on option Using configuration parameter file: /grid19/app/grid/19.0/crs/install/crsconfig_params The log of current session can be found at: /grid19/app/gridbase/crsdata/testdbnode1/crsconfig/rootcrs_testdbnode1_2025-12-17_06-04-45PM.log 2025/12/17 18:04:56 CLSRSC-594: Executing installation step 1 of 19: 'ValidateEnv'. 2025/12/17 18:04:56 CLSRSC-594: Executing installation step 2 of 19: 'CheckFirstNode'. 2025/12/17 18:04:58 CLSRSC-594: Executing installation step 3 of 19: 'GenSiteGUIDs'. 2025/12/17 18:04:59 CLSRSC-594: Executing installation step 4 of 19: 'SetupOSD'. Redirecting to /bin/systemctl restart rsyslog.service 2025/12/17 18:04:59 CLSRSC-594: Executing installation step 5 of 19: 'CheckCRSConfig'. 2025/12/17 18:05:00 CLSRSC-594: Executing installation step 6 of 19: 'SetupLocalGPNP'. 2025/12/17 18:05:12 CLSRSC-594: Executing installation step 7 of 19: 'CreateRootCert'. 2025/12/17 18:05:16 CLSRSC-594: Executing installation step 8 of 19: 'ConfigOLR'. 2025/12/17 18:05:37 CLSRSC-594: Executing installation step 9 of 19: 'ConfigCHMOS'. 2025/12/17 18:05:37 CLSRSC-594: Executing installation step 10 of 19: 'CreateOHASD'. 2025/12/17 18:05:43 CLSRSC-594: Executing installation step 11 of 19: 'ConfigOHASD'. 2025/12/17 18:05:43 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service' 2025/12/17 18:06:08 CLSRSC-594: Executing installation step 12 of 19: 'SetupTFA'. 2025/12/17 18:06:08 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'. 2025/12/17 18:06:08 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'. 2025/12/17 18:06:41 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'. 2025/12/17 18:06:47 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'. 2025/12/17 18:07:20 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. CRS-2672: Attempting to start 'ora.evmd' on 'testdbnode1' CRS-2672: Attempting to start 'ora.mdnsd' on 'testdbnode1' CRS-2676: Start of 'ora.evmd' on 'testdbnode1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'testdbnode1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'testdbnode1' CRS-2676: Start of 'ora.gpnpd' on 'testdbnode1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testdbnode1' CRS-2672: Attempting to start 'ora.gipcd' on 'testdbnode1' CRS-2676: Start of 'ora.cssdmonitor' on 'testdbnode1' succeeded CRS-2676: Start of 'ora.gipcd' on 'testdbnode1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'testdbnode1' CRS-2672: Attempting to start 'ora.diskmon' on 'testdbnode1' CRS-2676: Start of 'ora.diskmon' on 'testdbnode1' succeeded CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /grid19/app/gridbase/diag/crs/testdbnode1/crs/trace/ocssd.trc CRS-2674: Start of 'ora.cssd' on 'testdbnode1' failed CRS-2679: Attempting to clean 'ora.cssd' on 'testdbnode1' CRS-2681: Clean of 'ora.cssd' on 'testdbnode1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'testdbnode1' CRS-2677: Stop of 'ora.gipcd' on 'testdbnode1' succeeded CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'testdbnode1' CRS-2677: Stop of 'ora.cssdmonitor' on 'testdbnode1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdbnode1' CRS-2677: Stop of 'ora.gpnpd' on 'testdbnode1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdbnode1' CRS-2677: Stop of 'ora.mdnsd' on 'testdbnode1' succeeded CRS-2673: Attempting to stop 'ora.evmd' on 'testdbnode1' CRS-2677: Stop of 'ora.evmd' on 'testdbnode1' succeeded CRS-4000: Command Start failed, or completed with errors. 2025/12/17 18:19:13 CLSRSC-119: Start of the exclusive mode cluster failed Died at /grid19/app/grid/19.0/crs/install/crsinstall.pm line 2583. What do you think ? What can be the issue? Let's start troubleshooting.Troubleshooting Steps: Step1: Based on the error message, there appears to be an issue with the voting devices. We will verify the availability and permissions of the voting files. [root@testdbnode1# ls -ltr /dev/oracleasm/disks brw-rw---- 1 grid oinstall 65, 33 Dec 17 17:31 OCRVD1 brw-rw---- 1 grid oinstall 65, 49 Dec 17 17:31 OCRVD2 brw-rw---- 1 grid oinstall 65, 65 Dec 17 17:31 OCRVD3 As shown above, the permissions of the voting disks at the OS level are set correctly, as expected. Next, we will verify the oracleasm configuration and status. The oracleasm service is loaded and the disks are mounted properly. [root@testdbnode1# oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes As shown above, the oracleasm status is OK. Next, let us review the actual error message from the CRS logs. CRS Log File: 2025-12-16 17:17:20.383 : OCRASM:2276859712 ASM Error Stack: ORA-29701: unable to connect to Cluster Synchronization Service 2025-12-16 17:17:20.433 : OCRRAW:2276859712: proprioo: Failed to open [+OCRVD]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE. 2025-12-16 17:17:20.433 : OCRRAW:2276859712: proprioo: No OCR/OLR devices are usable OCRUTL:2276859712: u_fill_errorbuf: Error Info : (Insufficient quorum to open OCR devices) default:2276859712: u_set_gb1_comp_error: comptype '107' : error '0' 2025-12-16 17:17:20.433 : OCRRAW:2276859712: proprinti: Could not open raw devices From the above messages, it is clear that there is an issue related to the OCR disks. Accordingly, we engaged the Storage and System teams to verify any problems at the storage or OS level. Both teams have confirmed that there are no issues at the storage or operating system layers. Even though the OCR disks are visible on both nodes and oracleasm can access these disks at the OS level, root.sh is still unable to discover them during the CRS configuration phase. Step2: Let us check for any known bugs or documented issues related to this behavior. The following known bugs have been reported for the above error messages encountered during root.sh execution, and they match our environment. 1) Bug 32410237 - oracleasm configure -p not discovering disks on RHEL8 2) Bug 32812376 - ROOT.SH IS FAILING WTH THE ERRORS CLSRSC-119: START OF THE EXCLUSIVE MODE CLUSTER FAILED This bug indicates that the oracleasm utility fails to discover ASM disks on RHEL 8 during root.sh execution, even though the disks are visible and accessible at the OS level. Step3: Let's check the existing version of oracleasm utility. [root@testdbnode1# rpm -qa | grep -i oracleasm oracleasmlib-2.0.17-1.el8.x86_64 oracleasm-support-2.1.12-1.el8.x86_64 [root@testdbnode2# rpm -qa | grep -i oracleasm oracleasmlib-2.0.17-1.el8.x86_64 oracleasm-support-2.1.12-1.el8.x86_64 Here, the installed oracleasm RPMs are for OEL8. What will be the solution to fix this issue ? Step4: Let us remove the existing oracleasm RPM packages for OEL 8, download the OEL 7 oracleasm RPMs packages, and install them. By root user: #Command to remove oracleasm RPMs [root@testdbnode1# rpm -e oracleasmlib-2.0.17-1.el8.x86_64 [root@testdbnode1# rpm -e oracleasm-support-2.1.12-1.el8.x86_64 [root@testdbnode2# rpm -e oracleasmlib-2.0.17-1.el8.x86_64 [root@testdbnode2# rpm -e oracleasm-support-2.1.12-1.el8.x86_64 #Command to install oracleasm RPMs for OEL7 [root@testdbnode1# rpm -ivh oracleasmlib-2.0.12-1.el7.x86_64.rpm [root@testdbnode1# rpm -ivh oracleasm-support-2.1.11-2.el7.x86_64.rpm [root@testdbnode2# rpm -ivh oracleasmlib-2.0.12-1.el7.x86_64.rpm [root@testdbnode2# rpm -ivh oracleasm-support-2.1.11-2.el7.x86_64.rpm #Command to check the installed oracleasm RPMs for OEL7 [root@testdbnode1# rpm -qa | grep -i oracleasm oracleasmlib-2.0.12-1.el7.x86_64.rpm oracleasm-support-2.1.11-2.el7.x86_64.rpm [root@testdbnode2# rpm -qa | grep -i oracleasm oracleasmlib-2.0.12-1.el7.x86_64.rpm oracleasm-support-2.1.11-2.el7.x86_64.rpm #Load the new Oracle ASMLib specific to OL7 [root@testdbnode1# # oracleasm init [root@testdbnode2# # oracleasm init #Check the whether ASMLib is up using [root@testdbnode1# oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes [root@testdbnode2# oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes #Configure the ASMLIB [root@testdbnode1# oracleasm configure -i [root@testdbnode2# oracleasm configure -i #Enable the Oracle ASMLIB file system [root@testdbnode1# oracleasm configure -e [root@testdbnode2# oracleasm configure -e #Execute scandisks and listdisks commands to verify the disks status. [root@testdbnode1# oracleasm scandisks [root@testdbnode2# oracleasm scandisks [root@testdbnode1# oracleasm listdisks [root@testdbnode2# oracleasm listdisks Step5: Now let's run the root.sh again and check. [root@testdbnode1 install]#sh /grid19/app/grid/19.0/root.sh The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /grid19/app/grid/19.0 Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Relinking oracle with rac_on option Using configuration parameter file: /grid19/app/grid/19.0/crs/install/crsconfig_params The log of current session can be found at: /grid19/app/gridbase/crsdata/testdbnode1/crsconfig/rootcrs_testdbnode1_2025-12-17_08-42-02PM.log 2025/12/17 20:42:06 CLSRSC-594: Executing installation step 1 of 19: 'ValidateEnv'. 2025/12/17 20:42:06 CLSRSC-594: Executing installation step 2 of 19: 'CheckFirstNode'. 2025/12/17 20:42:07 CLSRSC-594: Executing installation step 3 of 19: 'GenSiteGUIDs'. 2025/12/17 20:42:08 CLSRSC-594: Executing installation step 4 of 19: 'SetupOSD'. 2025/12/17 20:42:08 CLSRSC-594: Executing installation step 5 of 19: 'CheckCRSConfig'. 2025/12/17 20:42:09 CLSRSC-594: Executing installation step 6 of 19: 'SetupLocalGPNP'. 2025/12/17 20:42:09 CLSRSC-594: Executing installation step 7 of 19: 'CreateRootCert'. 2025/12/17 20:42:13 CLSRSC-594: Executing installation step 8 of 19: 'ConfigOLR'. 2025/12/17 20:42:13 CLSRSC-594: Executing installation step 9 of 19: 'ConfigCHMOS'. 2025/12/17 20:42:44 CLSRSC-594: Executing installation step 10 of 19: 'CreateOHASD'. 2025/12/17 20:42:45 CLSRSC-594: Executing installation step 11 of 19: 'ConfigOHASD'. 2025/12/17 20:42:45 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service' 2025/12/17 20:42:59 CLSRSC-594: Executing installation step 12 of 19: 'SetupTFA'. 2025/12/17 20:42:59 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'. 2025/12/17 20:42:59 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'. 2025/12/17 20:43:00 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'. 2025/12/17 20:43:04 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'. 2025/12/17 20:43:34 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. ASM has been created and started successfully. [DBT-30001] Disk groups created successfully. Check /grid19/app/gridbase/cfgtoollogs/asmca/asmca-251217PM084333.log for details. 2025/12/17 20:44:08 CLSRSC-482: Running command: '/grid19/app/grid/19.0/bin/ocrconfig -upgrade grid oinstall' CRS-4256: Updating the profile Successful addition of voting disk 41967c82c7fe4fe6bfd435aecd1baf6a. Successful addition of voting disk 674ed99094ee4fc0bfd51dcff6121b06. Successful addition of voting disk b5af545036604f2abf9297c4089790c4. Successfully replaced voting disk group with +OCRVD. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 41967c82c7fe4fe6bfd435aecd1baf6a (/dev/oracleasm/disks/OCRVD1) [OCRVD] 2. ONLINE 674ed99094ee4fc0bfd51dcff6121b06 (/dev/oracleasm/disks/OCRVD2) [OCRVD] 3. ONLINE b5af545036604f2abf9297c4089790c4 (/dev/oracleasm/disks/OCRVD3) [OCRVD] Located 3 voting disk(s). 2025/12/17 20:44:53 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'. 2025/12/17 20:45:48 CLSRSC-343: Successfully started Oracle Clusterware stack 2025/12/17 20:45:48 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'. 2025/12/17 20:46:45 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'. 2025/12/17 20:47:01 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded As shown above, the root.sh script executed successfully after implementing the solution. Please proceed with running the root.sh script on the remaining node as well. 🎉 Enjoy the troubleshooting journey!!! 📝 Stay tuned for a detailed blog post on this case !!! |
Thanks for reading this post ! Please comment if you like this post ! Click FOLLOW to get future blog updates !
Thank you for visiting my blog ! Thanks for your comment !