CLSRSC-507: The root script cannot proceed on this node

Environment:

Current Database: Oracle RAC 19.19

Operating System : HP-UX 11.31

Error:

CLSRSC-507: The root script cannot proceed on this node Node1 because either the first-node operations have not completed on node Node3 or there was an error in obtaining the status of the first-node operations.

Change in environment:

Node1 was in unresponsive state where basic commands like bdf, ls,glance, etc. were not responding and not returning any output. Due to this, the System Team decided to reboot the server. However, post-reboot, ASM failed to start. To resolve the issue, the System Team proceeded to deconfigure and reconfigure Oracle Clusterware (CRS).

During the CRS reconfiguration phase, the root.sh script failed with the error message mentioned above.

Cause:

During CRS reconfiguration, the root.sh script failed at Step 8 with the following error messages:

root@rac1:/oracle_19c_grid/app/oracle/product/19.3#sh root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /oracle_19c_grid/app/oracle/product/19.3

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Smartmatch is deprecated at /oracle_19c_grid/app/oracle/product/19.3/crs/install/crsupgrade.pm line 6512.

Using configuration parameter file: /oracle_19c_grid/app/oracle/product/19.3/crs/install/crsconfig_params

The log of current session can be found at:

/oracle_19c_grid/app/orabase/crsdata/rac1/crsconfig/rootcrs_rac1_2025-08-02_12-18-42AM.log

2025/08/02 12:19:00 CLSRSC-594: Executing installation step 1 of 19: 'ValidateEnv'.

2025/08/02 12:19:00 CLSRSC-594: Executing installation step 2 of 19: 'CheckFirstNode'.

2025/08/02 12:19:04 CLSRSC-594: Executing installation step 3 of 19: 'GenSiteGUIDs'.

2025/08/02 12:19:05 CLSRSC-594: Executing installation step 4 of 19: 'SetupOSD'.

2025/08/02 12:19:05 CLSRSC-594: Executing installation step 5 of 19: 'CheckCRSConfig'.

2025/08/02 12:19:08 CLSRSC-594: Executing installation step 6 of 19: 'SetupLocalGPNP'.

2025/08/02 12:19:11 CLSRSC-594: Executing installation step 7 of 19: 'CreateRootCert'.

2025/08/02 12:19:12 CLSRSC-594: Executing installation step 8 of 19: 'ConfigOLR'.

2025/08/02 12:19:15 CLSRSC-507: The root script cannot proceed on this node rac1 because either the first-node operations have not completed on node rac3 or there was an error in obtaining the status of the first-node operations.

Died at /oracle_19c_grid/app/oracle/product/19.3/crs/install/oraocr.pm line 271.

Troubleshooting Steps:

Step1: Based on the error message, it was evident that the RAC installation had been previously attempted on Node3 (rac3), and the root.sh script was not completed successfully at that time. To verify this, the root.sh script log on rac3 was reviewed. Upon investigation, it was found that this Oracle 19c RAC cluster was actually upgraded from 12c in the past. During the upgrade process, the rootupgrade.sh script was executed successfully without any errors. Since this was an upgrade and not a fresh installation, rootupgrade.sh was used instead of root.sh.

rac3:/home/oracle#cd /oracle_19c_grid/app/oracle/product/19.3/install

rac3:/oracle_19c_grid/app/oracle/product/19.3/install#cat root_rac_2021-05-14_21-49-16-20.log

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /oracle_19c_grid/app/oracle/product/19.3

Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite.

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Using configuration parameter file: /oracle_19c_grid/app/oracle/product/19.3/crs/install/crsconfig_params

The log of current session can be found at:

/oracle_19c_grid/app/orabase/crsdata/rac3/crsconfig/rootcrs_rac3_2021-05-14_09-50-49PM.log

2021/05/14 21:53:26 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.

2021/05/14 21:53:27 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2021/05/14 21:53:27 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.

2021/05/14 21:53:27 CLSRSC-4005: Failed to patch Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.

2021/05/14 21:54:08 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.

2021/05/14 21:54:08 CLSRSC-464: Starting retrieval of the cluster configuration data

PRCT-1470 : failed to reset the Rapid Home Provisioning (RHP) repository

PRCR-1172 : Failed to execute "srvmhelper" for -getmgmtdbnodename

2021/05/14 21:55:29 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.

2021/05/14 21:58:15 CLSRSC-693: CRS entities validation completed successfully.

2021/05/14 21:58:59 CLSRSC-515: Starting OCR manual backup.

2021/05/14 22:01:35 CLSRSC-516: OCR manual backup successful.

2021/05/14 22:05:56 CLSRSC-486:

At this stage of upgrade, the OCR has changed.

Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2021/05/14 22:05:57 CLSRSC-541:

To downgrade the cluster:

1. All nodes that have been upgraded must be downgraded.

2021/05/14 22:05:57 CLSRSC-542:

2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2021/05/14 22:07:16 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2021/05/14 22:07:16 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.

2021/05/14 22:07:29 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.

2021/05/14 22:09:06 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.

2021/05/14 22:09:07 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.

2021/05/14 22:13:39 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2021/05/14 22:13:40 CLSRSC-482: Running command: '/oracle_12C_grid/app/oracle/product/12.2.0/bin/crsctl start rollingupgrade 19.0.0.0.0'

CRS-1131: The cluster was successfully set to rolling upgrade mode.

2021/05/14 22:13:45 CLSRSC-482: Running command: '/oracle_19c_grid/app/oracle/product/19.3/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /oracle_12C_grid/app/oracle/product/12.2.0 -oldCRSVersion 12.2.0.1.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2021/05/14 22:13:53 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

2021/05/14 22:14:27 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2021/05/14 22:14:53 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

2021/05/14 22:15:47 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.

2021/05/14 22:15:52 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.

2021/05/14 22:17:06 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.

2021/05/14 22:17:06 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.

2021/05/14 22:18:19 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.

2021/05/14 22:19:38 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.

2021/05/14 22:19:39 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

2021/05/14 22:22:31 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.

2021/05/14 22:23:48 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.

2021/05/14 22:24:56 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.

2021/05/14 22:27:49 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.

Successfully taken the backup of node specific configuration in OCR.

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'sys'..

Operation successful.

2021/05/14 22:29:18 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.

2021/05/14 22:30:11 CLSRSC-474: Initiating upgrade of resource types

2021/05/14 22:32:14 CLSRSC-475: Upgrade of resource types successfully initiated.

2021/05/14 22:34:32 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.

2021/05/14 22:35:16 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Question remains same why root.sh script was failed on Node1 then ? What next ?

Step2: Upon reviewing the error messages in root.sh, it appears that the failure could be due to one of two reasons: either the root.sh script had previously failed on Node3 (rac3), or there was an issue in retrieving the status of the first node's operations during the current execution.

CLSRSC-507: The root script cannot proceed on this node rac1 because either the first-node operations have not completed on node rac3 or there was an error in obtaining the status of the first-node operations.

Died at /oracle_19c_grid/app/oracle/product/19.3/crs/install/oraocr.pm line 271.

Let's look at the line number 271 in the script "/oracle_19c_grid/app/oracle/product/19.3/crs/install/oraocr.pm" on Node1.

oracle@rac1:/oracle_19c_grid/app/oracle/product/19.3/crs/install#ls -ltr oraocr.pm

-rwxr-xr-x 1 root oinstall 89044 May 3 2024 oraocr.pm

Open the file using the vi editor (vi <filename>), press ESC and type the following to enable line numbers (:set nu), navigate to line number 271. You will find the following code at that line:

oracle@rac1:/oracle_19c_grid/app/oracle/product/19.3/crs/install#view oraocr.pm

263 if ($CFG->ASM_STORAGE_USED)

264 {

265 trace("Retrieve OCR locations from the sync file");

266 my $syncFile = getOCRLocSyncFile();

267 if (! -e $syncFile)

268 {

269 my $localNode = tolower_host();

270 my $installNode = getInstallNode();

271 die(dieformat(507, $localNode, $installNode));

272 }

273

274 my @ocr_locations = read_file(getOCRLocSyncFile());

275 $OCR = $ocr_locations[0];

276 chomp($OCR);

277 }

From the above code, it is evident that the script is attempting to access the OCR file but is unable to read it successfully. To understand this behavior in more detail, we need to examine what exactly the script is trying to read.

Search for the function getOCRLocSyncFile() within the same file. You can do this in vi editor by typing (/getOCRLocSyncFile).

Once located, review the code starting at the corresponding line number to understand the logic implemented within the function.

2136 sub getOCRLocSyncFile

2137 #------------------------------------------------------------------------------

2138 # Function : Populates the sync filename to store the list of OCR locations

2139 # Args : None

2140 # Returns : OCR Sync filename

2141 #------------------------------------------------------------------------------

2142 {

2143 my $sync_file = catfile($CFG->params('GPNPGCONFIGDIR'),

2144 'gpnp', 'seed', 'asm', 'ocrloc.conf');

2145 trace("Sync file: $sync_file");

2146 return $sync_file;

The analyzed code attempts to read the file located at: $CRS_HOME/gpnp/seed/asm/ocrloc.conf

To validate this, the ocrloc.conf file was checked across all nodes in the cluster. It was observed that this file was missing on all nodes, including the one where the root.sh script was executed.

Since the ocrloc.conf file was not present, the root.sh script failed when it attempted to access and read this file during Clusterware configuration.

oracle@rac1:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -ltr

-rw-r--r-- 1 oracle oinstall 3746 May 14 2021 credentials.xml

oracle@rac2:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -ltr

-rw-r--r-- 1 oracle oinstall 3746 May 14 2021 credentials.xml

oracle@rac3:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -ltr

-rw-r--r-- 1 oracle oinstall 3746 May 14 2021 credentials.xml

oracle@rac4:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -ltr

-rw-r--r-- 1 oracle oinstall 3746 May 14 2021 credentials.xml

What be the next plan of action ?

Step 3: To validate, the same file was checked on other 19c cluster nodes. It was observed that the ocrloc.conf file was present on those systems.

oracle@testdb1:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -tlr

-rw-r--r-- 1 oracle oinstall 3779 Oct 21 2022 credentials.xml

-rwxr-x--- 1 oracle oinstall 59 Oct 21 2022 ocrloc.conf

oracle@testdb2:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -tlr

-rw-r--r-- 1 oracle oinstall 3779 Oct 21 2022 credentials.xml

-rwxr-x--- 1 oracle oinstall 59 Oct 21 2022 ocrloc.conf

oracle@testdb3:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -tlr

-rw-r--r-- 1 oracle oinstall 3779 Oct 21 2022 credentials.xml

-rwxr-x--- 1 oracle oinstall 59 Oct 21 2022 ocrloc.conf

oracle@testdb4:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#ls -tlr

-rw-r--r-- 1 oracle oinstall 3779 Oct 21 2022 credentials.xml

-rwxr-x--- 1 oracle oinstall 59 Oct 21 2022 ocrloc.conf

Let's check the content of this file.

oracle@testdb1:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#cat ocrloc.conf

+OCR_VOTE/testdb-cluster/OCRFILE/registry.255.1118667903

This file contains the OCR location. Let's find out the current OCR location from existing cluster.

#To check OCR location

oracle@rac1:/home/oracle#cd /var/opt/oracle

oracle@rac1:/var/opt/oracle#ls -ltr *.loc

-rw-r--r-- 1 root sys 68 Oct 21 2022 oraInst.loc

-rw-r--r-- 1 root oinstall 132 Oct 21 2022 olr.loc

-rw-r--r-- 1 root oinstall 262 Jun 2 19:28 ocr.loc

oracle@rac1:/var/opt/oracle#cat ocr.loc

ocrconfig_loc=+DG_OCR_VOTE_XP8/rac-luster/OCRFILE/registry.255.1198761621

local_only=false

Let's create same file ocrloc.conf in the location "$CRS_HOME/gpnp/seed/asm/" and add OCR location of existing cluster and save the file. Let's do it on problematic node only first.

oracle@rac1:/home/oracle#cd /oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm

oracle@rac1:/oracle_19c_grid/app/oracle/product/19.3/gpnp/seed/asm#vi ocrloc.conf

+DG_OCR_VOTE_XP8/rac-luster/OCRFILE/registry.255.1198761621

Step 4: Now run the root.sh script again.

root@rac1:/oracle_19c_grid/app/oracle/product/19.3#sh root.sh

Performing root user operation.

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /oracle_19c_grid/app/oracle/product/19.3

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Relinking oracle with rac_on option

Smartmatch is deprecated at /oracle_19c_grid/app/oracle/product/19.3/crs/install/crsupgrade.pm line 6512.

Using configuration parameter file: /oracle_19c_grid/app/oracle/product/19.3/crs/install/crsconfig_params

The log of current session can be found at:

/oracle_19c_grid/app/orabase/crsdata/rac1/crsconfig/rootcrs_rac1_2025-08-02_06-01-20PM.log

2025/08/02 18:01:41 CLSRSC-594: Executing installation step 1 of 19: 'ValidateEnv'.

2025/08/02 18:01:42 CLSRSC-594: Executing installation step 2 of 19: 'CheckFirstNode'.

2025/08/02 18:01:46 CLSRSC-594: Executing installation step 3 of 19: 'GenSiteGUIDs'.

2025/08/02 18:01:46 CLSRSC-594: Executing installation step 4 of 19: 'SetupOSD'.

2025/08/02 18:01:47 CLSRSC-594: Executing installation step 5 of 19: 'CheckCRSConfig'.

2025/08/02 18:01:48 CLSRSC-594: Executing installation step 6 of 19: 'SetupLocalGPNP'.

2025/08/02 18:01:51 CLSRSC-594: Executing installation step 7 of 19: 'CreateRootCert'.

2025/08/02 18:01:52 CLSRSC-594: Executing installation step 8 of 19: 'ConfigOLR'.

2025/08/02 18:01:54 CLSRSC-594: Executing installation step 9 of 19: 'ConfigCHMOS'.

2025/08/02 18:01:54 CLSRSC-594: Executing installation step 10 of 19: 'CreateOHASD'.

2025/08/02 18:01:57 CLSRSC-594: Executing installation step 11 of 19: 'ConfigOHASD'.

2025/08/02 18:02:14 CLSRSC-594: Executing installation step 12 of 19: 'SetupTFA'.

2025/08/02 18:02:14 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.

2025/08/02 18:02:15 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.

2025/08/02 18:02:20 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.

2025/08/02 18:02:23 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

2025/08/02 18:02:30 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.

2025/08/02 18:03:27 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2672: Attempting to start 'ora.evmd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /oracle_19c_grid/app/orabase/diag/crs/rac1/crs/trace/ocssd.trc

CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start.

CRS-4406: Oracle High Availability Services synchronous start failed.

CRS-41053: checking Oracle Grid Infrastructure for file permission issues

PRVG-2032 : Group of file "/oracle_19c_grid/app/orabase/crsdata/rac1/ocr/day_.ocr" did not match the expected value on node "rac1". [Expected = "root(0)" ; Found = "sys(3)"]

PRVG-2032 : Group of file "/oracle_19c_grid/app/orabase/crsdata/rac1/ocr/day.ocr" did not match the expected value on node "rac1". [Expected = "root(0)" ; Found = "sys(3)"]

PRVG-2032 : Group of file "/oracle_19c_grid/app/orabase/crsdata/rac1/ocr/backup00.ocr" did not match the expected value on node "rac1". [Expected = "root(0)" ; Found = "sys(3)"]

PRVH-0102 : File "/oracle_19c_grid/app/oracle/product/19.3/bin/oraipsecsetup" does not exist on node "rac1".

PRVH-0102 : File "/oracle_19c_grid/app/oracle/product/19.3/bin/orald" does not exist on node "rac1".

PRVH-0102 : File "/oracle_19c_grid/app/oracle/product/19.3/gpnp/wallets/peer/cwallet.sso" does not exist on node "rac1".

PRVH-0102 : File "/oracle_19c_grid/app/oracle/product/19.3/gpnp/wallets/root/ewallet.p12" does not exist on node "rac1".

PRVH-0102 : File "/oracle_19c_grid/app/oracle/product/19.3/gpnp/profiles/peer/profile.xml" does not exist on node "rac1".

PRVH-0154 : File "/oracle_19c_grid/app/oracle/product/19.3/lib/libcrf19.so" is empty on node "rac1".

PRVG-2032 : Group of file "/oracle_19c_grid/app/oracle/product/19.3/bin/tfactl" did not match the expected value on node "rac1". [Expected = "oinstall(107)|root(0)" ; Found = "sys(3)"]

CRS-4000: Command Start failed, or completed with errors.

2025/08/02 18:13:24 CLSRSC-117: (Bad argc for has:clsrsc-117)

Died at /oracle_19c_grid/app/oracle/product/19.3/crs/install/crsinstall.pm line 2114.

After creating the missing ocrloc.conf file, we observed that Step 8 in the root.sh execution progressed successfully. The script continued and completed up to Step 17.

However, it is now failing at the stage where it attempts to start the cluster, indicating a different error unrelated to the earlier missing file issue.

Let's troubleshoot this error now.....

Step 5: From the error messages, it is clear that there is some error while starting CSSD daemon.

Error Message: