CRS-42216: No interfaces are configured on the local node for interface

Database : Oracle Database 19.0.0.0 EE

Operating System : Linux 86_64 Bit

Database Type : Oracle 5 Node RAC DB

Incident Summary: This is a five-node RAC setup where single public and private Ethernet interfaces are used. The CRS services were terminated unexpectedly due to the following error messages. Upon checking with the client, network teaming was found to be configured at the OS level for both public and private Ethernet interfaces. Only logical name (a single interface) is visible at the cluster level, whereas two physical network cards are visible at the OS level. Out of the two Ethernet cards, one had gone out of the teaming configuration. The OS team added this Ethernet card back, but instead of reconfiguring it under teaming, they configured bonding. After this change, the CRS services failed to start. Error Messages: CRS-42216: No interfaces are configured on the local node for interface CRS-7503: The Oracle Grid Infrastructure process 'gipcd' observed communication issues between node CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node CRS-1656: The CSS daemon is terminating due to a fatal error CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 26591 experienced fatal signal or exception code 6. CRS-1013: The OCR location in an ASM disk group is inaccessible Steps performed to troubleshoot the issue : Step1 : Lets first understand the issue before proceeding further. Since the CRS is down on Node1, let's verify the Ethernet card details from another running node. [root@testdb2 ~]# cd /oracle_grid/app/19.0/gridhome/bin/oifcfg getif bond0  10.20.30.0  global  public bond1  10.1.2.0  global  cluster_interconnect,asm Here, bond1 represents the Private Ethernet interface, and bond0 represents the Public Ethernet interface. Network teaming was configured for the private Ethernet (bond1) at the OS level. Two physical Ethernet cards were associated with bond1, and one of them ens6f1 had gone out of the teaming configuration. #Command to check the existing Network Teaming at OS layer. root@testdb1 ~]# nmcli c s NAME UUID    TYPE DEVICE bond1 92306dc1-4142-23de-097b-b1464cfab5ee team bond1 bond1-port1 352a19af-6edd-1308-6669-eal6fa00a8f9 ethernet ens6f1 bond1-port2 8d1973ff-b8e2-85ba-c302-5c5bdaa3d881 ethernet ens12fl Here, you can see that teaming was used instead of bonding. As confirmed by the Network/OS team, the Ethernet card ens6f1 was automatically removed from the teaming configuration. Step2 : Let's check the alert log file of the cluster. Cluster Alert Log File: 2025-11-05 16:35:17.284 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. 2025-11-05 16:35:18.439 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. 2025-11-05 16:35:18.504 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. 2025-11-05 16:35:22.662 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. .... 2025-11-05 16:35:22.729 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. 2025-11-05 16:35:23.813 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. 2025-11-05 16:35:23.933 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. 2025-11-05 16:35:24.962 [GIPCD(22921)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.*)?:10.1.2.0: available interface definitions are [bond0(:.*)?:10.20.30.0][bond0(:.*)?:[fe80:0:0:0:0:0:0:0]]. ...... 2025-11-05 16:58:45.899 [GIPCD(23534)]CRS-7503: The Oracle Grid Infrastructure process 'gipcd' observed communication issues between node 'testdb1' and node 'testdb5', interface list of local node 'testdb1' is '10.1.2.201:14948;', interface list of remote node 'testdb5' is '10.1.2.205:50250;'. 2025-11-05 16:59:06.887 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb2'. 2025-11-05 16:59:06.888 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb3'. 2025-11-05 16:59:06.888 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb4'. 2025-11-05 16:59:06.888 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb5'. 2025-11-05 16:59:44.943 [GIPCD(23534)]CRS-7503: The Oracle Grid Infrastructure process 'gipcd' observed communication issues between node 'testdb1' and node 'testdb3', interface list of local node 'testdb1' is '10.1.2.201:14948;', interface list of remote node 'testdb3' is '10.1.2.203:12411;'. 2025-11-05 17:00:05.932 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb2'. 2025-11-05 17:00:05.932 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb3'. 2025-11-05 17:00:05.933 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb4'. 2025-11-05 17:00:05.933 [OCSSD(26591)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'testdb5'. 2025-11-05 17:00:22.610 [CSSDAGENT(26527)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:5:3} in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ohasd_cssdagent_root.trc. 2025-11-05 17:00:22.612 [OCSSD(26591)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ocssd.trc. 2025-11-05 17:00:22.687 [OHASD(21113)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:5:3} in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ohasd.trc. 2025-11-05 17:00:23.612 [OCSSD(26591)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ocssd.trc 2025-11-05 17:00:23.613 [OCSSD(26591)]CRS-1603: CSSD on node testdb1 has been shut down. 2025-11-05 17:00:23.994 [OCSSD(26591)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ocssd.trc. 2025-11-05T17:00:28.626428+05:30 Errors in file /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ocssd.trc  (incident=105): CRS-8503 [] [] [] [] [] [] [] [] [] [] [] [] 2025-11-05 17:00:28.614 [OCSSD(26591)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 26591 experienced fatal signal or exception code 6. Incident details in: /oracle_grid/app/grid/diag/crs/testdb1/crs/incident/incdir_105/ocssd_i105.trc 2025-11-05 17:00:29.454 [CSSDMONITOR(65725)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 65725 2025-11-05 17:00:33.648 [CRSCTL(50533)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_50533.trc. 2025-11-05 17:00:33.661 [CRSCTL(30485)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_30485.trc. 2025-11-05 17:00:35.584 [CRSCTL(65944)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_65944.trc. 2025-11-05 17:00:36.320 [CRSCTL(66173)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_66173.trc. 2025-11-05 17:00:38.558 [CRSCTL(66531)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_66531.trc. 2025-11-05 17:00:39.292 [OHASD(21113)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:5:3} in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ohasd.trc. 2025-11-05 17:00:39.946 [CSSDAGENT(66969)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 66969 2025-11-05 17:00:42.063 [CRSCTL(66736)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_66736.trc. 2025-11-05 17:00:44.163 [CRSCTL(66926)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_66926.trc. 2025-11-05 17:00:49.685 [OCRCONFIG(67361)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ocrconfig_67361.trc. 2025-11-05 17:00:59.452 [OCRDUMP(68283)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/ocrdump_68283.trc. 2025-11-05 17:01:16.291 [CRSCTL(69036)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_69036.trc. 2025-11-05 17:01:22.048 [CRSCTL(69336)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_69336.trc. 2025-11-05 17:01:28.780 [CRSCTL(69517)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /oracle_grid/app/grid/diag/crs/testdb1/crs/trace/crsctl_69517.trc. 2025-11-05 17:24:33.011 [OHASD(21113)]CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb1' 2025-11-05 17:24:34.584 [GPNPD(23328)]CRS-2329: Grid Plug and Play Daemon(GPNPD) on node testdb1 shut down. 2025-11-05 17:24:34.582 [MDNSD(23207)]CRS-5602: mDNS service stopping by request. ...... Upon reviewing the cluster alert log file, multiple error messages were observed. The initial error, “CRS-42216: No interfaces are configured on the local node,” suggests a problem with the Ethernet interface configuration on the affected node. Step3 : The Network/OS team confirmed that one of the Ethernet cards had gone out of the teaming configuration. The same Ethernet card was re-added at the OS level under a different name, but this time configured using bonding instead of teaming, as the original interface had been removed. Since the bond0 and bond1 interfaces (used by the cluster) were not modified—and only their underlying physical interfaces were affected and renamed—no cluster-level configuration changes were required. Private IP ping tests between the local and remote nodes were successful in both directions, confirming that network connectivity was intact. root@testdb1 ~]# nmcli c s NAME UUID    TYPE DEVICE bond1 92306dc1-4142-23de-097b-b1464cfab5ee bond bond1 ens12fl 5cffbca4-edc7-d963-bcel-c41f2c733dc5 ethernet ens12fl ens6f1 6c0laac9-fcbb-f202-48fc-82146b96bce2 ethernet eth1 OS team has added reomoved ethernet card as bonding again instead of teaming. Step3 : Let's start the cluster now. The cluster startup was initiated, but the network services failed during the process. As a result, the ASM instance and other cluster-related services did not come up successfully. By root: #crsctl start crs -wait [root@testdb1 ~]# ps -ef | grep d.bin root       21113       1  1 16:50 ?        00:00:22 /oracle_grid/app/19.0/gridhome/bin/ohasd.bin reboot grid       23207       1  0 16:50 ?        00:00:03 /oracle_grid/app/19.0/gridhome/bin/mdnsd.bin grid       23208       1  0 16:50 ?        00:00:05 /oracle_grid/app/19.0/gridhome/bin/evmd.bin grid       23328       1  0 16:50 ?        00:00:05 /oracle_grid/app/19.0/gridhome/bin/gpnpd.bin grid       23534       1  0 16:50 ?        00:00:17 /oracle_grid/app/19.0/gridhome/bin/gipcd.bin root       26393       1  5 16:50 ?        00:01:46 /oracle_grid/app/19.0/gridhome/bin/osysmond.bin root      506218   90671  0 17:24 pts/4    00:00:00 grep --color=auto d.bin Step4 : As network-layer changes were performed, network socket files were cleared on the affected node. #First stop the cluster on problematic node. [root@testdb1 ~]# cd /oracle_grid/app/19.0/gridhome/bin [root@testdb1 bin]# ./crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testdb1' CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'testdb1' CRS-2673: Attempting to stop 'ora.crf' on 'testdb1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdb1' CRS-2673: Attempting to stop 'ora.evmd' on 'testdb1' CRS-2677: Stop of 'ora.cssdmonitor' on 'testdb1' succeeded CRS-2677: Stop of 'ora.crf' on 'testdb1' succeeded CRS-2677: Stop of 'ora.drivers.acfs' on 'testdb1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdb1' CRS-2673: Attempting to stop 'ora.gipcd' on 'testdb1' CRS-2677: Stop of 'ora.mdnsd' on 'testdb1' succeeded CRS-2677: Stop of 'ora.evmd' on 'testdb1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'testdb1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'testdb1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testdb1' has completed CRS-4133: Oracle High Availability Services has been stopped. #Now clear the Network Socket Files located under below directories. /var/tmp/.oracle /usr/tmp/.oracle /tmp/.oracle [root@testdb1 bin]# cd /var/tmp/.oracle [root@testdb1 .oracle]# ls -ltr -rw-r--r--  1 root root     0 Apr 22  2024 ora_gipc_OHASD_soleowner_lock prw-r--r--. 1 root root     0 Nov  5 16:50 npohasd prw-r--r--  1 root root     0 Nov  5 16:50 npohasd2 srwxrwxrwx  1 root root     0 Nov  5 16:50 ora_gipc_OHASD_soleowner -rw-r--r--  1 root root     0 Nov  5 16:50 sprocr_local_conn_0_PROL_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 sprocr_local_conn_0_PROL -rw-r--r--  1 root root     0 Nov  5 16:50 ora_gipc_testdb1_INIT_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 ora_gipc_testdb1_INIT -rw-r--r--  1 root root     0 Nov  5 16:50 ora_gipc_scls_opct_testdb1_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 ora_gipc_scls_opct_testdb1 -rw-r--r--  1 root root     0 Nov  5 16:50 sOHASD_IPC_SOCKET_11_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 sOHASD_IPC_SOCKET_11 -rw-r--r--  1 root root     0 Nov  5 16:50 sOHASD_UI_SOCKET_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 sOHASD_UI_SOCKET -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_EVMD_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_EVMD -rw-r--r--. 1 grid oinstall 6 Nov  5 16:50 mdnsd.pid srwxr-xr-x  1 grid oinstall 0 Nov  5 16:50 mdnsd -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_GPNPD_testdb1_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_GPNPD_testdb1 -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_GPNPD_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_GPNPD -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 sSYSTEM.evm.acceptor.auth_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 sSYSTEM.evm.acceptor.auth -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 sAevm_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 sAevm -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_GIPCD_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_GIPCD -rw-r--r--  1 root root     0 Nov  5 16:50 ora_gipc_testdb1_MOND_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 ora_gipc_testdb1_MOND -rw-r-----  1 root root     0 Nov  5 16:50 ora_gipc_stestdb1CRFM_MIIPC_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 ora_gipc_stestdb1CRFM_MIIPC -rw-r-----  1 root root     0 Nov  5 16:50 ora_gipc_stestdb1CRFM_CLIIPC_lock srwxrwxrwx  1 root root     0 Nov  5 16:50 ora_gipc_stestdb1CRFM_CLIIPC -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_agent_ag_CSSDAG_z_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_agent_ag_CSSDAG_z -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_CSSD_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_testdb1_CSSD -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 sOCSSD_LL_testdb1__lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 sOCSSD_LL_testdb1_ -rw-r--r--  1 grid oinstall 0 Nov  5 16:50 ora_gipc_css_ctrllcl_CSSD_BCCM_lock srwxrwxrwx  1 grid oinstall 0 Nov  5 16:50 ora_gipc_css_ctrllcl_CSSD_BCCM [root@testdb1 .oracle]# rm -rf * [root@testdb1 .oracle]# cd  /tmp/.oracle [root@testdb1 .oracle]# ls -ltr total 0 [root@testdb1 .oracle]# cd /usr/tmp/.oracle [root@testdb1 .oracle]# ls -tlr total 0 After clearing Network Socket Files, now start the cluster on problematic node. [root@testdb1 ~]# /oracle_grid/app/19.0/gridhome/bin/crsctl start crs -wait CRS-4123: Starting Oracle High Availability Services-managed resources CRS-2672: Attempting to start 'ora.evmd' on 'testdb1' CRS-2672: Attempting to start 'ora.mdnsd' on 'testdb1' CRS-2676: Start of 'ora.mdnsd' on 'testdb1' succeeded CRS-2676: Start of 'ora.evmd' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'testdb1' CRS-2676: Start of 'ora.gpnpd' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'testdb1' CRS-2676: Start of 'ora.gipcd' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testdb1' CRS-2672: Attempting to start 'ora.crf' on 'testdb1' CRS-2676: Start of 'ora.crf' on 'testdb1' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'testdb1' CRS-2672: Attempting to start 'ora.diskmon' on 'testdb1' CRS-2676: Start of 'ora.diskmon' on 'testdb1' succeeded CRS-2676: Start of 'ora.cssd' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'testdb1' CRS-2672: Attempting to start 'ora.ctssd' on 'testdb1' CRS-2676: Start of 'ora.ctssd' on 'testdb1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'testdb1' CRS-2676: Start of 'ora.asm' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.storage' on 'testdb1' CRS-2676: Start of 'ora.storage' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'testdb1' CRS-2676: Start of 'ora.crsd' on 'testdb1' succeeded CRS-6017: Processing resource auto-start for servers: testdb1 CRS-2673: Attempting to stop 'ora.testdb1.vip' on 'testdb4' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'testdb2' CRS-2672: Attempting to start 'ora.ons' on 'testdb1' CRS-2672: Attempting to start 'ora.chad' on 'testdb1' CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'testdb2' succeeded CRS-2673: Attempting to stop 'ora.scan3.vip' on 'testdb2' CRS-2677: Stop of 'ora.testdb1.vip' on 'testdb4' succeeded CRS-2672: Attempting to start 'ora.testdb1.vip' on 'testdb1' CRS-2677: Stop of 'ora.scan3.vip' on 'testdb2' succeeded CRS-2672: Attempting to start 'ora.scan3.vip' on 'testdb1' CRS-2676: Start of 'ora.chad' on 'testdb1' succeeded CRS-2676: Start of 'ora.testdb1.vip' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'testdb1' CRS-2676: Start of 'ora.scan3.vip' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'testdb1' CRS-2676: Start of 'ora.LISTENER.lsnr' on 'testdb1' succeeded CRS-33672: Attempting to start resource group 'ora.asmgroup' on server 'testdb1' CRS-2672: Attempting to start 'ora.asmnet1.asmnetwork' on 'testdb1' CRS-2676: Start of 'ora.asmnet1.asmnetwork' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'testdb1' CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'testdb1' succeeded CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'testdb1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'testdb1' CRS-2676: Start of 'ora.ons' on 'testdb1' succeeded CRS-2676: Start of 'ora.asm' on 'testdb1' succeeded CRS-33676: Start of resource group 'ora.asmgroup' on server 'testdb1' succeeded. CRS-6016: Resource auto-start has completed for server testdb1 CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources CRS-4123: Oracle High Availability Services has been started. Step5 : Now check the cluster services. [root@testdb1 ~]# ps -ef | grep pmon grid      918092       1  0 17:39 ?        00:00:00 asm_pmon_+ASM1 root      928012   90671  0 17:39 pts/4    00:00:00 grep --color=auto pmon [root@testdb1 ~]# /oracle_grid/app/19.0/gridhome/bin/crsctl stat res -t -------------------------------------------------------------------------------- Name           Target  State        Server                   State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr                ONLINE  ONLINE       testdb1               STABLE                ONLINE  ONLINE       testdb2               STABLE                ONLINE  ONLINE       testdb3               STABLE                ONLINE  ONLINE       testdb4               STABLE                ONLINE  ONLINE       testdb5               STABLE ora.chad                ONLINE  ONLINE       testdb1               STABLE                ONLINE  ONLINE       testdb2               STABLE                ONLINE  ONLINE       testdb3               STABLE                ONLINE  ONLINE       testdb4               STABLE                ONLINE  ONLINE       testdb5               STABLE ora.helper                OFFLINE OFFLINE      testdb1               IDLE,STABLE                OFFLINE OFFLINE      testdb2               STABLE                OFFLINE OFFLINE      testdb3               STABLE                OFFLINE OFFLINE      testdb4               STABLE                OFFLINE OFFLINE      testdb5               STABLE ora.net1.network                ONLINE  ONLINE       testdb1               STABLE                ONLINE  ONLINE       testdb2               STABLE                ONLINE  ONLINE       testdb3               STABLE                ONLINE  ONLINE       testdb4               STABLE                ONLINE  ONLINE       testdb5               STABLE ora.ons                ONLINE  ONLINE       testdb1               STABLE                ONLINE  ONLINE       testdb2               STABLE                ONLINE  ONLINE       testdb3               STABLE                ONLINE  ONLINE       testdb4               STABLE                ONLINE  ONLINE       testdb5               STABLE ora.proxy_advm                OFFLINE OFFLINE      testdb1               STABLE                OFFLINE OFFLINE      testdb2               STABLE                OFFLINE OFFLINE      testdb3               STABLE                OFFLINE OFFLINE      testdb4               STABLE                OFFLINE OFFLINE      testdb5               STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.DG_ARCHIVAL.dg(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.DG_DATA.dg(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.DG_FRA.dg(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.DG_MGMTDB.dg(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.DG_REDO.dg(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.LISTENER_SCAN1.lsnr       1        ONLINE  ONLINE       testdb5               STABLE ora.LISTENER_SCAN2.lsnr       1        ONLINE  ONLINE       testdb3               STABLE ora.LISTENER_SCAN3.lsnr       1        ONLINE  ONLINE       testdb1               STABLE ora.LISTENER_SCAN4.lsnr       1        ONLINE  ONLINE       testdb4               STABLE ora.LISTENER_SCAN5.lsnr       1        ONLINE  ONLINE       testdb2               STABLE ora.MGMTLSNR       1        ONLINE  ONLINE       testdb4               169.254.27.253 10.17                                                              6.146.199,STABLE ora.OCR_DISK.dg(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.asm(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               Started,STABLE       2        ONLINE  ONLINE       testdb3               Started,STABLE       3        ONLINE  ONLINE       testdb2               Started,STABLE       4        ONLINE  ONLINE       testdb4               Started,STABLE       5        ONLINE  ONLINE       testdb5               Started,STABLE ora.asmnet1.asmnetwork(ora.asmgroup)       1        ONLINE  ONLINE       testdb1               STABLE       2        ONLINE  ONLINE       testdb3               STABLE       3        ONLINE  ONLINE       testdb2               STABLE       4        ONLINE  ONLINE       testdb4               STABLE       5        ONLINE  ONLINE       testdb5               STABLE ora.cvu       1        ONLINE  ONLINE       testdb5               STABLE ora.testdb1.vip       1        ONLINE  ONLINE       testdb1               STABLE ora.testdb2.vip       1        ONLINE  ONLINE       testdb2               STABLE ora.testdb3.vip       1        ONLINE  ONLINE       testdb3               STABLE ora.testdb4.vip       1        ONLINE  ONLINE       testdb4               STABLE ora.testdb5.vip       1        ONLINE  ONLINE       testdb5               STABLE ora.mgmtdb       1        ONLINE  ONLINE       testdb4               Open,STABLE ora.testdb.db       1        OFFLINE OFFLINE                               Instance Shutdown,ST                                                              ABLE       2        ONLINE  INTERMEDIATE testdb2               Mounted (Closed),HOM                                                              E=/oracle_home/app/o                                                              racle/product/19.0/d                                                              bhome,STABLE       3        ONLINE  INTERMEDIATE testdb3               Mounted (Closed),HOM                                                              E=/oracle_home/app/o                                                              racle/product/19.0/d                                                              bhome,STABLE       4        ONLINE  INTERMEDIATE testdb4               Mounted (Closed),HOM                                                              E=/oracle_home/app/o                                                              racle/product/19.0/d                                                              bhome,STABLE       5        ONLINE  INTERMEDIATE testdb5               Mounted (Closed),HOM                                                              E=/oracle_home/app/o                                                              racle/product/19.0/d                                                              bhome,STABLE ora.qosmserver       1        ONLINE  ONLINE       testdb2               STABLE ora.rhpserver       1        OFFLINE OFFLINE                               STABLE ora.scan1.vip       1        ONLINE  ONLINE       testdb5               STABLE ora.scan2.vip       1        ONLINE  ONLINE       testdb3               STABLE ora.scan3.vip       1        ONLINE  ONLINE       testdb1               STABLE ora.scan4.vip       1        ONLINE  ONLINE       testdb4               STABLE ora.scan5.vip       1        ONLINE  ONLINE       testdb2               STABLE -------------------------------------------------------------------------------- [root@testdb1 ~]# /oracle_grid/app/19.0/gridhome/bin/crsctl stat res -t -init -------------------------------------------------------------------------------- Name           Target  State        Server                   State details -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm       1        ONLINE  ONLINE       testdb1               STABLE ora.cluster_interconnect.haip       1        ONLINE  ONLINE       testdb1               STABLE ora.crf       1        ONLINE  ONLINE       testdb1               STABLE ora.crsd       1        ONLINE  ONLINE       testdb1               STABLE ora.cssd       1        ONLINE  ONLINE       testdb1               STABLE ora.cssdmonitor       1        ONLINE  ONLINE       testdb1               STABLE ora.ctssd       1        ONLINE  ONLINE       testdb1               OBSERVER,STABLE ora.diskmon       1        OFFLINE OFFLINE                               STABLE ora.drivers.acfs       1        ONLINE  ONLINE       testdb1               STABLE ora.evmd       1        ONLINE  ONLINE       testdb1               STABLE ora.gipcd       1        ONLINE  ONLINE       testdb1               STABLE ora.gpnpd       1        ONLINE  ONLINE       testdb1               STABLE ora.mdnsd       1        ONLINE  ONLINE       testdb1               STABLE ora.storage       1        ONLINE  ONLINE       testdb1               STABLE -------------------------------------------------------------------------------- Issue has been resolved by clearing Network Socket Files.

Thanks for reading this post ! Please comment if you like this post ! Click on FOLLOW to get next blog updates !