Oracle Database EBS RAC Environment Error on DB1

Updated May 18, 2018

I am getting the following errors on my Node1 of DB. I am using RAC Environment.

Database 11g
EBS 12.3 R2

This is Node1 which is not running the CRS services.
However I am running the EBS thru Node2 of RAC Environment.

# ocrcheck

PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfolclcpi1, dep
4, loc=kgfokge
AMDU-00204: Disk N0007 is in currently mounted diskgroup OCR_VOTE_DISK
AMDU-00201: Disk N0007: '/dev/rdisk/disk62'
AMDU-00204: Disk N0006 is in currently mounte
] [8]

# crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.

# olsnodes -n
PRCO-19: Failure retrieving list of nodes in the cluster
PRCO-2: Unable to communicate with the clusterware

# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online

# crsstat
HA Resource Target State
----------- ------ -----
error connecting to CRSD at [(ADDRESS=(PROTOCOL=IPC)(KEY=ora_crsqs))] clsccon 184

# oifcfg getif
PRIF-12: failed to initialize cluster support services

# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

# crsstat
HA Resource Target State
----------- ------ -----
error connecting to CRSD at [(ADDRESS=(PROTOCOL=IPC)(KEY=ora_crsqs))] clsccon 184

root
# crsctl start crs

CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

Oracle

$ srvctl start database -d PROD

PRCD-1027 : Failed to retrieve database PROD
PRCR-1070 : Failed to check if resource ora.prod.db is registered
Cannot communicate with crsd

Comments

22 Aug 2013 6:18 am Sushma
Hello,

I think you need to dig into the crsd log files.

One question - though. Was this system working at some point or is it a new install.

I would start debugging at:$CRS_HOME/log/NODENAME/alert*
and $CRS_HOME/log/NODENAME/crs* .../css*

Did you use ASM to store the OCR and voting devices? If so also check the ASM alertlogs, if other checks are inconclusive.

thanks

22 Aug 2013 9:48 am amit butola
I am using ASM and I check the logs and then replied to you.

$CRS_HOME parameter is not set.

Now how I can find the home of CRS_HOME.

22 Aug 2013 9:48 am Sushma
It was working since the last six years.
We have umount the local volume and rename it.

But when it stopped, we mount the local volume again with same name.

23 Aug 2013 6:22 am Sushma
Hello,

apologies. it is:
grep ORA_CRS_HOME /etc/init.d/init.crsd

thanks

23 Aug 2013 6:22 am Sushma
Hello,

so you changed the LV names on one system and it stopped working. And it still works on the other system, where you didn't change it?

Have a look at the owner/groups of the lv you created and compare them with the ones on the working system.

thanks