Instance not Working in Cluster Node Runtime as Command : srvctl status database -d xyz Given?

Updated May 18, 2018

HI all
I have 2 cluster node (RAC 11gR2). All the database are still up and running (checked manually set oracle_sid xyz1 sqlplus / as sysdba)
But when i give the command : srvctl status database -d xyz.
instance xyz1 is not running on node1
instance xyz2 is not running on node2
Please guide me to solve it.
Thanks in Advance!

Comments

17 Nov 2015 2:25 pm Sugandh

Go to each of the instances and run. Select name, open_mode from v$database;

17 Nov 2015 2:26 pm Jyoti Pandey

What is the output of crstctl stat res -t.

17 Nov 2015 2:28 pm Sushma

C:UsersU0001>set oracle_sid=xyz1
C:UsersU0001>sqlplus
SQL*Plus: Release 11.2.0.3.0 Production on Wed Nov 11 02:21:57 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter user-name: sys as sysdba
Enter password:
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select instance_name , status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
xyz1 OPEN
xyz2 OPEN
SQL> ^C
C:UsersU0001>set oracle_sid=pqr1
C:UsersU0001>sqlplus
SQL*Plus: Release 11.2.0.3.0 Production on Wed Nov 11 02:22:41 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter user-name: sys as sysdba
Enter password:
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select instance_name, status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
pqr1 OPEN
pqr2 OPEN
SQL> ^C
C:UsersU0001>crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
C:UsersU0001>crsctl check cluster -all
**************************************************************
node01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
C:UsersU0001>crs_stat -t
2015-11-11 02:27:25.237:
CLSD: A file system error occured while attempting to create default permissions
for file "S:pp .2.0gridlogode01clientocr_40344_3.log" during log o
pen processing for process "UNKNOWN". Additional diagnostic data: LFI-00133: Try
ing to create file S:pp .2.0gridlogode01clientocr_40344_3.log that
already exists.
LFI-01509: fopen() failed(OSD return value = 5).
2015-11-11 02:27:25.239:
CLSD:An error was encountered while attempting to open log file "S:pp .2.0g
ridlogode01clientocr_40344_3.log". Additional diagnostics: (:CLSD00157:)
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE node01
ora.FLASH.dg ora....up.type ONLINE ONLINE node01
ora....ER.lsnr ora....er.type ONLINE ONLINE node01
ora....N1.lsnr ora....er.type ONLINE ONLINE node02
ora....N2.lsnr ora....er.type ONLINE ONLINE node02
ora....N3.lsnr ora....er.type ONLINE ONLINE node02
ora....VOTE.dg ora....up.type ONLINE ONLINE node01
ora.xyz.db ora....se.type ONLINE OFFLINE
ora.asm ora.asm.type ONLINE ONLINE node01
ora....SM1.asm application ONLINE ONLINE node01
ora....01.lsnr application ONLINE ONLINE node01
ora....s01.gsd application OFFLINE OFFLINE
ora....s01.ons application ONLINE ONLINE node01
ora....s01.vip ora....t1.type ONLINE ONLINE node02
ora....SM2.asm application ONLINE ONLINE node02
ora....02.lsnr application ONLINE ONLINE node02
ora....s02.gsd application OFFLINE OFFLINE
ora....s02.ons application ONLINE ONLINE node02
ora....s02.vip ora....t1.type ONLINE ONLINE node02
ora.cvu ora.cvu.type ONLINE ONLINE node01
ora....tadm.db ora....se.type ONLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node01
ora.oc4j ora.oc4j.type ONLINE ONLINE node01
ora.ons ora.ons.type ONLINE ONLINE node01
ora....ry.acfs ora....fs.type ONLINE ONLINE node01
ora.scan1.vip ora....ip.type ONLINE ONLINE node02
ora.scan2.vip ora....ip.type ONLINE ONLINE node02
ora.scan3.vip ora....ip.type ONLINE ONLINE node02
ora.seed.db ora....se.type ONLINE ONLINE node02
ora.pqr.db ora....se.type ONLINE ONLINE node02
C:UsersU0001>crstctl stat res -t
'crstctl' is not recognized as an internal or external command,
operable program or batch file.
C:UsersU0001>crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.FLASH.dg
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.LISTENER.lsnr
ONLINE INTERMEDIATE node01 Not All Endpoints R
egistered
ONLINE ONLINE node02
ora.OCR_VOTE.dg
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.asm
ONLINE ONLINE node01 Started
ONLINE ONLINE node02 Started
ora.gsd
OFFLINE OFFLINE node01
OFFLINE OFFLINE node02
ora.net1.network
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.ons
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.registry.acfs
ONLINE ONLINE node01
ONLINE ONLINE node02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node02
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node02
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node02
ora.xyz.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.node01.vip
1 ONLINE INTERMEDIATE node02 FAILED OVER
ora.node02.vip
1 ONLINE ONLINE node02
ora.cvu
1 ONLINE ONLINE node01
ora.abc.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.oc4j
1 ONLINE ONLINE node01
ora.scan1.vip
1 ONLINE ONLINE node02
ora.scan2.vip
1 ONLINE ONLINE node02
ora.scan3.vip
1 ONLINE ONLINE node02
ora.seed.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE node02 Open
ora.pqr.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE node02 Open
C:UsersU0001>

17 Nov 2015 2:29 pm Jyoti Pandey

What is the db name you are using? Try to start instance. srvctl start instance -d xyz -i <instance name>. Post the output.

17 Nov 2015 2:30 pm Sushma

C:UsersU0001>srvctl start instance -d XYZ -i XYZ1
PRCR-1013 : Failed to start resource ora.sped.db
PRCR-1064 : Failed to start resource ora.sped.db on node AAA01
CRS-0245: User doesn't have enough privilege to perform the operation

17 Nov 2015 2:30 pm Sushma

output- read write
select instance_name, status from gv$instance;
xyz1 open
xyz2 open

17 Nov 2015 2:31 pm Chandan Singh Parihar

The information is fetched from the ocr when you fire any commands using srvctl. It means the cluster registry doesn't know the status of your db. I would say run this command and paste the result this wont harm your db srvctl start database -d xyz

17 Nov 2015 2:32 pm Chandan Singh Parihar

It won't harm anything looks the db was started using sqlplus not using srvctl utility.

17 Nov 2015 2:32 pm Sushma

But the database are still up and running so why we use this coomand - srvctl start database -d xyz

17 Nov 2015 2:33 pm Chandan Singh Parihar

It'll not start the db it would make the cluster services aware about the state of instances.

17 Nov 2015 2:34 pm Sushma

Ok thanks, if i do not use srvctl start database and leave as it is.
It's not harmfull na (for business prospective/dba prospective?
Thanks in advance.

17 Nov 2015 2:36 pm Chandan Singh Parihar

Just think if the db is running and you say again startup what happen? Simply it would say db is already running. Same way srvctl is a global command which in back ground runs the sqls. Just fire it in dev if you are not sure.

17 Nov 2015 2:37 pm Romil Tripathi

First check configuration using
Srvctl config database -d xyz
If config is there so execute
Srvctl start database -d xyz

17 Nov 2015 2:37 pm Sushma

But database already still up and running if in this case i run srvctl start database xyz
it's not harmfull?

17 Nov 2015 2:38 pm Romil Tripathi

No harmful it might give you error: resource already started in that case you can do one thing. Shutdown the DB and then run srvctl start database -d xyz

17 Nov 2015 2:39 pm Abhijeet Mudgal

Check if the database is added in cluster. Crsctl stat res -t

17 Nov 2015 2:40 pm Sushma

C:UsersU0001>set oracle_sid=xyz1
C:UsersU0001>sqlplus
SQL*Plus: Release 11.2.0.3.0 Production on Wed Nov 11 02:21:57 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter user-name: sys as sysdba
Enter password:
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select instance_name , status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
xyz1 OPEN
xyz2 OPEN
SQL> ^C
C:UsersU0001>set oracle_sid=pqr1
C:UsersU0001>sqlplus
SQL*Plus: Release 11.2.0.3.0 Production on Wed Nov 11 02:22:41 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter user-name: sys as sysdba
Enter password:
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select instance_name, status from gv$instance;
INSTANCE_NAME STATUS
---------------- ------------
pqr1 OPEN
pqr2 OPEN
SQL> ^C
C:UsersU0001>crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
C:UsersU0001>crsctl check cluster -all
**************************************************************
node01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
C:UsersU0001>crs_stat -t
2015-11-11 02:27:25.237:
CLSD: A file system error occured while attempting to create default permissions
for file "S:pp .2.0gridlogode01clientocr_40344_3.log" during log o
pen processing for process "UNKNOWN". Additional diagnostic data: LFI-00133: Try
ing to create file S:pp .2.0gridlogode01clientocr_40344_3.log that
already exists.
LFI-01509: fopen() failed(OSD return value = 5).
2015-11-11 02:27:25.239:
CLSD:An error was encountered while attempting to open log file "S:pp .2.0g
ridlogode01clientocr_40344_3.log". Additional diagnostics: (:CLSD00157:)
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE node01
ora.FLASH.dg ora....up.type ONLINE ONLINE node01
ora....ER.lsnr ora....er.type ONLINE ONLINE node01
ora....N1.lsnr ora....er.type ONLINE ONLINE node02
ora....N2.lsnr ora....er.type ONLINE ONLINE node02
ora....N3.lsnr ora....er.type ONLINE ONLINE node02
ora....VOTE.dg ora....up.type ONLINE ONLINE node01
ora.xyz.db ora....se.type ONLINE OFFLINE
ora.asm ora.asm.type ONLINE ONLINE node01
ora....SM1.asm application ONLINE ONLINE node01
ora....01.lsnr application ONLINE ONLINE node01
ora....s01.gsd application OFFLINE OFFLINE
ora....s01.ons application ONLINE ONLINE node01
ora....s01.vip ora....t1.type ONLINE ONLINE node02
ora....SM2.asm application ONLINE ONLINE node02
ora....02.lsnr application ONLINE ONLINE node02
ora....s02.gsd application OFFLINE OFFLINE
ora....s02.ons application ONLINE ONLINE node02
ora....s02.vip ora....t1.type ONLINE ONLINE node02
ora.cvu ora.cvu.type ONLINE ONLINE node01
ora....tadm.db ora....se.type ONLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node01
ora.oc4j ora.oc4j.type ONLINE ONLINE node01
ora.ons ora.ons.type ONLINE ONLINE node01
ora....ry.acfs ora....fs.type ONLINE ONLINE node01
ora.scan1.vip ora....ip.type ONLINE ONLINE node02
ora.scan2.vip ora....ip.type ONLINE ONLINE node02
ora.scan3.vip ora....ip.type ONLINE ONLINE node02
ora.seed.db ora....se.type ONLINE ONLINE node02
ora.pqr.db ora....se.type ONLINE ONLINE node02
C:UsersU0001>crstctl stat res -t
'crstctl' is not recognized as an internal or external command,
operable program or batch file.
C:UsersU0001>crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.FLASH.dg
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.LISTENER.lsnr
ONLINE INTERMEDIATE node01 Not All Endpoints R
egistered
ONLINE ONLINE node02
ora.OCR_VOTE.dg
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.asm
ONLINE ONLINE node01 Started
ONLINE ONLINE node02 Started
ora.gsd
OFFLINE OFFLINE node01
OFFLINE OFFLINE node02
ora.net1.network
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.ons
ONLINE ONLINE node01
ONLINE ONLINE node02
ora.registry.acfs
ONLINE ONLINE node01
ONLINE ONLINE node02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node02
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node02
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node02
ora.xyz.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.node01.vip
1 ONLINE INTERMEDIATE node02 FAILED OVER
ora.node02.vip
1 ONLINE ONLINE node02
ora.cvu
1 ONLINE ONLINE node01
ora.abc.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.oc4j
1 ONLINE ONLINE node01
ora.scan1.vip
1 ONLINE ONLINE node02
ora.scan2.vip
1 ONLINE ONLINE node02
ora.scan3.vip
1 ONLINE ONLINE node02
ora.seed.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE node02 Open
ora.pqr.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE node02 Open
C:UsersU0001>

17 Nov 2015 2:40 pm Abhijeet Mudgal

We had simillar issue due to different Oracle home path. Oracle home path on both node is same or different?