11GR2RAC丢失OCR和CONTROLFILE重建恢复过程

今天准备用自己的rac虚拟机做点实验的，可是虚拟机拉起来时，频频报错，估计好久不用了（前一阵子有厂商来推销虚拟化全在那上做测试了），挨个检查发现是OCR出故障了，没办法重建下吧，顺便记录下重建的过程

1.使用orccheck命令报错

[root@rac1 ~]# ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

PROC-26: Error while accessing the physical storage

2.使用ocrconfig -restore恢复仍然报错

[root@rac1 rac-cluster]# ocrconfig -restore/u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr

PROT-35: The configured OCR locations are not accessible.

3.强制停掉两个节点的crs（两个节点都要执行）

[root@rac1 rac-cluster]# crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'rac1'

CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 ~]# /u01/app/11.2.0/grid/bin/crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'

CRS-2673: Attempting to stop 'ora.evmd' on 'rac2'

CRS-2673: Attempting to stop 'ora.asm' on 'rac2'

CRS-2677: Stop of 'ora.evmd' on 'rac2' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded

CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'

CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded

CRS-2673: Attempting to stop 'ora.crf' on 'rac2'

CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'

CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'

CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

4.在节点一上已独占方式启动(只启动ASM实例但不启动CRS)

[root@rac1 ~]# crsctl start crs -excl -nocrs

CRS-4123: Oracle High Availability Services has been started.

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'

CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'rac1'

CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
      

5.重建原votedisk所在的磁盘组

[root@rac1 ~]# su - grid

[grid@rac1 ~]$ sqlplus /as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Sun Feb 2800:09:322016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0- 64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL>create diskgroup systemdg normal redundancy disk'/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd' ATTRIBUTE 'compatible.rdbms'='11.2', 'compatible.asm'='11.2';

Diskgroup created.

6.使用最新的OCR备份进行恢复

[root@rac1 ~]# ocrconfig -restore/u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr

[root@rac1 ~]# ocrcheck

Status of Oracle Cluster Registry isas follows :

     Version                  :          3

     Total space (kbytes)     :     262120

     Used space (kbytes)      :       2900

     Available space (kbytes) :     259220

     ID                       : 1207909490

     Device/File Name         :  +SYSTEMDG

                                    Device/File integrity check succeeded

                                    Device/Filenot configured

                                    Device/Filenot configured

                                    Device/Filenot configured

                                    Device/Filenot configured

     Cluster registry integrity check succeeded

     Logical corruption check succeeded

7.恢复votedisk

[root@rac1 ~]# crsctl query css votedisk

Located 0 voting disk(s).
[grid@rac1 ~]$ crsctl replace votedisk +SYSTEMDG

Successful addition of voting disk a0e3b9c0c5934f79bf67aa783618b59d.

Successful addition of voting disk 422a673bdbc94f97bff51f66f84a105e.

Successful addition of voting disk a1b110709eb84f39bf351f1b01aa0c7d.

Successfully replaced voting diskgroupwith+SYSTEMDG.

CRS-4266: Voting file(s) successfully replaced

[root@rac1 rac-cluster]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Diskgroup

--  -----    -----------------                --------- ---------

 1. ONLINE   03bd9e6851cf4fb4bf4c92e24a3d71cc (/dev/asm-diskb) [SYSTEMDG]

 2. ONLINE   362f2cfb5b3d4f29bf319c668e0efbe4 (/dev/asm-diskc) [SYSTEMDG]

 3. ONLINE   57659c53f4284fbdbfeabafb20c3fbdd (/dev/asm-diskd) [SYSTEMDG]

Notes：如果在恢复时遇到如下错误：

[grid@rac1 ~]$ crsctl replace votedisk +SYSTEMDG

CRS-4602: Failed 27toadd voting file a0e3b9c0c5934f79bf67aa783618b59d.

CRS-4602: Failed 27toadd voting file 422a673bdbc94f97bff51f66f84a105e.

CRS-4602: Failed 27toadd voting file a1b110709eb84f39bf351f1b01aa0c7d.

Failed toreplace voting diskgroupwith+SYSTEMDG.

CRS-4000: Command Replace failed, or completed with errors.

--登陆grid用户，检查并修改asm_diskstring参数

SQL>alter system set asm_diskstring='/dev/asm*';

System altered.

--检查并修改spfile参数，并重启ASM

SQL>create spfile from memory;

File created.

8.重启crs

[root@rac1 ~]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'

CRS-2673: Attempting to stop 'ora.asm' on 'rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'

CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded

CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'

CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'

CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'

CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@rac1 ~]# crsctl start has

CRS-4123: Oracle High Availability Services has been started.

9.由于启动较慢，过几分钟查看状态

[root@rac1 ~]# crs_stat -t

Name           Type           Target    State     Host       

------------------------------------------------------------

ora.DATADG.dg  ora....up.type ONLINE    ONLINE    rac1       

ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1       

ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1       

ora....EMDG.dg ora....up.type ONLINE    ONLINE    rac1       

ora.asm        ora.asm.type   ONLINE    ONLINE    rac1       

ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac1       

ora.gsd        ora.gsd.type   OFFLINE   OFFLINE              

ora....network ora....rk.type ONLINE    ONLINE    rac1       

ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE              

ora.ons        ora.ons.type   ONLINE    ONLINE    rac1       

ora.orcl.db    ora....se.type ONLINE    OFFLINE              

ora....SM1.asm application    ONLINE    ONLINE    rac1       

ora....C1.lsnr application    ONLINE    ONLINE    rac1       

ora.rac1.gsd   application    OFFLINE   OFFLINE              

ora.rac1.ons   application    ONLINE    ONLINE    rac1       

ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1       

ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac1       

ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1       

[root@rac1 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

10.但是在启动db的时候报错了。。。

PRCR-1079 : Failed to start resource ora.orcl.db

CRS-5017: The resource action "ora.orcl.db start" encountered the following error:

ORA-00205: error in identifying control file, check alert logfor more info

. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-5017: The resource action "ora.orcl.db start" encountered the following error:

ORA-00205: error in identifying control file, check alert logfor more info

. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'rac1' failed

CRS-2674: Start of 'ora.orcl.db' on 'rac2' failed

CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy

--查看了下trace发现SYSTEMDG中的控制文件没了，装RAC的时候就没规划好，导致控制文件放到了votedisk里，但是SYSTEMDG刚刚被重建了，所以丢了

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.8.222)(PORT=1521))' SCOPE=MEMORY SID='orcl1';

ALTERDATABASE MOUNT /* db agent *//* {1:40511:558} */

This instance was first to mount

NOTE: Loaded library: System

SUCCESS: diskgroup DATADG was mounted

SUCCESS: diskgroup SYSTEMDG was mounted

NOTE: dependency betweendatabase orcl and diskgroup resource ora.DATADG.dg is established

ORA-00210: cannot open the specified control file

ORA-00202: control file: '+SYSTEMDG/orcl/controlfile/current.261.904952269'

ORA-17503: ksfdopn:2 Failed toopenfile+SYSTEMDG/orcl/controlfile/current.261.904952269

ORA-15012: ASM file'+SYSTEMDG/orcl/controlfile/current.261.904952269' does not exist

ORA-205 signalled during: ALTERDATABASE MOUNT /* db agent *//* {1:40511:558} */...

Sat Feb 2723:48:142016

Shutting down instance (abort)

License high water mark =1

USER (ospid: 21150): terminating the instance

Instance terminated byUSER, pid =21150

Sat Feb 2723:48:142016

Instance shutdown complete

11.将数据库启动到nomount阶段

[grid@rac1 trace]$ srvctl start database-d orcl -o nomount

12.在节点一上使用rman恢复一份控制文件到SYSTEMDG上

[oracle@rac1 ~]$ rman target /

Recovery Manager: Release 11.2.0.4.0- Production on Sat Feb 2723:53:352016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (not mounted)

RMAN>restore controlfile to'+SYSTEMDG'from'+DATADG/orcl/controlfile/current.260.901699963';

Starting restore at 2016/02/2723:53:43

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=34 instance=orcl1 device type=DISK

channel ORA_DISK_1: copied control file copy

Finished restore at 2016/02/2723:53:52

13.已恢复好的控制文件路径和名称

ASMCMD> pwd

+systemdg/orcl/CONTROLFILE

ASMCMD> ls

current.261.904953225

14.节点一上登陆db，修改control_files参数

[oracle@rac1 ~]$ sqlplus /as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 2723:55:092016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0- 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining andReal Application Testing options

SQL>alter system set control_files='+DATADG/orcl/controlfile/current.260.901699963','+SYSTEMDG/orcl/CONTROLFILE/current.261.904953225' scope=spfile sid='*';

System altered.

15.重启数据库，并检查数据库状态

[grid@rac1 trace]$ srvctl stop database-d orcl

[grid@rac1 trace]$ srvctl start database-d orcl

[grid@rac1 trace]$ srvctl status database-d orcl

Instance orcl1 is running on node rac1

Instance orcl2 is running on node rac2

16.保险起见，查看两个节点上的控制文件参数是否正确

[oracle@rac1 ~]$ sqlplus /as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 2723:58:472016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0- 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining andReal Application Testing options

SQL> show parameter control_file

NAME                                 TYPE          VALUE

------------------------------------ ----------- ------------------------------------------------------------------------------------------------

control_file_record_keep_time        integer       7

control_files                        string        +DATADG/orcl/controlfile/current.260.901699963, +SYSTEMDG/orcl/controlfile/current.261.904953225

[oracle@rac2 ~]$ sqlplus /as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 2723:59:292016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0- 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining andReal Application Testing options

SQL> show parameter control_file

NAME                                 TYPE           VALUE

------------------------------------ ----------- ------------------------------------------------------------------------------------------------

control_file_record_keep_time        integer       7

control_files                        string        +DATADG/orcl/controlfile/current.260.901699963, +SYSTEMDG/orcl/controlfile/current.261.904953225

OK，收工~~~~