The client called at 7am on Monday morning with a database down scenario. They arrived to messages from their users that the database was inaccessible. When they logged in, they found the database was down and that it would not start normally:

$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Feb 14 15:47:52 2016

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size 3712904 bytes
Variable Size 1476397176 bytes
Database Buffers 654311424 bytes
Redo Buffers 13062144 bytes
ORA-00205: error in identifying control file, check alert log for more info

They looked the alert log and saw messages like this:

ALTER DATABASE MOUNT
Sun Feb 14 15:47:58 2016
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/u01/app/oracle/recovery_area/ora12c/control02.ctl'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-205 signalled during: ALTER DATABASE MOUNT...
Sun Feb 14 15:47:58 2016
Checker run found 1 new persistent data failures
Sun Feb 14 15:48:00 2016
Using default pga_aggregate_limit of 2048 MB

Because they knew there had been storage corruption from the storage admins, they called us in a panic. Thankfully, and luckily, Oracle forces multiplexing of the control files, so this was an easy recovery. Just find a good copy of the control file and copy it over the corrupted one:

$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sun Feb 14 15:49:45 2016

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

SQL> show parameter control_files

NAME                              TYPE        VALUE
--------------------------------- ----------- ------------------------------
control_files                     string      /u01/app/oracle/oradata/ora12c
                                              /control01.ctl,/u01/app/oracl
                                              e/recovery_area/ora12c/control
                                              02.ctl
SQL> host
$ cd /u01/app/oracle/recovery_area/ora12c
$ cp /u01/app/oracle/oradata/ora12c/control01.ctl ./control02.ctl
$ exit
exit

SQL> alter database mount;

Database altered.

SQL> alter database open;

Database altered.

It was a very simple fix, but it really highlights knowing a lot of different recovery scenarios, including some of the easy ones.

Link to script that contains the examples in this post.

 

 

 

 

Comment