Translate into your own language

Showing posts with label Patching. Show all posts
Showing posts with label Patching. Show all posts

Wednesday, April 25, 2018

UtilSession failed: Error in checking for identical patches for composite patch


Yesterday while applying the April 2018 PSU, we encountered the below error:

host01> $ORACLE_HOME/OPatch/opatch apply 
Oracle Interim Patch Installer version 11.2.0.3.6 
Copyright (c) 2013, Oracle Corporation. All rights reserved. 

Oracle Home : /u01/sq/DB01/db/11.2.0.4 
Central Inventory : /opt/oraInventory/oraInventory 
from : /u01/sq/DB01/db/11.2.0.4/oraInst.loc 
OPatch version : 11.2.0.3.6 
OUI version : 11.2.0.4.0 
Log file location : /u01/sq/DB01/db/11.2.0.4/cfgtoollogs/opatch/opatch2018-04-24_12-19-05PM_1.log 

Verifying environment and performing prerequisite checks... 
UtilSession failed: Error in checking for identical patches for composite patch. This means a constituent to be applied has an earlier creation date than a constituent installed in the Oracle Home. This patch is improperly packaged and will not be applied. 
Log file location: /u01/sq/DGRIDO01/db/11.2.0.4/cfgtoollogs/opatch/opatch2018-04-24_12-19-05PM_1.log 

The message which it was showing was completely clueless. From the error message it is very difficult to get the root cause of this error.

So we raised a case with Oracle and they suggested to use the latest OPatch version to resolve the issue. Though we had valid OPatch version, we replaced it with the latest one and the issue was resolved.

Wednesday, May 4, 2016

Oracle DBA Interview Questions and Answers - Patching,Cloning and Upgrade

In which months oracle release CPU patches?

JAN, APR, JUL, OCT

When we applying single Patch, can you use opatch utility?

Yes, you can use Opatch incase of single patch. The only type of patch that cannot be used with OPatch is a patchset.

Is it possible to apply OPATCH without downtime?

As you know for apply patch your database and listener must be down. When you apply OPTACH it will update your current ORACLE_HOME. Thus coming to your question to the point in fact it is not possible without or zero downtime in case of single instance but in RAC you can Apply Opatch without downtime as there will be more separate ORACLE_HOME and more separate instances (running once instance on each ORACLE_HOME).

When you moved oracle binary files from one ORACLE_HOME server to another server then which oracle utility will be used to make this new ORACLE_HOME usable?
 

Relink all.

You have collection of patch (nearly 100 patches) or patchset. How can you apply only one patch from it?

With Napply itself (by providing patch location and specific patch id) you can apply only one patch from a collection of extracted patch. For more information check the opatch util NApply –help. It will give you clear picture.

For Example:

opatch util napply <patch_location> -id 9 -skip_subset -skip_duplicate
This will apply only the patch id 9 from the patch location and will skip duplicate and subset of patch installed in your ORACLE_HOME.

If both CPU and PSU are available for given version which one, you will prefer to apply?

From the above discussion it is clear once you apply the PSU then the recommended way is to apply the next PSU only. In fact, no need to apply CPU on the top of PSU as PSU contain CPU (If you apply CPU over PSU will considered you are trying to rollback the PSU and will require more effort in fact). So if you have not decided or applied any of the patches then, I will suggest you to go to use PSU patches. For more details refer: Oracle Products [ID 1430923.1], ID 1446582.1

PSU is superset of CPU then why someone choose to apply a CPU rather than a PSU?

CPUs are smaller and more focused than PSU and mostly deal with security issues. It seems to be theoretically more consecutive approach and can cause less trouble than PSU as it has less code changing in it. Thus any one who is concerned only with security fixes and not functionality fixes, CPU may be good approach.

How to Download Patches, Patchset or Opatch from metalink?

If you are using latest support.oracle.com then after login to metalink Dashboard
- Click on "Patches & Updates" tab
- On the left sidebar click on "Latest Patchsets" under "Oracle Server/Tools".
- A new window will appear.
- Just mouseover on your product in the "Latest Oracle Server/Tools Patchsets" page.
- Corresponding oracle platform version will appear. Then simply choose the patchset version and click on that.
- You will go the download page. From the download page you can also change your platform and patchset version.

REFERENCES:
http://docs.oracle.com/cd/E11857_01/em.111/e12255/e_oui_appendix.htm
Oracle® Universal Installer and OPatch User's Guide
11g Release 2 (11.2) for Windows and UNIX
Part Number E12255-11


What is the recent Patch applied?
  
January 2016 PSU patch

What is OPatch?

It is the utility to apply the patch.

How to Apply Opatch in Oracle?

1. You MUST read the Readme.txt file included in opatch file, look for any prereq. steps/ post installation steps or and DB related changes. Also, make sure that you have the correct opatch version required by this patch.
2.Make sure you have a good backup of database.
3. Make a note of all Invalid objects in the database prior to the patch.
4. Shutdown All the Oracle Processes running from that Oracle Home , including the Listener and Database instance, Management agent etc.
5. You MUST Backup your oracle Home and Inventory
tar -cvf $ORACLE_HOME $ORACLE_HOME/oraInventory | gzip > Backup_Software_Version.tar.gz
6. Unzip the patch in $ORACLE_HOME/patches
7. cd to the patch direcory and do opatch -apply to apply the patch.
8. Read the output/log file to make sure there were no errors.

Patching Oracle Software with OPatch ?

opatch napply <patch_location> -skip_subset -skip_duplicate
OPatch skips duplicate patches and subset patches (patches under <patch_location> that are subsets of patches installed in the Oracle home).

What is Opactch in Oracle?

OPATCH Utility (Oracle RDBMS Patching)

1. Download the required Patch from Metalink based on OS Bit Version and DB Version.
2. Need to down the database before applying patch.
3. Unzip and Apply the Patch using ”opatch apply” command.On successfully applied of patch you will see successful message “OPatch succeeded.“, Crosscheck your patch is applied by using “opatch lsinventory” command .
4. Each patch has a unique ID, the command to rollback a patch is “opatch rollback -id  <patch no.>” command.On successfully applied of patch you will see successful message “OPatch succeeded.“, Crosscheck your patch is applied by using “opatch lsinventory” command .
5. Patch file format will be like, “p<patch no.>_<db version>_<os>.zip”
6. We can check the opatch version using “opatch -version” command.
7. Generally, takes 2 minutes to apply a patch.
8. To get latest Opatch version download “patch 6880880 - latest opatch tool”, it contains OPatch directory.
9. Contents of downloaded patches will be like “etc,files directories and a README file”
10. Log file for Opatch utility can be found at $ORACLE_HOME/cfgtoollogs/opatch
11. OPatch also maintains an index of the commands executed with OPatch and the log files associated with it in the history.txt file located in the <ORACLE_HOME>/cfgtoollogs/opatch directory.
12. Starting with the 11.2.0.2 patch set, Oracle Database patch sets are full installations of the Oracle Database software. This means that you do not need to install Oracle Database 11g Release 2 (11.2.0.1) before installing Oracle Database 11g Release 2 (11.2.0.2).
13. Direct upgrade to Oracle 10g is only supported if your database is running one of the following releases: 8.0.6, 8.1.7, 9.0.1, or 9.2.0. If not, you will have to upgrade the database to one of these releases or use a different upgrade option (like export/ import).
14.Direct upgrades to 11g are possible from existing databases with versions 9.2.0.4+, 10.1.0.2+ or 10.2.0.1+. Upgrades from other versions are supported only via intermediate upgrades to a supported upgrade version.

http://avdeo.com/2008/08/19/opatch-utility-oracle-rdbms-patching/

Oracle version 10.2.0.4.0 what does each number refers to?

Oracle version number refers:
10 – Major database release number
 2 – Database Maintenance release number
 0 – Application server release number
 4 – Component Specific release number
 0 – Platform specific release number

Types of Patches?

How to rollback a patch?

What is PSU?

What is Rolling Patch?

How to check installed Patches?

How much time will it take for Patching?

Common issues faced in Patching?


Cloning
=======
What is Cloning?

How to do take RMAN Cloning? Explain Steps?

Upgrade
=======

What is rolling upgrade?

It is a new ASM feature from Database 11g.ASM instances in Oracle database 11g release(from 11.1) can be upgraded or patched using rolling upgrade feature. This enables us to patch or upgrade ASM nodes in a clustered environment without affecting database availability.During a rolling upgrade we can maintain a functional cluster while one or more of the nodes in the cluster are running in different software versions.Rolling upgrade can be used only for Oracle database 11g releases(from 11.1).

Steps to Upgrade in Oracle ?

Manual upgrade which involves the following steps:
1.Backup the database.
2.In UNIX/Linux environments, set the $ORACLE_HOME and $PATH variables to point to the new 11g Oracle home.
3.Analyze the existing instance using the "$ORACLE_HOME/rdbms/admin/utlu111i.sql" script.
4.Start the original database using the STARTUP UPGRADE command and proceed with the upgrade by running the "$ORACLE_HOME/rdbms/admin/catupgrd.sql" script.
5.Recompile invalid objects.
6.Restart the database.
7.Run the "$ORACLE_HOME/rdbms/admin/utlu111s.sql" script and check the result of the upgrade.
8.Troubleshoot any issues or abort the upgrade.

What happens when you give "STARTUP UPGRADE"?

$sqlplus "/as sysdba"
SQL> STARTUP UPGRADE

Note:
----
The UPGRADE keyword enables you to open a database based on an earlier Oracle Database release. It also restricts logons to AS SYSDBAsessions, disables system triggers, and performs additional operations that prepare the environment for the upgrade.

You might be required to use the PFILE option to specify the location of your initialization parameter file.
Once the database is started in upgrade mode, only queries on fixed views execute without errors until after the catupgrd.sql script is run. Before running catupgrd.sql, queries on any other view or the use of PL/SQL returns an error.

What is the difference between startup Upgrade and Migrate ?

startup migrate:
---------------
Used to upgrade a database till 9i.

Startup Upgrade
---------------
From 10G  we are using startup upgrade to upgrade database.

What happens internally when you use startup upgrade/migrate?

It will adjust few database (init) parameters (irrespective of what you have defined) automatically to certain values in order to run upgrade scripts smoothely.
in other way..it will issue few alter statements to set certain parameters which are required to complete the upgrade scripts without any issues.


Common issues faced in Upgrade?

Error is related to timezone file
Started database in upgrade mode and fired catupgrd.sql :

SQL> startup upgrade
ORACLE instance started.
Total System Global Area 6413680640 bytes
Fixed Size                  2160112 bytes
Variable Size            1946159632 bytes
Database Buffers         4429185024 bytes
Redo Buffers               36175872 bytes
Database mounted.
Database opened.
SQL> @catupgrd.sql
DOC>#######################################################################
DOC>#######################################################################
DOC>
DOC>   The first time this script is run, there should be no error messages
DOC>   generated; all normal upgrade error messages are suppressed.
DOC>
DOC>   If this script is being re-run after correcting some problem, then
DOC>   expect the following error which is not automatically suppressed:
DOC>
DOC>   ORA-00001: unique constraint () violated
DOC>#
   FROM registry$database
        *
ERROR at line 2:
ORA-00942: table or view does not exist
This  error is related to timezone file  which must be version 4 for Oracle version 11g.If timezone is not version 4 than patch needs to be applied.
Query to check timezone file  is:
SQL> select * from v$timezone_file;
FILENAME        VERSION
———— ———-
timezlrg.dat          4
SQL> select * from v$timezone_file;
FILENAME        VERSION
———— ———-
timezlrg.dat          4
So I had correct version.I remember applying patch before upgrade.I got lucky because patch existed for version 10.2.0.3.
If there is no patch for your Oracle versions than patch can be download for similar version and  applied manually.
Instructions are below:
1. Download the identified patch.
2. Unzip the patch, and locate the 2 files timezone.dat and timezlrg.dat in the “files/oracore/zoneinfo” directory of the uncompressed patch (or from the relevant .jar file of a   patchset). If there is also a readme.txt in this location then make a note of this as well.
3. Backup your existing files in $ORACLE_HOME/oracore/zoneinfo – THIS CAN BE VITAL, DO NOT SKIP.
Note:
Before going on with step 4, make sure the current files are not in use.
On Windows the files will simply refuse to be removed when the are in use.
On Unix replacing the files whilst they are in use can cause the files to become corrupt. Use the fuser command before replacing the files to make sure they are not in use.
4. Copy the 2 .dat files and possibly the readme.txt file that were found in step 2 into the $ORACLE_HOME/oracore/zoneinfo directory.
5. Restart the database (in case of installation on a database), or restart the client applications (in case of client install). Note that the database did not need to be down before the time zone files were applied, but it does need to be restarted afterwards.

Tuesday, May 3, 2016

OPatch failed with error code 73 " Following executables are active :"

The process of applying patch is to stop all the services from the existing home where we are going to apply the patch. We do our best but sometime when we issue the opatch apply command it throws the below error:

OPatch failed with error code 73 " Following executables are active :". See below figure:



Solution: The solution of this error is very simple. We have to use /sbin/fuser command to find the executable process id. It gives extra m with the process id like 123m. Now we have to grep the process id and kill it.  Please find the below figure:


After killing the executable when we run the opatch apply, it completes successfully.
Please find the below figure.




Friday, April 15, 2016

ORA-15001: diskgroup "'+DATA_DISK" does not exist or is not mounted after OS Patching

We regularly encounter this type of error when Linux admin apply the patch on database server.
But when the Linux team confirms and you try to start the database you can see the below error:

Note- It has nothing to do with the database

ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA_DISK/DBNAME/spfileDBNAME.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA_DISK/DBNAME/spfileDBNAME.ora
ORA-15056: additional error message
ORA-15001: diskgroup "'+DATA_DISK" does not exist or is not mounted

No need to worry here. We can simply match the "oracleasm" and "kernal" version.
if it does not match then immediatly contact Linux team who performed the patch.
Below are the simple steps for matching:

[oracle@host1~]$ uname -a
Linux tryprorarac2c.intra.searshc.com 2.6.18-408.el5xen #1 SMP Fri Dec 11 14:07:27 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

[oracle@host1 ~]$ rpm -qa |grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.8-1.el5
oracleasm-2.6.18-408.el5-2.0.5-1.el5
[oracle@host1~]$ uname -r
2.6.18-408.el5xen
[oracle@host1~]$

Linux team will update the oracleasm version to match with the kernal version and then we can start the database successfully.

Now match the "oracleasm" and "kernal" version again and verify again. We can see it matches perfectly.

[oracle@host1~]$ rpm -qa |grep oracleasm
oracleasm-2.6.18-408.el5xen-2.0.5-1.el5
oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.8-1.el5
[oracle@host1~]$ uname -r
2.6.18-408.el5xen

Sunday, March 20, 2016

Step by step how to apply PSU patch in Oracle

STEP 1:Check the operating system bit version (32 bit or 64 bit).

 [oracle@host1 ~]$ uname -a

 Linux host11.intra.searshc.com 2.6.18-308.1.1.el5xen #1 SMP Fri Feb 17 17:11:09 EST 2012 x86_64 x86_64 x86_64 GNU/Linux.

here we can see it is Linux operating system with bit version  x86_64.

If it is AIX server then use below command to check the bit version:

host03-oracle:/home/oracle(PROD)-->getconf -a | grep KERN
KERNEL_BITMODE:                         64


STEP 2:Check the oracle database version where we have to apply the patch.

There are various method to check the database version. I would be using lsinventory command because it shows many more details apart from database version.

[oracle@host1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0.2/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0.2/db_1/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version         : 11.2.0.2.0
Log file location : /u01/app/oracle/product/11.2.0.2/db_1/cfgtoollogs/opatch/opatch2016-03-21_00-14-10AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0.2/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-03-21_00-14
-10AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.
-----------------------------------------------------------------------
-----------------------------------------------------------------------

the above is the only parical output. You can see lot of other detials as well.
Here you can notice two things:
1. OPatch version    : 11.2.0.3.5
2. Oracle Database 11g : 11.2.0.2.0

I will cover the OPatch version in the later steps. Here we will concentrate only on oracle database version.

So from above two steps we have found that the operating system bit version is
x86_64 and database version is 11.2.0.2.0.

STEP 3: Now check in the patch advisory document what patch oracle has released for 11.2.0.2.0. You can find the patch advisory document for each quarter's patch.


STEP 4: Download the patch from my oracle support(meta link). While downloading make sure you are downloading the patch for the correct operating system bit version.
Here we have to downlaod the patch for operating system bit version
x86_64 and database version 11.2.0.2.0.


STEP 5. Move the downloaded zipped file into the server. And then unzip it using the appropriate unzip command into the server only.

STEP 6. Now you can unzip the file on your local system and open the read me file from it.

STEP 7: In the prerequisite section you can see the OPatch version required to apply this patch. It clearly mention if higher version of OPatch is needed or not. If so then download the OPatch and move it to the Oracle home. Make sure you rename or remove the old OPatch direcotry from the Oracle home. After this verify the new opatch version using lsinventory command. OPatch version will be changed from what was earlier. Now we are good to go.


STEP 8: Count the invalid object and make a note of it.


STEP 9: If listener is running from the home where we are go apply to patch then stop it otherwise no need to stop it. Many time it is running from Grid home. So no need to stop it if is running from Grid home.

STEP 10: If the server is being monitored by OEM grid control then put it into blackout.

STEP 11: Check the space into the mount point what oracle home is reffering to.

STEP 12: Shut down all the databases running from that home.

STEP 13: Go to the directory where you have moved and unzipped the patch:

[oracle@host1  tmp]$ cd p20760997_112030_Linux-x86-64
[oracle@host1 p14841437_112020_Linux-x86-64]$ ls -ltr
total 4
drwxr-xr-x 43 ralam0 ldap-users 4096 Feb 11  2013 20760997
[oracle@host1 p14841437_112020_Linux-x86-64]$ cd 20760997

STEP 14: Now issue the following command:

$$ORACLE_HOME/OPatch/opatch apply 

   
             ------------------------------------------------------------------------------
             -------------------------------------------------------------------------------
             --------------------------------------------------------------------------------

OPatch succeeded


STEP 15: Start up the database and run the catbundle command for each databases running under that home.

$cd $ORACLE_HOME/rdbms/admin
SQL>SQLPLUS “/ AS SYSDBA”
SQL> STARTUP
 SQL> @catbundle.sql psu apply
     
STEP 16: Run the utlrp.sql script to validate the objects.

SQL>@utlrp.sql

STEP 17. Count the invalid object and make sure it is lesser or same than what it was in earlier count before patching.




Step by step how to apply CPU(SPU) Patch in Oracle

STEP 1:Check the operating system bit version (32 bit or 64 bit).

 [oracle@host1 ~]$ uname -a

 Linux host11.intra.searshc.com 2.6.18-308.1.1.el5xen #1 SMP Fri Feb 17 17:11:09 EST 2012 x86_64 x86_64 x86_64 GNU/Linux.

here we can see it is Linux operating system with bit version  x86_64.

If it is AIX server then use below command to check the bit version:

host03-oracle:/home/oracle(PROD)-->getconf -a | grep KERN
KERNEL_BITMODE:                         64


STEP 2:Check the oracle database version where we have to apply the patch.

There are various method to check the database version. I would be using lsinventory command because it shows many more details apart from database version.

[oracle@host1 ~]$ $ORACLE_HOME/OPatch/opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11.2.0.2/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0.2/db_1/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version         : 11.2.0.2.0
Log file location : /u01/app/oracle/product/11.2.0.2/db_1/cfgtoollogs/opatch/opatch2016-03-21_00-14-10AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11.2.0.2/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-03-21_00-14
-10AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.
-----------------------------------------------------------------------
-----------------------------------------------------------------------

the above is the only parical output. You can see lot of other detials as well.
Here you can notice two things:
1. OPatch version    : 11.2.0.3.5
2. Oracle Database 11g : 11.2.0.2.0

I will cover the OPatch version in the later steps. Here we will concentrate only on oracle database version.

So from above two steps we have found that the operating system bit version is
x86_64 and database version is 11.2.0.2.0.

STEP 3: Now check in the patch advisory document what patch oracle has released for 11.2.0.2.0. You can find the patch advisory document for each quarter's patch.


STEP 4: Download the patch from my oracle support(meta link). While downloading make sure you are downloading the patch for the correct operating system bit version.
Here we have to downlaod the patch for operating system bit version
x86_64 and database version 11.2.0.2.0.


STEP 5. Move the downloaded zipped file into the server. And then unzip it using the appropriate unzip command into the server only.

STEP 6. Now you can unzip the file on your local system and open the read me file from it.

STEP 7: In the prerequisite section you can see the OPatch version required to apply this patch. It clearly mention if higher version of OPatch is needed or not. If so then download the OPatch and move it to the Oracle home. Make sure you rename or remove the old OPatch direcotry from the Oracle home. After this verify the new opatch version using lsinventory command. OPatch version will be changed from what was earlier. Now we are good to go.


STEP 8: Count the invalid object and make a note of it.


STEP 9: If listener is running from the home where we are go apply to patch then stop it otherwise no need to stop it. Many time it is running from Grid home. So no need to stop it if is running from Grid home.

STEP 10: If the server is being monitored by OEM grid control then put it into blackout.

STEP 11: Check the space into the mount point what oracle home is reffering to.

STEP 12: Shut down all the databases running from that home.

STEP 13: Go to the directory where you have moved and unzipped the patch:

[oracle@host1  tmp]$ cd p14841437_112020_Linux-x86-64
[oracle@host1 p14841437_112020_Linux-x86-64]$ ls -ltr
total 4
drwxr-xr-x 43 ralam0 ldap-users 4096 Feb 11  2013 14841437
[oracle@host1 p14841437_112020_Linux-x86-64]$ cd 14841437
[oracle@hsot1 14841437]$ ls -ltr
total 164
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 11830776
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 11830777
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586486
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586488
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586489
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586491
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586492
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586493
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586494
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586495
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12586496
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12846268
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 12846269
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 13386082
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 13769501
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 13769502
drwxr-xr-x 4 ralam0 ldap-users 4096 Feb 11  2013 13769503

STEP 14: Now issue the following command:

$opatch napply -skip_subset -skip_duplicate

     This command will show logs as below :

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch  13528551     : applied on Thu Mar 22 05:19:12 CDT 2012
Unique Patch ID:  14454920
    Created on 6 Jan 2012, 05:04:26 hrs PST8PDT
    Bugs fixed:
      13528551

Patch  13499128     : applied on Thu Mar 22 05:19:05 CDT 2012
Unique Patch ID:  14454920
    Created on 6 Jan 2012, 04:58:55 hrs PST8PDT
    Bugs fixed:
    13499128

Patch  13466801     : applied on Thu Mar 22 05:19:00 CDT 2012
Unique Patch ID:  14454920
  Created on 6 Jan 2012, 04:50:15 hrs PST8PDT
    Bugs fixed:
      13466801

             ------------------------------------------------------------------------------
             -------------------------------------------------------------------------------
             --------------------------------------------------------------------------------

OPatch succeeded


STEP 15: Start up the database and run the catbundle command for each databases running under that home.

$cd $ORACLE_HOME/rdbms/admin
SQL>SQLPLUS “/ AS SYSDBA”
SQL> STARTUP
        SQL> @catbundle.sql cpu apply
        Below are the log of catbundle command
SQL> @?/rdbms/admin/catbundle.sql cpu apply
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.
Generating apply and rollback scripts...
Check the following file for errors:
/u01/app/oracle/product/11.2.0/dbhome_          1/cfgtoollogs/catbundle/catbundle_CPU_IKBTEST1_GENERATE_2012Mar22_05_22 _13.log
Apply script: /u01/app/oracle/product/11.2.0/dbhome_ 1/rdbms/admin/catbundle_CPU_IKBTEST1_APPLY.sql
Rollback script: /u01/app/oracle/product/11.2.0/dbhome_ 1/rdbms/admin/catbundle_CPU_IKBTEST1_ROLLBACK.sql
PL/SQL procedure successfully completed.
Executing script file...
SQL> COLUMN spool_file NEW_VALUE spool_file NOPRINT
SQL> SELECT '/u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/catbundle/' || 'catbundle_CPU_' || name || '_APPLY_' || TO_CHAR(SYSDATE, 'YYYYMonDD_hh24 _mi_ss', 'NLS_DATE_LANGUAGE=''AMERICAN''') || '.log' AS spool_file FROM v $database;
            ------------------------------------------------------------------------------------------------------------
            ------------------------------------------------------------------------------------------------------------
            ------------------------------------------------------------------------------------------------------------
            ------------------------------------------------------------------------------------------------------------
SQL> SPOOL &spool_file
SQL> exec dbms_registry.set_session_namespace('SERVER')
PL/SQL procedure successfully completed.
SQL> ALTER SESSION SET current_schema = SYS;
Session altered.
SQL> PROMPT Updating registry...
Updating registry...
SQL> INSERT INTO registry$history
  2    (action_time, action,
  3     namespace, version, id,
  4     bundle_series, comments)
  5  VALUES
  6    (SYSTIMESTAMP, 'APPLY',
  7     SYS_CONTEXT('REGISTRY$CTX','NAMESPACE'),
  8     '11.2.0.3',
  9     1,
  10     'CPU',
  11     'CPUJan2012');
1 row created.
SQL>  COMMIT;
Commit complete.
SQL> SPOOL off
SQL> SET echo off
Check the following log file for errors:
/u01/app/oracle/product/11.2.0/dbhome_ 1/cfgtoollogs/catbundle/catbundle_CPU_IKBTEST1_APPLY_2012Mar22_05_22_ 14.log

STEP 16: Run the utlrp.sql script to validate the objects.

SQL>@utlrp.sql

STEP 17. Count the invalid object and make sure it is lesser or same than what it was in earlier count before patching.





Saturday, March 19, 2016

Minimal Downtime Patching Using Cloned RDBMS(Oracle) Home

Jan 2014 patch is out since some time and we need to now apply it as there is no major issue reported with it. The target is 8 node cluster with around 30+ DBs running on it. These are the critical business applications which cannot be taken offline as they are running some critical global HR and Financial apps. We really struggled to get every application owner to agree to same downtime (DBA's nigthmare!!). Also due to approvals issue we have to cancel the patching twice. This was b'coz all are sharing same infrastructure. 

There are two fundamental problems that we have.
1. Amount of downtime. 
2. Get every one to agree to same downtime window, as they all have different business priorities. 

Considering this, we considered the following approach, which solves our both the issues.
Idea is very simple, where we clone the existing RDBMS Home and Patch it and then switch ORACLE HOME of the database to run out of the new patched home. 
So following is the outline of the process.

1. Clone oracle home.
2. Apply the patches on the cloned home.
3. Switch DB to run from new home and run sql script.

We tried this on three node RAC cluster running 11.2.0.3 with latest OPatch.
So first we need to clone the RDBMS home on all the nodes. 
One needs to do this on all oracle rac nodes, in this case on 3 nodes, since this a 3 node RAC.

[oracle@Node1a 11.2.0]$  cp -rP dbhome_1/ dbhome_2/
cp: cannot open `dbhome_1/bin/nmo' for reading: Permission denied
cp: cannot open `dbhome_1/bin/nmb' for reading: Permission denied
cp: cannot open `dbhome_1/bin/nmhs' for reading: Permission denied
[oracle@Node1a 11.2.0]$ pwd
/u01/app/oracle/product/11.2.0

-- Unset all the ENV variables to make sure you dont point to ORIGINAL ENV

[oracle@Node1a 11.2.0]$ unset ORACLE_HOME
[oracle@Node1a 11.2.0]$ unset PATH
[oracle@Node1a 11.2.0]$ unset ORACLE_SID

[oracle@Node1a 11.2.0]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2
[oracle@Node1a 11.2.0]$ cd $ORACLE_HOME
[oracle@Node1a ~]$ cd $ORACLE_HOME
[oracle@Node1a dbhome_2]$ cd clone/bin/

[oracle@Node1a bin]$ ./clone.pl ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB_home2 ORACLE_BASE=/u01/app/oraInventory

./runInstaller -clone -waitForCompletion  "ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2" "ORACLE_HOME_NAME=OraDB_home2" 

"ORACLE_BASE=/u01/app/oraInventory" -silent -noConfig -nowait
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3820 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-03-31_08-01-10AM. Please wait ...Oracle Universal Installer, Version 

11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.

You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2014-03-31_08-01-10AM.log
.................................................................................................... 100% Done.

Installation in progress (Monday, March 31, 2014 7:01:19 AM CDT)
...............................................................................                                                 79% Done.
Install successful

Linking in progress (Monday, March 31, 2014 7:01:26 AM CDT)
Link successful

Setup in progress (Monday, March 31, 2014 7:02:12 AM CDT)
Setup successful

End of install phases.(Monday, March 31, 2014 7:02:35 AM CDT)
WARNING:
The following configuration scripts need to be executed as the "root" user.
/u01/app/oracle/product/11.2.0/dbhome_2/root.sh
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts

The cloning of OraDB_home2 was successful.

Run Root script - 
[root@Node1a ~]# cd /u01/app/oracle/product/11.2.0/dbhome_2/
[root@Node1a dbhome_2]# ./root.sh
Check /u01/app/oracle/product/11.2.0/dbhome_2/install/root_Node1c.intra.searshc.com_2014-03-31_08-04-35.log for the output of root script

This clone script needs be executed on all nodes of the cluster

UpdateNodeList - 
Command - 
$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={Node1a,Node1b,Node1c}"

This script needs be executed on all nodes of the cluster

[oracle@Node1a bin]$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_2
[oracle@Node1a bin]$
[oracle@Node1a bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={Node1a,Node1b,Node1c}"
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 3819 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

[oracle@Node1b bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={Node1a,Node1b,Node1c}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3820 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

[oracle@Node1c bin]$  ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={Node1a,Node1b,Node1c}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3820 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

NOTE - THIS IS ONLY NEEDED IF YOU ARE PATCHING ON EXADATA. IF YOU DO THIS ON NON EXADATA SYSTEMS, THE START OF THE DB POST HOME SWITCH WILL FAIL.
----------------------------------------------------------------------------------------------------
As the software owner, relink the oracle binary to use RDS. The cloning step causes the oracle binary to be relinked with UDP instead of RDS:
[oracle@Node1a lib]$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_2
[oracle@Node1a lib]$ pwd
/u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib

make -f ins_rdbms.mk ipc_rds ioracle

[oracle@Node1a lib]$ make -f ins_rdbms.mk ipc_udp ioracle
rm -f /u01/app/oracle/product/11.2.0/dbhome_2/lib/libskgxp11.so
cp /u01/app/oracle/product/11.2.0/dbhome_2/lib//libskgxpr.so /u01/app/oracle/product/11.2.0/dbhome_2/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11.2.0/dbhome_2/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib/ 

..............................
/u01/app/oracle/product/11.2.0/dbhome_2/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11.2.0/dbhome_2/lib
test ! -f /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracle ||\
           mv -f /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracle /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracleO
mv /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/lib/oracle /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracle
chmod 6751 /u01/app/oracle/product/11.2.0/dbhome_2/bin/oracle

Post-clone verification:

Verify that all nodes have the same version of OPatch. If these versions do not match, you may need to update OPatch on the nodes running older versions so that all nodes have the same version.

Command - $ORACLE_HOME/OPatch/opatch version

[oracle@Node1a lib]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 11.2.0.3.4
OPatch succeeded.

Command - $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node

[oracle@Node1a lib]$ $ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME | grep node
Rac system comprising of multiple nodes
  Local node = Node1a
  Remote node = Node1b
  Remote node = Node1c


Now as you the new cloned home you need to apply the patch on this new home. In this case I am planning to apply Jan 2014 SPU. Download and copy the patch on target system.

PATCHING OUTPUT -

[oracle@Node1a 17478415]$ /u01/app/oracle/product/11.2.0/dbhome_2/OPatch/opatch napply -skip_subset -skip_duplicate
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.2.0/dbhome_2
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11.2.0/dbhome_2/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch2014-03-31_08-08-27AM_1.log

Verifying environment and performing prerequisite checks...

The following patches are identical and are skipped:
[ 13742433 13742434 13742435 13742436 13742438 14062795 14062797 14480675 14480676 15862016 15862017 15862018 15862019 15862020 15862021 15862022 

15862023 15862024 16314467 16794241 16794242 16794244  ]

Checking skip_duplicate
Checking skip_subset

Patching in rolling mode.

Remaining nodes to be patched:
'Node1b' 'Node1c'
What is the next node to be patched?
Node1b
You have selected 'Node1b' from 'Node1b' 'Node1c'

The node 'Node1b' will be patched next.
.......................
The node 'Node1c' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'Node1c'.
(Oracle Home = '/u01/app/oracle/product/11.2.0/dbhome_2')

Is the node ready for patching? [y|n]
Y
User Responded with: Y
Updating nodes 'Node1c'
   Apply-related files are:

Please shutdown Oracle instances running out of this ORACLE_HOME on 'Node1b'.
(Oracle Home = '/u01/app/oracle/product/11.2.0/dbhome_2')

Is the node ready for patching? [y|n]
Y
User Responded with: Y
Updating nodes 'Node1b'
   Apply-related files are:
     FP = "/u01/app/oracle/product/11.2.0/dbhome_2/.patch_storage/NApply/2014-03-31_08-08-27AM/rac/copy_files.txt"

The node 'Node1c' has been patched.  You can restart Oracle instances on it.

Patches 17333197,17333198,17333199,17333202,17333203,17478415,17748830,17748831,17748832,17748833,17748834,17748835 successfully applied.
OPatch Session completed with warnings.
Log file location: /u01/app/oracle/product/11.2.0/dbhome_2/cfgtoollogs/opatch/opatch2014-03-31_08-08-27AM_1.log
OPatch completed with warnings.

As you can see now that we have patch in place on new home, we are ready to switch the DB to new home. I am currently doing this for only one database i.e. GPDB

Confirm the database is using the old ORACLE_HOME with the following command

[oracle@Node1b lib]$ srvctl config database -d gpdb -a
Database unique name: GPDB
Database name: GPDB
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1  <This is the old home>

Modify the database resource using the following command.

[oracle@Node1b ~]$ srvctl modify database -d GPDB -o /u01/app/oracle/product/11.2.0/dbhome_2
[oracle@Node1b ~]$ srvctl config database -d gpdb -a
Database unique name: GPDB
Database name: GPDB
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_2   <This is the new home>

Edit the /etc/oratab files on all nodes to set new ORACLE_HOME for the database instance on each node

GPDB:/u01/app/oracle/product/11.2.0/dbhome_2:N

One instance at a time, stop the instance and start the instance. As the instance is started, it will pick up the new ORACLE_HOME defined in the 

database configuration and use that for restarting. 
[oracle@Node1b ~]$ srvctl stop database -d GPDB
[oracle@Node1b ~]$ srvctl start database -d GPDB

Now connect to DB and run following query - 

SQL> @?/rdbms/admin/catbundle.sql cpu apply; 

Once the query is done use the following query to check whether patch is applied or not.

SQL> select * from registry$history;
31-MAR-14 10.46.55.494052 AM                                                APPLY                          SERVER
11.2.0.3                                9
CPUJan2014
CPU