Step by step - How to upgrade from 11.2.0.3 to 11.2.0.4 Grid and RDBMS in RAC & Data Guard Environment

This document is for upgrading databases in RAC and dataguard environment. In this upgrade, Grid and RDBMS home is upgraded. This upgrade is done by out-of-place approach(new home for both GI and DB). There are basically 6 steps to complete this requirement:

Step 1: Prerequisite checks and backups

Step 2: New Grid Binary Installation:

            - Fill in the response file
            - Run the prerequisite checks
            - Install the new GI binary

Step 3: Manual steps performed after Grid binary installation to upgrade the Grid.

Step 4: New RDBMS installation

           -Fill in the response file
           -Run the prerequisite checks
           -Install the new RDBMS binary

Step 5: Manual steps performed after new binary installation to upgrade the RDBMS

Step 6: Upgrade the database on standby side.

Now we will move step by step in detail:

Step 1: Prerequisite checks and backups

1) Check storage space for the new binary to be installed. We must have around 20 GB of space         available for each homes(Grid and RDBMS) on each node.
Also issue df -kh and make sure we space available in /tmp and /(root).

2) Remove the audit files and any large log files on each node(not deleting will slowdown the upgrade).

$cd $GRID_HOME/rdbms/audit
$rm *.aud

3) Take full database backup using RMAN and also take full database export.

4) Backup of Grid and RDBMS binaries using tar.

5) Backup of Oracle Inventory

6) Backup of OCR on each nodes.

7) Backup of ASM metadata.

8) Secure dataguard.

Step 2: New Grid home installation:

1) Fill in the response file for the new binary home. Edit the response file according to the requirement.

2) Run the installer to install the GRID using this response file.

Step 3: Grid Upgrade:

Manual Grid upgrade step after new binary installation:

1) On each node, copy listener.ora and sqlnet.ora to new Grid home(11.2.0.4).

2) If necessary, adjust existing ASM instance memory parameter:

     Log in as sysasm:
     SQL>show parameter memory_target

     if the value is smaller than 1536 m, then issue the following command:

     SQL>alter system set memory_target=1536m scope=spfile;

    The number 1536m has proven to be efficient in most environment, the change will not be     effective util next restart.

    SQL>show parameter memory_max_target

   If the value is smaller than 4096, then issue the following command:

   SQL>alter system set memory_max_target=4096 scope=spfile;

   The number 4096m has proven to be sufficient in most environment, the change will not be effective until next restart.

3) Stop clusterware and database resources gracefully on each node.

    #crsctl stop crs

    This command will bring down the database so no need to shutdown the database exclusively.

    Make sure all CRS deamons are sto resopped on each node:

  #crsctl stat resource -t

4) On each node, execute rootupgrade.sh script from new 11.2.0.4 home to upgrade clusterware.

    The rootupgrade.sh script can only be run sequentially node by node, and not in parallel between nodes.

  Note: This script takes 15-20 minutes to complete.

While switching to root user to execute rootupgrade.sh, "su -" or "su - root" provides the full root environment. While sudo, pbrun,"su root" or "su" or similar facilities don't. It is recommended to execute rootupgrade.sh with full root acces to avoid any issue.

5) Once the Grid upgrade script finishes, all the clusterware and database services automatically comes up. If not then manually start it.

6) Verify and cluster and database services on each node.

     #crsctl check crs
     #crsctl stat resource -t
     #ocrcheck -local
     $srvctl status database -d dbname

7) After all the nodes are upgraded, update environment variables to point to new home 11.2.0.4

    Below files needs to be updated:

    1) .bash_profile
    2) /etc/oratab

Note: Once the grid is installed and upgraded in the primary servers, do the grid installation and upgrade on the standby side too. Otherwise it will not allow to install new 11.2.0.4 RDBMS home on standby side.

Step 4: New RDBMS Home Installation

Like grid installation, first thing we need to do is to prepare the response file. Fill all the necessary parameter which are required to install the new home. And then run the installer using this response file. Install new RDBMS(11.2.0.4) on standby also.

Step 5: RDBMS Upgrade

1) Before we start database upgrade, we need to create the restore point.

    SQL>CREATE RESTORE POINT before_upgrade GURANTEE FLASHBACK DATABASE;

2)Copy the parameter file, password file, tnsnames.ora and sqlnet.ora to the new RDBMS home. For      RAC copy on all the nodes.

3) Count invalid object and make a note of it.

4) Check the space in system tablespace.

5) For RAC only, change the instance to exclusive mode(i.e. cluster_database=false)

6) Gather dictionary stats from existing binary home.

      SQL>execute dbms_stats.gather_dictionary_stats;

7) Execute the following pre-check script from the existing home to check instance fulfill pre-upgrade requirement.

    SQL>@/u01/sql/ora_2db/11.2.0.4/rdbms/admin/utlu112i.sql

Fix all the errors which is reported by this script before proceeding to the actual upgrade.
For RAC this step only needs to be executed only once, not from all the nodes.

8) Startup instance with startup upgrade option in SQLPLUS with new oracle home(11.2.0.4).
     For RAC this step needs to be executed from one node only(not from all nodes). And then run the       upgrade script.

     SQL>startup upgrade
   
     SQL>@/u01/sq/ora_2/db/11.2.0.4/rdbms/admin/catupgrd.sql

9) Then restart the instance and run the following sql script:

     SQL>shut immediate

     SQL>startup

     SQL>@/u01/sq/ora_2/db/11.2.0.4/rdbms/admin/catuppst.sql

    (For RAC, this script needs to be executed only from one node)

10) Run the post-upgrade check script from new home.

      SQL>@/u01/sq/ora_2/db/11.2.0.4/rdbms/admin/utlu112s.sql

      (For RAC, this script needs to be executed from only one node)

11) Recompile objects and compile the object list to ensure no new invalid objects

      SQL>@?/rdbms/admin/utluiobj.sql

      SQL>@?/rdbms/admin/utlrp.sql

      SQL>@?/rdbms/admin/utluiobj.sql

     (for RAC, this script needs to be executed only from one node)

12) For RAC, fallback instance from exclusive mode(cluster_database=true)

13) Update database resources in CRS registry:
   
       $srvctl upgrade database -d dbname -o /u01/sq/ora_2/db/11.2.0.4

14) Startup the remaining instances of the database.

      $srvctl start instance -d dbname -i instance_name

15) After database is upgraded, update the environment variable so that it points to 11.2.0.4 in the              below files:

       /etc/oratab for oracle user
       .bash_profile for oracle user
       listener.ora file

16) If there is oracle valet then add VNCR parameter in the listener.ora file.
      (for RAC, it applies to all nodes).


17) Shutdown and restart the database.

18) Ask application team to verify everything is running fine for them.

19) Drop restore point befor_upgrade.


STEP 6 - UPGRADE THE DATABASE ON STANDBY SIDE

Perform these steps only if there is a physical standby database associated with the database being upgraded.

1) Copy below file from 11.2.0.3 home to new home 11.2.0.4:

      tnsnames.ora
      sqlnet.ora
      parameter file
      password file

2) Shutdown the primary database as follows:

    $srvctl stop database -d DBNAME -o immediate

3) Shutdown the standby database as follows:

   $srvctl stop database -d DBNAME

4) Edit the /etc/oratab and listener.ora file to point to the new 11.2.0.4 home

5) Update the OCR configuration

   $srvctl upgrade database -d DBNAME -o /u01/sq/ora_2/db/11.2.0.4

6) Start the standby database in mount mode from new home 11.2.0.4

    $srvctl start database -d DBNAME -o mount

7) Start the primary database in open mode

    $srvctl start database -d DBNAME

8) Start the redo apply on standby database

    SQL>alter database recover managed standby database disconnect from session

9)  Verify the standby alert log file and make sure logs are shipping properly, also check the archive
      gap.


Suggestions are most welcome :)