How to apply PSU Patch in Oracle RAC environment

Patch Description:

Patch 26030799 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.170718

(Jul2017) (Includes Database PSU 11.2.0.4.170718)

Platform: Linux x86-64, Solaris SPARC, Solaris x86-64, IBM AIX, HP-UX Itanium

Released: July 18, 2017

Notes:

• Patch can be applied to both Oracle Database Home refers to Enterprise Edition or Standard Edition Database software.

• The GI PSU patch includes updates for both the Clusterware home and Database home that can be applied in a rolling fashion.

• This patch is Database Vault installable.

• This patch is Data Guard Standby First Installable.

• Data Guard Standby-First Patch Apply for details on how to remove risk and reduce downtime when applying this patch.

• GI PSUs are cumulative and include the Database PSU and associated CPU program security content.

Check whether below components are configured in your environment.
Q: Is Audit Vault enabled in your database?
Ans: NO

Q: Is Dataguard configured in your environment?
Ans: NO

Note: If Dataguard is configured in your environment then apply the patch DR first.
Then you can start to apply patch to production DC.

Step 1: Client Confirmation
Get the confirmation from client for Patching Activity at least a week prior activity.

Advantages:

• To get total number of servers to be patched.
• To download a patch.
• To upload the patch to target servers.
• To perform prerequisites.
• To confirm client if you find any conflict in prerequisites.
• To inform client that we are ready for patching if no conflict found.

Step 2: Data Capture
Check the current opatch version

By Grid User:
$opatch version

By Oracle User:
$opatch version

You must use the OPatch utility version 11.2.0.3.6 or later to apply this patch.
Oracle recommends that you use the latest released OPatch version for 11.2 releases, which is available for download from My Oracle Support patch 6880880 by selecting ARU link for the 11.2.0.0.0 release. Replace the same with existing opatch version by taking backup of the current one.
$opatch version => It should be 11.2.0.3.6 or later

Step 3: Download Opatch
Go to Oracle Metalink Support, select valid Oracle RDBMS Version and Operating System details, and download Grid Infrastructure PSU patch for RAC database.

Step 4: Upload Patch
Upload this patch file to the target location and unzip the same.

Step 5: Perform Prerequisites
You will get any conflict if any after executing below commands. If you will not get any conflict then you are OK to go ahead.

As a grid user (Go to the Patch Number Location):
$opatch lsinventory -detail -oh $GRID_HOME
$opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$opatch lsinventory -detail -oh $GRID_HOME
$crsctl check crs
$id

As an oracle user (Go to the Patch Number Location):
$opatchlsinventory -detail -oh $ORACLE_HOME
$opatch prereq CheckConflictgainstOhWithDetail -ph ./
$opatch lsinventory -detail -oh $ORACLE_HOME
$srvctl config database -d db_name -v -f
$ps –ef | grep pmon
$ps –ef | grep tns
$id
SQL>set lines 300 pages 3000
SQL>col action_time for a34
SQL>col version for a14
SQL>col comments for a28
SQL>col bundle_series for a28
SQL>col action_name for a23
SQL>col action for a10
SQL>col namespace for a9
SQL>select action,bundle_series,comments,action_time,version from registry$history;

As root User:
#crsctl check cluster -all

Step 6: Final Confirmation
Initiate mail to client prior the activity that “We are starting patching activity in next 10-15 minutes.

Step 7: Backup
Take tar backup of Both Grid and Oracle Home

By grid User:
$tar -cvzf /backup_location/TAR_BKP_Oracle_Binaries.tar.gz $GRID_HOME

By Oracle User:
$tar -cvzf /backup_location/TAR_BKP_Oracle_Binaries.tar.gz $ORACLE_HOME


Step 8: OCM Configuration
You need to create OCM (Oracle Configuration Manager) response file which is not an optional. You can create it by either oracle user or grid user.

$cd $GRID_HOME
/u01/app/11.2.0/grid
$GRID_HOME/OPatch/ocm/bin/emocmrsp  -no_banner -output /home/grid/response_rac1.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file /home/grid/response_rac1.rsp) was successfully created.

Opatch Auto:
The Opatch utility has automated the patch application for the Oracle Grid Infrastructure (GI) home and the Oracle RAC database homes when run with root privileges. It must be executed on each node in the cluster if the GI home or Oracle RAC database home is in non-shared storage. The utility should not be run in parallel on the cluster nodes.

Step 9: Apply Patch
1. GI Home and the Database Homes that are not shared and ACFS file system is not configured.
2. GI Home is not shared, Database Home is shared, ACFS may be used.
In our case, both Grid and Oracle Homes are not shared and we are not using ACFS file system, hence we are following 1^st method of applying patch.

As root user, execute the following command on each node of the cluster:
#/u01/app/11.2.0/grid/OPatch/opatch auto /home/grid/26030799 -ocmrf /home/grid/response_rac1.rsp
Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /home/grid -patchn 26030799 -ocmrf /home/grid/response_rac1.rsp -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params
This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2017-08-05_11-57-19.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/11.2.0/grid/cfgtoollogs/opatchauto2017-08-05_11-57-19.report.log
2017-08-05 11:57:19: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Unable to determine if /u01/app/oracle/product/11.2.0/dbhome_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Unable to determine if /u01/app/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Stopping RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully
patch /home/grid/26030799/25869727  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1
patch /home/grid/26030799/25920335/custom/server/25920335  apply successful for home  /u01/app/oracle/product/11.2.0/dbhome_1
Stopping CRS...
Stopped CRS successfully
patch /home/grid/26030799/25869727  apply successful for home  /u01/app/11.2.0/grid
patch /home/grid/26030799/25920335  apply successful for home  /u01/app/11.2.0/grid
patch /home/grid/26030799/22502505  apply successful for home  /u01/app/11.2.0/grid
Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0/dbhome_1 ...
Started RAC /u01/app/oracle/product/11.2.0/dbhome_1 successfully
opatch auto succeeded.

Q: What does the below command do?
#/u01/app/11.2.0/grid/OPatch/opatch auto /home/grid/26030799 -ocmrf /home/grid/response_rac1.rsp
Ans: It performs below tasks

1. First it will verify the below detail

• crs_home=/u01/app/11.2.0/grid     
• owner=grid     
• opatch_ver=11.2.0.3.16
• rac_home=/u01/app/oracle/product/11.2.0/dbhome_1     
• owner=oracle     
• opatch_ver=11.2.0.3.16

2.  Performs below prerequisites for both Grid and Database.
Oracle:
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckComponents -ph /home/grid/26030799/25869727 -oh /u01/app/oracle/product/11.2.0/dbhome_1
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckComponents -ph /home/grid/26030799/25920335/custom/server/25920335 -oh /u01/app/oracle/product/11.2.0/dbhome_1
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /home/grid/26030799/25869727 -oh /u01/app/oracle/product/11.2.0/dbhome_1
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /home/grid/26030799/25920335/custom/server/25920335 -oh /u01/app/oracle/product/11.2.0/dbhome_1

Grid:
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /home/grid/26030799/25869727 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /home/grid/26030799/25920335 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /home/grid/26030799/22502505 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /home/grid/26030799/25869727 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /home/grid/26030799/25920335 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /home/grid/26030799/22502505 -oh /u01/app/11.2.0/grid

3. Stops dbconsole and agent.
/u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl stop dbconsole
/u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl stop agent

4. Check patch applicability the patch to both GI and Oracle Home.
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckApplicable -ph /home/grid/26030799/25869727 -oh /u01/app/oracle/product/11.2.0/dbhome_1
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckApplicable -ph /home/grid/26030799/25920335/custom/server/25920335 -oh /u01/app/oracle/product/11.2.0/dbhome_1

5. Stops the Oracle Home
/u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl stop home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /u01/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rac1 -f

6. Runs the pre-patch script for ORACLE_HOME.
/home/grid/26030799/25920335/custom/server/25920335/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1

7. Applies patch to Oracle Home first using below command
Oracle Home:
u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /home/grid/26030799/25869727 -local -silent -ocmrf /home/grid/response_rac1.rsp -oh /u01/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /home/grid/26030799/25920335/custom/server/25920335 -local -silent -ocmrf /home/grid/response_rac1.rsp -oh /u01/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc

8. Runs post patch script for Database Home
/home/grid/26030799/25920335/custom/server/25920335/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_1

9. Runs root unlock script
/u01/app/11.2.0/grid/crs/install/rootcrs.pl -unlock : run as root
/sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin : run as root

- Checks patch applicability for Grid Home
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /home/grid/26030799/25869727 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /home/grid/26030799/25920335 -oh /u01/app/11.2.0/grid
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /home/grid/26030799/22502505 -oh /u01/app/11.2.0/grid

- Applies patch to Grid Home
/u01/app/11.2.0/grid/OPatch/opatch napply /home/grid/26030799/25869727 -local -silent -ocmrf /home/grid/response_rac1.rsp -oh /u01/app/11.2.0/grid -invPtrLoc /u01/app/11.2.0/grid/oraInst.loc
/u01/app/11.2.0/grid/OPatch/opatch napply /home/grid/26030799/25920335 -local -silent -ocmrf /home/grid/response_rac1.rsp -oh /u01/app/11.2.0/grid -invPtrLoc /u01/app/11.2.0/grid/oraInst.loc
/u01/app/11.2.0/grid/OPatch/opatch napply /home/grid/26030799/22502505 -local -silent -ocmrf /home/grid/response_rac1.rsp -oh /u01/app/11.2.0/grid -invPtrLoc /u01/app/11.2.0/grid/oraInst.loc
      Starts dbconsole and agent
/u01/app/11.2.0/grid/bin/emctl start dbconsole
      Runs post-root scripts
/u01/app/11.2.0/grid/rdbms/install/rootadd_rdbms.sh : run as root
/u01/app/11.2.0/grid/crs/install/rootcrs.pl -patch : run as root
/u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl start dbconsole
/u01/app/oracle/product/11.2.0/dbhome_1/bin/emctl start agent
        Starts Oracle Home
/u01/app/oracle/product/11.2.0/dbhome_1/bin/srvctl start home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /u01/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rac1

Step 10: Check everything is up and running
As a grid user (Go to the Patch Number Location):
$opatch lsinventory -detail -oh $GRID_HOME
$opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$opatch lsinventory -detail -oh $GRID_HOME
$crsctl check crs
$id
As an oracle user (Go to the Patch Number Location):
$opatchlsinventory -detail -oh $ORACLE_HOME
$opatch prereq CheckConflictgainstOhWithDetail -ph ./
$opatch lsinventory -detail -oh $ORACLE_HOME
$srvctl config database -d db_name -v -f
$ps –ef | grep pmon
$ps –ef | grep tns
$id
SQL>set lines 300 pages 3000
SQL>col action_time for a34
SQL>col version for a14
SQL>col comments for a28
SQL>col bundle_series for a28
SQL> col action_name for a23
SQL> col action for a10
SQL> col namespace for a9
SQL> select action,bundle_series,comments,action_time,version from registry$history;

As root User:
#crsctl check cluster -all

Step 11: Repeat Step 8 and Step 9 on 2^nd node.

Step 12: Post-Patch Installations
Perform these steps on only one node.

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the

Bu Oracle User:
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP

SQL> @catbundle.sql psu apply

SQL> QUIT
Check the following log files in $ORACLE_BASE/cfgtoollogs/catbundle for any errors: 
catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log

Step 13: Run below object compilation script
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

Step 14: Run OJVM script
This patch now includes the OJVM Mitigation patch (Patch:19721304). If an OJVM PSU is installed or planned to be installed, no further actions are necessary. Otherwise, the workaround of using the OJVM Mitigation patch can be activated.

cd $ORACLE_HOME/rdbms/admin
$sqlplus / as sysdba
SQL > @dbmsjdev.sql
SQL > exec dbms_java_dev.disable

Step 15: Upgrade the catalog
If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it:

$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;

Rollback Plan:

·    GI Home and Database Homes that are not shared and ACFS file system is not configured.

·    GI Home is not shared; Database Home is shared and ACFS may be used.

Step 1: As root user, execute the following command on each node of the cluster.

# opatch auto /home/grid/26030799 -rollback -ocmrf /home/grid/response_rac1.rsp

Step 2: Check everything is up and running fine after rolling back the above patch.

Step 3: Execute Step 1 and Step 2 in 2^nd node.

Step 4: Follow the below steps only on the node on which you were previously executed catbundle.sql script during the patch application.:

Start all database instances running from the Oracle home.

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA 

SQL> STARTUP

SQL> @catbundle_PSU_<ORACLE_SID>_ROLLBACK.sql

SQL> QUIT

Step 5: Rollback OJVM script.

If the OJVM PSU was applied for a previous GI PSU patch, you may see invalid Java classes after execution of the catbundle.sql script. If this is the case, run utlrp.sql to re-validate these Java classes. 

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog

SQL> CONNECT / AS SYSDBA

SQL> @utlrp.sql   

Step 6: Verify opatch inventory whether you have successfully rolled back the previously applied patch.

By Grid and Oracle User:

$opatch lsinventory -detail -oh $ORACLE_HOME