Sunday, February 14, 2016

Query to verify Number of connections connection to each node by service

Here is the query to check # of connections on each node by the service_name.

col service_name format a25
select 
inst_id,
count(*) as connections,
service_name
from 
gv$session
group by
inst_id, service_name;

Friday, February 12, 2016

RAC SERVICE EXAMPLE


My notes on Oracle RAC Services --Moid

1. To check the status of all services in a RAC Database
10g
srvctl config service -d racdb
11g
srvctl config service -d racdb | grep -IE "Service name|Preferred instances"


2. To check the status of the service called crm
10g
srvctl status service -d racdb -s crm
11g



3a. To add a service
srvctl add service -d racdb -s CLROCM_DataRead_P -r racdb1,racdb2


 srvctl add service -d racdb-s oocert_n1pn2p -r ,oocert2 -P BASIC -e SELECT

x39 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = x39-3001.xyz.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = x39_n1pn2p)
    )
  )

set linesize 200
set pagesize 200
col username format a12
col machine format a20
Select
inst_id,
username,
machine,
failover_type,
failover_method,
failed_over
from
gv$session
where
username='AFSHAN'
order by 1,machine;



3b) Verify service is created.
srvctl status service -d racdb -s CLROCM_DataRead_P


3c) Enable the service
srvctl enable service -d racdb -s CLROCM_DataRead_P


3d) If service is already enabled, then start the service.
10g
srvctl start service -d racdb -s CLROCM_DataRead_P

11g



3e) Verify service is created, enabled and started.
10g
srvctl status service -d racdb -s CLROCM_DataRead_P


11g




4) How to relocated the service?
srvctl relocate service -d AQCLP -s AQSVC3P -i AQCLP1  -t AQCLP2 -f

srvctl relocate service -d AQCLP -s AQSVC6P -i AQCLP1 -t AQCLP2 -f
Note: Contributed by Ahmad Bhutta

RAC : SERVICES

In this post, I will explain the concept of services and their benefits in 11g R2 RAC.
What is a service?
A service is a logical grouping of sessions performing similar kind of work. They  enable you to group database workloads and route work to the optimal instances that are assigned to offer the service.It is recommended that all the users who connect using a service have the same service-level requirements. We can use services to manage workloads or a group of applications. For example , all on-line users can use one service whereas batch users can use another service and reporting can use yet another service.
Benefits of services:
– Services make the work more manageable, measurable, tunable and recoverable.
– Services provide an abstraction layer and permit clients and middle tiers to access required data from the database independent of where the instances reside.
– Dynamic: As the load increases/reduces, the number of instances supporting a service can be increased/decreased accordingly.
- High availability : In case an instance crashes, the services supported by it failover to another surviving instance.
- Tuning and Monitoring Aid
. Tracing can be enabled for all the sessions who connect using a service: This can be used to identify the culprit SQL statements issued by the users connected via a service.  . Performance statistics are collected and aggregated by service: This can be used to find out the performance of all the sessions belonging to a service.
. Two Performance thresholds can be specified for a service: SERVICE_ELAPSED_TIME and SERVICE_CPU_TIME: The AWR monitors the service time and CPU time and publishes alerts when the performance esceeds the thresholds. In response to these alerts, you can change the priority of a job, stop overloaded process , or relocate, expand, shrink , start or stop a service.
- Resource Management Aid : We can bind resource consumer groups to services. The users who connect using a service will  get the resources as per the allocation specified for the respective consumer group in the active resource plan. For example, if OLTP and BATCH services are defined to run on the same instance and our requirement is as follows:
. During day time OLTP users should get more resources whereas BATCH users should get less resources
  . During night, BATCH users should get more resources whereas OLTP users should get less resources
We can have two consumer groups : OLTP_GRP mapped to OLTP service and
                                                                          BATCH_GRP mapped to BATCH service
so that all users who connect using OLTP service are assigned OLTP_GRP and all users who connect using BATCH service are assigned BATCH_GRP .
We can have two resource plans :
  . DAY_PLAN in which OLTP_GRP is given more resources and BATCH_GRP is given less resources.
  . NIGHT_PLAN in which BATCH_GRP is given more resources and OLTP_GRP is given less resources.
- Job Control Aid : We can associate jobs with a job class and job class can be mapped to service.
– If service is mapped to a consumer group, the jobs will get the resources as per the allocation specified for the respective consumer group in the active resource plan. If  jobs of a job class are taking more time than desired, we can change the consumer group mapping of the respective service to one with more resources.
– If the instance running the jobs of a job class crashes, the services on the crashed instance fail over to another instance and all the jobs of the job class mapped to thst service also failover along with it.
i.e.
       Job1  —–+
       Job2  ——|——– Jobclass1 —— Service1
       Job3  ——+                                  |
                                                  Consumer group
                                                          |
                                                 Resource Plan
– Using Transparent Application Failover:  When a node crashes, all the services running on the instance hosted by that node fail over to another instance. From now onwards, the users who try to connect using failed over services will connect to the instances supporting that service. But what happens to the sessions who were already connected using that service at the time of crash? That is decided by the TAF (Transparent application Failover) policy which can be defined for the service. We can have 3 types of TAF policies : Basic, Preconnect and None.
BASIC failover. In this approach, the application connects to a backup node only after the primary connection fails. This approach has low overhead, but the end user experiences a delay while the new connection is created.
PRECONNECT failover. In this approach, the application simultaneously connects to both a primary and a backup node. This offers faster failover, because a pre-spawned connection is ready to use. But the extra connection adds everyday overhead by duplicating connections.
  We can also specify FAILOVER_TYPE for a service i.e. what kind of activity going on in the connected sessions will be resumed after the failover .  It can be Select or Session.
SELECT failover. With SELECT failover, Oracle Net keeps track of all SELECT statements issued during the transaction, tracking how many rows have been fetched back to the client for each cursor associated with a SELECT statement. If the connection to the instance is lost, Oracle Net establishes a connection to another Oracle RAC node and re-executes the SELECT statements, repositioning the cursors so the client can continue fetching rows as if nothing has happened. The SELECT failover approach is best for data warehouse systems that perform complex and time-consuming transactions.
 SESSION failover. When the connection to an instance is lost, SESSION failover results only in the establishment of a new connection to another Oracle RAC node; any work in progress is lost. SESSION failover is ideal for online transaction processing (OLTP) systems, where transactions are small.
The following types of transactions do not automatically fail over and must be restarted by TAF:
  • Transactional statements. Transactions involving INSERT, UPDATE, or DELETE statements are not supported by TAF.
  •  ALTER SESSION statements. ALTER SESSION and SQL*Plus SET statements do not fail over.
  • Transactions using temporary segments in the TEMP tablespace and global temporary tables do not fail over.
  • PL/SQL package states. PL/SQL package states are lost during failover.
– Connection Load balancing : If we have multiple instances supporting a service, We can spread user connections across all those instances. For each service, you can define the method that you want the listener to use for load balancing by setting the connection load balancing goal, CLB_GOAL. It can have two values : CLB_GOAL_LONG(default) and CLB_GOAL_SHORT.
Where’s the configuration of services stored?
Data Dictionary (Bulk of service configuration)
DBA_SERVICES
DBA_RSRC_GROUP_MAPPINGS
DBA_THRESHOLDS
DBA_SCHEDULER_JOB_CLASSES
TNSNAMES.ORA
Server-side
Entries for LOCAL_LISTENER and REMOTE_LISTENER
Client-side (in tnsnames.ora)
CONNECT_DATA
Special entries for PRECONNECT services
Resource profile in Cluster Registry (OCR)
Resource
Start/stop script
Dependencies
Restart policy
Stringpairs
Instance list
Preferred/available instances
Enabled/disabled
TAF policy
Initialization Parameters
LOCAL_LISTENER
REMOTE_LISTENERS
DB_DOMAIN
DISPATCHERS
STATISTICS_LEVEL
References:

Wednesday, February 10, 2016

DATAGUARD BROKER COMMAND

dgmgrl utility
dgmgrl - Data Guard Manager (Observer) Utility, in Oracle

$dgmgrl [-silent | -echo] [username/password[@connect_identifier] [dgmgrl_command]]
$dgmgrl  /
$dgmgrl sys/india123
$dgmgrl sys/pwd@oltp
$dgmgrl sys/test@dgprimary "show database 'prod'"
$dgmgrl -logfile observer.log / "stop observer"
$dgmgrl -silent sys/test@dgprimary "show configuration verbose"
$dgmgrl  / "show configuration verbose"

ADD - Adds a standby database to the broker configuration.
DGMGRL> ADD DATABASE db_name [AS CONNECT IDENTIFIER IS conn_identifier] [MAINTAINED AS {PHYSICAL|LOGICAL}];
DGMGRL> ADD DATABASE 'primary' AS CONNECT IDENTIFIER IS primary MAINTAINED AS PHYSICAL;
DGMGRL> ADD DATABASE 'standby' AS CONNECT IDENTIFIER IS standby MAINTAINED AS LOGICAL;
DGMGRL> ADD DATABASE 'standby' AS CONNECT IDENTIFIER IS standby.foo.com;

CONNECT - Connects to an Oracle database instance.
DGMGRL> CONNECT username/password[@connect_identifier]
DGMGRL> CONNECT /
DGMGRL> CONNECT sys;
DGMGRL> CONNECT sys@test;
DGMGRL> CONNECT sys/pwd;
DGMGRL> CONNECT sys/pwd@dwh;
DGMGRL> CONNECT /@dwh;
$dgmgrl connect sys

CONVERT - Converts a database from one type to another (from Oracle 11g).
DGMGRL> CONVERT DATABASE database_name TO {SNAPSHOT STANDBY|PHYSICAL STANDBY};
DGMGRL> CONVERT DATABASE 'standby' to SNAPSHOT STANDBY;

DGMGRL> CONVERT DATABASE 'standby' to PHYSICAL STANDBY;

Prints Popular Posts Dictionary English dictionary online Archiver Automatic storage management

CREATE - Creates a broker configuration. 

DGMGRL> CREATE CONFIGURATION config_name AS PRIMARY DATABASE IS db_name CONNECT IDENTIFIER IS conn_ident;

DGMGRL> CREATE CONFIGURATION 'dg' AS PRIMARY DATABASE IS 'primary' CONNECT IDENTIFIER IS primary;

DGMGRL> CREATE CONFIGURATION 'dg_test' AS PRIMARY DATABASE IS 'test' CONNECT IDENTIFIER IS test;

DISABLE - Disables a configuration, a database, or fast-start failover (FSFO).

DGMGRL> DISABLE CONFIGURATION;
DGMGRL> DISABLE CONFIGURATION;

DGMGRL> DISABLE DATABASE database_name;
DGMGRL> DISABLE DATABASE 'standby';

DGMGRL> DISABLE FAST_START FAILOVER [FORCE | CONDITION condition];
DGMGRL> DISABLE FAST_START FAILOVER;
DGMGRL> DISABLE FAST_START FAILOVER FORCE;
DGMGRL> DISABLE FAST_START FAILOVER CONDITION '1578';

EDIT - Edits a configuration, database, or instance.
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS {MaxProtection|MaxAvailability|MaxPerformance};
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXPROTECTION;
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXPERFORMANCE;


DGMGRL> EDIT CONFIGURATION SET PROPERTY property_name = value;
DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverThreshold = 45;
DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverAutoReinstate = FALSE;
DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverAutoReinstate = TRUE;
DGMGRL> EDIT CONFIGURATION SET PROPERTY BYSTANDERSFOLLOWROLECHANGE= 'NONE';


DGMGRL> EDIT DATABASE database_name SET PROPERTY property_name = value;
DGMGRL> EDIT DATABASE standby SET PROPERTY 'LogArchiveFormat'='log_%t_%s_%r_%d.arc';

DGMGRL> EDIT DATABASE prodb SET PROPERTY LogXptMode=SYNC;
DGMGRL> EDIT DATABASE prodb SET PROPERTY LogXptMode=ASYNC;
DGMGRL> EDIT DATABASE prodb SET PROPERTY LogXptMode=ARCH;
DGMGRL> EDIT DATABASE standby SET PROPERTY LogShipping=OFF;
DGMGRL> EDIT DATABASE standby SET PROPERTY LogShipping=ON;
DGMGRL> EDIT DATABASE prodb SET PROPERTY LogArchiveTrace=8;
DGMGRL> EDIT DATABASE prodb SET PROPERTY NetTimeout=60;
DGMGRL> EDIT DATABASE standby SET PROPERTY 'ReopenSecs'=300;
DGMGRL> EDIT DATABASE prodb SET PROPERTY ArchiveLagTarget=1200;
DGMGRL> EDIT DATABASE prodb SET PROPERTY FastStartFailoverTarget='standby_name';
DGMGRL> EDIT DATABASE standby SET PROPERTY 

'StandbyArchiveLocation'='/oradata/archive/';
DGMGRL> EDIT DATABASE standby SET PROPERTY 'DbFileNameConvert' = '/u01/od01/datafile/, /oradisk/od01/datafile/';
DGMGRL> EDIT DATABASE primary SET PROPERTY DelayMins='720';
DGMGRL> EDIT DATABASE prodb SET PROPERTY RedoCompression ='ENABLE'
DGMGRL> EDIT DATABASE prodb SET PROPERTY RedoCompression ='DISABLE'
DGMGRL> EDIT DATABASE primary SET PROPERTY LogArchiveMinSucceedDest =1

DGMGRL> EDIT DATABASE database_name RENAME TO new database_name;
DGMGRL> EDIT DATABASE 'standbyb' RENAME TO 'standby'; 

DGMGRL> EDIT DATABASE database_name SET STATE = state [WITH APPLY INSTANCE = instance_name];
DGMGRL> EDIT DATABASE standby SET STATE='READ-ONLY';
DGMGRL> EDIT DATABASE standby SET STATE='OFFLINE';

DGMGRL> EDIT DATABASE standby SET STATE='APPLY-OFF';
DGMGRL> EDIT DATABASE standby SET STATE='APPLY-ON';

DGMGRL> EDIT DATABASE standby SET STATE='TRANSPORT-OFF';
DGMGRL> EDIT DATABASE standby SET STATE='TRANSPORT-ON';

DGMGRL> EDIT DATABASE prodb SET STATE='LOG-TRANSPORT-OFF';
DGMGRL> EDIT DATABASE standby SET STATE='ONLINE' WITH APPLY INSTANCE=standby2;

DGMGRL> EDIT INSTANCE instance_name [ON DATABASE database_name] SET AUTO PFILE [={init_file_path|OFF}];
DGMGRL> EDIT INSTANCE 'standby1' ON DATABASE 'standby' SET AUTO PFILE='initstandby1.ora';

DGMGRL> EDIT INSTANCE instance_name [ON DATABASE database_name] SET PROPERTY property_name = value;
DGMGRL> EDIT INSTANCE * ON DATABASE database_name SET PROPERTY property_name = value;
DGMGRL> EDIT INSTANCE 'proddb' ON DATABASE 'proddb' SET PROPERTY 'StandbyArchiveLocation'='/oradata/arch/';
Corrupted Data Guard Datafile Displays Information Prints Popular Posts Dictionary

ENABLE - Enables a configuration, a database, or fast-start failover (FSFO).
DGMGRL> ENABLE CONFIGURATION;
DGMGRL> ENABLE CONFIGURATION;

DGMGRL> ENABLE DATABASE database_name;
DGMGRL> ENABLE DATABASE 'standby';

DGMGRL> ENABLE FAST_START FAILOVER [CONDITION condition];
DGMGRL> ENABLE FAST_START FAILOVER;
DGMGRL> ENABLE FAST_START FAILOVER CONDITION '1578';
DGMGRL> ENABLE FAST_START FAILOVER CONDITION "Stuck Archiver";
DGMGRL> ENABLE FAST_START FAILOVER CONDITION 'Corrupted Controlfile';
DGMGRL> ENABLE FAST_START FAILOVER CONDITION 'Corrupted Dictionary';
DGMGRL> ENABLE FAST_START FAILOVER CONDITION 'Inaccessible Logfile';

EXIT - Exits the program.
DGMGRL> EXIT;

FAILOVER - Changes a standby database to be the primary database.
DGMGRL> FAILOVER TO standby_database_name [IMMEDIATE]
DGMGRL> FAILOVER TO "primary";
DGMGRL> FAILOVER TO "snapdb" IMMEDIATE;

HELP - Displays description and syntax for a command.
DGMGRL> HELP [command];
DGMGRL> HELP REINSTATE
DGMGRL> HELP EDIT

QUIT - Exits the program.
DGMGRL> QUIT;

REINSTATE - Changes a database marked for reinstatement into a viable standby.
DGMGRL> REINSTATE DATABASE database_name;
DGMGRL> REINSTATE DATABASE prim1;

REM - Comment to be ignored by DGMGRL.
DGMGRL> REM [comment];

REMOVE - Removes a configuration, Oracle database, or instance.
DGMGRL> REMOVE CONFIGURATION [PRESERVE DESTINATIONS];
DGMGRL> REMOVE CONFIGURATION;
DGMGRL> REMOVE CONFIGURATION PRESERVE DESTINATIONS;


DGMGRL> REMOVE DATABASE database_name [PRESERVE DESTINATIONS];
DGMGRL> REMOVE DATABASE standby;
DGMGRL> REMOVE DATABASE standby PRESERVE DESTINATIONS;

DGMGRL> REMOVE INSTANCE instance_name [ON DATABASE database_name];
DGMGRL> REMOVE INSTANCE inst1 ON DATABASE racdb;

SHOW - Displays information about a configuration, database, instance or FSFO.
DGMGRL> SHOW CONFIGURATION [VERBOSE];
DGMGRL> SHOW CONFIGURATION;
DGMGRL> SHOW CONFIGURATION VERBOSE;
English dictionary online Archiver Automatic storage management Corrupted Data Guard Datafile


DGMGRL> SHOW DATABASE [VERBOSE] db_name [property_name];
DGMGRL> SHOW DATABASE 'standby';
DGMGRL> SHOW DATABASE VERBOSE 'test';
DGMGRL> SHOW DATABASE 'dwhdb' 'StatusReport';
DGMGRL> SHOW DATABASE 'proddb' 'LogXptStatus';
DGMGRL> SHOW DATABASE 'proddb' 'InconsistentProperties';
DGMGRL> SHOW DATABASE 'proddb' 'InconsistentLogXptProps';
DGMGRL> SHOW DATABASE 'primary' 'ArchiveLagTarget';
DGMGRL> SHOW DATABASE 'primary' 'LogShipping';
DGMGRL> SHOW DATABASE 'primary' 'PreferredApplyInstance';
DGMGRL> SHOW DATABASE 'proddb' 'StatusReport';
DGMGRL> SHOW DATABASE 'primary' 'RecvQEntries';
DGMGRL> SHOW DATABASE 'proddb' 'SendQEntries';

DGMGRL> SHOW INSTANCE [VERBOSE] instance_name [property_name] [ON DATABASE db_name];
DGMGRL> SHOW INSTANCE inst1;
DGMGRL> SHOW INSTANCE VERBOSE inst3;
DGMGRL> SHOW INSTANCE primary 'TopWaitEvents';

DGMGRL> SHOW FAST_START FAILOVER;
DGMGRL> SHOW FAST_START FAILOVER;

SHUTDOWN - Shuts down a currently running Oracle instance.
DGMGRL> SHUTDOWN [NORMAL | IMMEDIATE | ABORT];
DGMGRL> SHUTDOWN;
DGMGRL> SHUTDOWN NORMAL;
DGMGRL> SHUT IMMEDIATE;
DGMGRL> SHUT ABORT;

SQL - Executes a SQL statement
DGMGRL> SQL "sql_statement";

START - Starts the fast-start failover(FSFO) observer.
DGMGRL> START OBSERVER [FILE=observer_configuration_file];
DGMGRL> START OBSERVER;

STARTUP - Starts an Oracle database instance.
DGMGRL> STARTUP [RESTRICT] [FORCE] [PFILE=filespec]
[NOMOUNT  |  MOUNT | OPEN [READ ONLY|READ WRITE]];
DGMGRL> STARTUP;
DGMGRL> STARTUP NOMOUNT;
DGMGRL> STARTUP MOUNT;
DGMGRL> STARTUP OPEN;
DGMGRL> STARTUP FORCE;
DGMGRL> STARTUP FORCE RESTRICT NOMOUNT;
DGMGRL> STARTUP PFILE=initdwh.ora NOMOUNT;

STOP - Stops the fast-start failover(FSFO) observer.
DGMGRL> STOP OBSERVER;

SWITCHOVER - Switches roles between a primary and standby database.
DGMGRL> SWITCHOVER TO standby_database_name;
DGMGRL> SWITCHOVER TO "standby";
Displays Information Prints Popular Posts Dictionary English dictionary online


VALIDATE - command to checks whether the database is ready for a role transition or not.

DGMGRL> VALIDATE DATABASE ...;    -- From Oracle Database 12c


$BDUMP/drc*.log
$ORACLE_HOME/rdbms/log/drc*.log

alter system set dg_broker_start=false;
alter system set dg_broker_start=false sid='*';
alter system set dg_broker_start=FALSE SCOPE=spfile SID='*';
alter system set dg_broker_start=true;
alter system set dg_broker_start=true sid='*';
alter system set dg_broker_start=TRUE SCOPE=spfile SID='*';

alter system set dg_broker_config_file1='/u01/dg_broker_config_files/dr1TESTP.dat' sid='*';
alter system set dg_broker_config_file2='/u01/dg_broker_config_files/dr2TESTP.dat' sid='*';

RAC PATCHING



HOW TO APPLY ORACLE PATCH (OPATCH) WITHOUT DOWNTIME.
====================================================

This is commonly asked question in interviews.

To apply Opatch
coditions are db and listener both must be down as opatch will update your current ORACLE_HOME with patches.
in single instance its not possible.
but for RAC instance its possible.
as in RAC there will be two seperate oracle home and two seperate instances running once instance on each oracle_home

use this command:

opatch napply -skip_subset -skip_duplicate -local -oh $ORACLE_HOME

when using -local parameter and -oh $ORACLE_HOME this means this patch session will only apply patch to current sourced ORACLE_HOME.


steps before applying patch:
----------------------------

1) check the database status.
wch_db.sql
-----------
select name,open_mode,database_name,created,log_mode,platform_name from v$database;


2) Check the object's invalid.

user_inv.sql
============
SELECT owner,COUNT(*) FROM dba_objects WHERE status = 'INVALID' GROUP BY owner;

count_inv.sql
-------------
select count(*) from dba_objects WHERE status ='INVALID';
3) Take backup of invalid's

create table bk_inv_ as select * from dba_objects  where status='INVALID';

4) check opatch version using 
opatch -v
if opatch version is not compatible check the readme file and
download the latest version and uncompress
in $ORACLE_HOME.

5) check oraInst.loc file pointing to your current $ORACLE_HOME or not.
cat /etc/oraInst.loc

inventory_loc=/u01/app/oracle/10.2.0/GlobalOraInv
inst_group=dba

if your server have more then one $ORACLE_HOME then comment the other $ORACLE_HOME and
uncomment the current $ORACLE_HOME

inventory must point to the current $ORACLE_HOME which is getting patched.

6) check free space on $ORACLE_HOME 
df -h $ORACLE_HOME

7) chek the utilities like 
which ld
which ar
which make
etc as per readme file.

8) unzip the patch 
unzip -d /loc_2_unzip p.zip

9) Go the patch directory
cd /loc_2_unzip/patch_number

10) Bring down the listner.
cd $ORACLE_HOME/bin
lsnrctl stop 

11) Bring down the database
Shutdown immediate.

12) export opatch 
export PATH=$PATH:$HOME:$ORACLE_HOME/OPatch:/bin

13) Start the patch
opatch napply -skip_subset -skip_duplicate

for RAC database then database can be up
as it may be having more then one instance
so you can bring down one instance and listener and apply the patch and open it
and then do the same on another node.
like this db will be up and no user will face issue in outage also.

to apply opatch in RAC instance
opatch napply -skip_subset -skip_duplicate -local -oh $ORACLE_HOME

when using -local parameter and
-oh $ORACLE_HOME this means this patch session will only apply patch to current ORACLE_HOME only.

--------------------------------------------------------
. All-Node Patch

. Shutdown all Oracle instances on all nodes
. Apply the patch to all nodes
. Bring all nodes up

. Minimum downtime

. Shutdown the Oracle instance on node 1
. Apply the patch to the Oracle instance on node 1
. Shutdown the Oracle instance on node 2
. Apply the patch to the Oracle instance on node 2
. Shutdown the Oracle instance on node 3
. At this point, instances on nodes 1 and 2 can be brought up
. Apply the patch to the Oracle instance on node 3
. Startup the Oracle instance on node 3

. (no downtime) 

. Shutdown the Oracle instance on node 1
. Apply the patch to the Oracle instance on node 1
. Start the Oracle instance on node 1
. Shutdown the Oracle instance on node 2
. Apply the patch to the Oracle instance on node 2
. Start the Oracle instance on node 2
. Shutdown the Oracle instance on node 3
. Apply the patch to the Oracle instance on node 3
. Start the Oracle instance on node 3
-------------------------------------------------------------

14) Once patch installation is completed need to do post patching steps.

a) starup the instance
startup
b) Loading modified sqlfiles into the database.
@$ORACLE_HOME/rdbms/admin/catbundle.sql cpu apply

to check the logs generated
catbundle_CPU__APPLY_.log
catbundle_CPU__GENERATE_.log

c) Recompiling Views in the Database
shutdown immediate
startup upgrade
@$ORACLE_HOME/cpu/view_recompile/view_recompile_jan2008cpu.sql
shutdown immediate
startup

If it is a RAC instance.
shutdown
startup nomount
alter database set cluster database=false scope=spfile;
shutdown
startup upgrade
@?/cpu/view_recompile/view_recompile_jan2008cpu.sql
shutdown
startup
alter database set cluster database=true scope=spfile;

restart the database.
cd $CRS_HOME/bin
srvctl start database -d 
15) If any invalid objects were reported, run the utlrp.sql script as follows



user_inv.sql
============
SELECT owner,
COUNT(*)
FROM dba_objects
WHERE status = 'INVALID'
GROUP BY owner;

count_inv.sql
-------------
select count(*)
from dba_objects
WHERE status ='INVALID';

if any new invalids seen then again take backup of invalid objects and compile it.

create table bk_inv_ as select * from dba_objects
where status='INVALID';

@?/rdbms/admin/utlrp.sql --- to compile the invalid objects.


16) Confirm that patch has been applied successfully or not at db level also.


post_patch.sql
--------------
col action_time for a40
col action for a15
col namespace for a15
col version for a15
col comments for a40
set pages 1000
set lines 170
select * from registry$history ;

Monday, February 1, 2016

ZUBIN NODE DELETION STEPS

         root@dbf0t00lpaora:$ su - grid
grid@dbf0t00lpaora:$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/grid

grid@dbf0t00lpaora:$ olsnodes
dbf0t00lpaora
dbf0t01lpbora

grid@dbf0t00lpaora:$ crsctl get cluster mode status
Cluster is running in "standard" mode

grid@dbf0t00lpaora:$ srvctl config gns
PRKF-1110 : Neither GNS server nor GNS client is configured on this cluster

grid@dbf0t00lpaora:$  oifcfg getif
bond0  172.27.110.0  global  public
bond1  172.27.117.0  global  cluster_interconnect

grid@dbf0t00lpaora:$ crsctl get node role config
Node 'dbf0t00lpaora' configured role is 'hub'

grid@dbf0t00lpaora:$ asmcmd showclustermode
ASM cluster : Flex mode disabled

grid@dbf0t00lpaora:$ asmcmd showclusterstate
Normal


grid@dbf0t00lpaora:$ srvctl status asm -detail
ASM is running on dbf0t00lpaora,dbf0t01lpbora
ASM is enabled.

grid@dbf0t00lpaora:$ crsctl get node role config -all
Node 'dbf0t00lpaora' configured role is 'hub'
Node 'dbf0t01lpbora' configured role is 'hub'

grid@dbf0t00lpaora:$ crsctl get node role status -all
Node 'dbf0t00lpaora' active role is 'hub'
Node 'dbf0t01lpbora' active role is 'hub

grid@dbf0t00lpaora:$  crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
               ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.FRA.dg
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
               ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
               ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.OCR_VD.dg
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
               ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.asm
               ONLINE  ONLINE       dbf0t00lpaora            Started,STABLE
               ONLINE  ONLINE       dbf0t01lpbora            Started,STABLE
ora.net1.network
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
               ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.ons
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
               ONLINE  ONLINE       dbf0t01lpbora            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       dbf0t00lpaora            169.254.28.162 172.2
                                                             7.117.125,STABLE
ora.cvu
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.dbf0t00lpaora.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.dbf0t01lpbora.vip
      1        ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.m.db
      1        ONLINE  ONLINE       dbf0t00lpaora            Open,STABLE
      2        ONLINE  ONLINE       dbf0t01lpbora            Open,STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       dbf0t00lpaora            Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       dbf0t01lpbora            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.test.db
      1        ONLINE  ONLINE       dbf0t00lpaora            Open,STABLE
      2        ONLINE  ONLINE       dbf0t01lpbora            Open,STABLE



root@dbf0t00lpaora:$  /u01/app/12.1.0/grid/bin/olsnodes -s
dbf0t00lpaora   Active
dbf0t01lpbora   Active

root@dbf0t00lpaora:$  /u01/app/12.1.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   10e7c885235c4f64bf28ba4117dbed61 (ORCL:OCR_VD) [OCR_VD]
Located 1 voting disk(s).

root@dbf0t00lpaora:$  /u01/app/12.1.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1608
         Available space (kbytes) :     407960
         ID                       :  793966683
         Device/File Name         :    +OCR_VD
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

root@dbf0t00lpaora:$  /u01/app/12.1.0/grid/bin/srvctl status database -d TEST
Instance TEST1 is running on node dbf0t00lpaora
Instance TEST2 is running on node dbf0t01lpbora

root@dbf0t00lpaora:$  /u01/app/12.1.0/grid/bin/srvctl config service -d TEST
root@dbf0t00lpaora:$  /u01/app/12.1.0/grid/bin/srvctl status service -d TEST


2.  A Removing an oracle database instance:

root@dbf0t00lpaora:$ su - oracle
oracle@dbf0t00lpaora:$ /u02/app/oracle/product/12.1.0/dbhome_1/bin/dbca -silent -deleteInstance -nodeList dbf0t01lpbora -gdbName TEST -instanceName TEST2 -sysDBAUserName sys -sysDBAPassword abc123
Deleting instance
1% complete
2% complete
6% complete
13% complete
20% complete
26% complete
33% complete
40% complete
46% complete
53% complete
60% complete
66% complete
Completing instance management.
100% complete
Look at the log file "/u02/app/oracle/cfgtoollogs/dbca/TEST.log" for further details.


oracle@dbf0t00lpaora:$ srvctl status database -d TEST
Instance TEST1 is running on node dbf0t00lpaora

oracle@dbf0t00lpaora:$  srvctl config database -d TEST -v
Database unique name: TEST
Database name: TEST
Oracle home: /u02/app/oracle/product/12.1.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/TEST/PARAMETERFILE/spfile.269.900859867
Password file: +DATA/TEST/PASSWORD/pwdtest.256.900859377
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: dba
Database instances: TEST1
Configured nodes: dbf0t00lpaora
Database is administrator managed

oracle@dbf0t00lpaora:$ sqlplus '/as sysdba'

SQL*Plus: Release 12.1.0.2.0 Production on Thu Jan 14 10:35:23 2016

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select inst_id, instance_name, status, to_char(startup_time,'DD-MON-YYYY HH24:MI:SS') as "START_TIME" from gv$instance order by inst_id;

   INST_ID INSTANCE_NAME    STATUS       START_TIME
---------- ---------------- ------------ --------------------
         1 TEST1            OPEN         14-JAN-2016 10:05:35

Check if the redo log thread and UNDO tablespace for the deleted instance is removed (which for my example, they were successfully removed). If not, manually remove them.

SQL> select thread# from v$thread where instance='TEST';

no rows selected

SQL> select thread# from v$thread where upper(instance) = upper('TEST');
no rows selected
SQL> select group# from v$log where thread# =2;

no rows selected

SQL> select member from v$logfile ;

MEMBER
--------------------------------------------------------------------------------
+DATA/TEST/ONLINELOG/group_2.263.900859605
+DATA/TEST/ONLINELOG/group_1.262.900859603

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

oracle@dbf0t00lpaora:$ srvctl config listener -a
Name: LISTENER
Type: Database Listener
Network: 1, Owner: grid
Home: <CRS home>
  /u01/app/12.1.0/grid on node(s) dbf0t01lpbora,dbf0t00lpaora
End points: TCP:1521
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:

oracle@dbf0t00lpaora:$ ssh dbf0t01lpbora
dbf0t01lpbora:~ # exit
logout
Connection to dbf0t01lpbora closed.

If you find any redo log and undo references of the deleted instance in the cluster, use the following commands to remove those references.

alter database disable thread 2;
alter database drop logfile group 3;
alter database drop logfile group 4;
drop tablespace undotbs2 including contents and datafiles;
alter system reset undo_tablespace scope=spfile sid = 'TEST2';
alter system reset instance_number scope=spfile sid = 'TEST2';

2. B Removing RDBMS software:
On the node which is to be deleted from the cluster, run the following command...

root@dbf0t01lpbora:$ su – oracle
oracle@dbf0t01lpbora:$


oracle@dbf0t01lpbora:$ echo $ORACLE_HOME
/u02/app/oracle/product/12.1.0/dbhome_1

oracle@dbf0t01lpbora:$  cd $ORACLE_HOME/oui/bin
oracle@dbf0t01lpbora:$ pwd
/u02/app/oracle/product/12.1.0/dbhome_1/oui/bin

oracle@dbf0t01lpbora:$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={dbf0t01lpbora}" -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16079 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

Now run the following command on node 2 , to deinstall oracle home from this node.

oracle@dbf0t01lpbora:$ cd $ORACLE_HOME/deinstall
oracle@dbf0t01lpbora:$ pwd
/u02/app/oracle/product/12.1.0/dbhome_1/deinstall

oracle@dbf0t01lpbora:$ ./deinstall –local

On any cluster node that remains in the cluster , run the following command ....

oracle@dbf0t00lpaora:$  cd $ORACLE_HOME/oui/bin
oracle@dbf0t00lpaora:$ pwd
/u02/app/oracle/product/12.1.0/dbhome_1/oui/bin

oracle@dbf0t00lpaora:$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={dbf0t00lpaora}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16070 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

Now verify the inventory and make sure that the database on node2 is completely removed.

oracle@dbf0t00lpaora:$ cd /u01/app/oraInventory/ContentsXML/
oracle@dbf0t00lpaora:$ ls
comps.xml  inventory.xml  libs.xml
oracle@dbf0t00lpaora:$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0/grid" TYPE="O" IDX="4" CRS="true">
   <NODE_LIST>
      <NODE NAME="dbf0t00lpaora"/>
      <NODE NAME="dbf0t01lpbora"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u02/app/oracle/product/12.1.0/dbhome_1" TYPE="O" IDX="5">
   <NODE_LIST>
      <NODE NAME="dbf0t00lpaora"/>
   </NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome2" LOC="/u01/app/11.2.0.4/grid" TYPE="O" IDX="2" REMOVED="T"/>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" REMOVED="T"/>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0.4/dbhome_1" TYPE="O" IDX="3" REMOVED="T"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

2. C Removing Node from Cluster:
Run the following command as root to determine whether the node you want to delete is active and whether it is pinned.

root@dbf0t01lpbora:$ export ORACLE_HOME=/u02/app/oracle/product/12.1.0/dbhome_1
root@dbf0t01lpbora:$ export GRID_HOME=/u01/app/12.1.0/grid

root@dbf0t01lpbora:$ $GRID_HOME/bin/olsnodes -s -t
dbf0t00lpaora   Active  Unpinned
dbf0t01lpbora   Active  Unpinned

Disable the Oracle Clusterware applications and daemons running on the node to be deleted from the cluster. Run the rootcrs.pl script as root from the Grid_home/crs/install directory on the node to be deleted.

root@dbf0t01lpbora:$ ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
Network 1 exists
Subnet IPv4: 172.27.110.0/255.255.255.0/bond0, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node dbf0t00lpaora
VIP Name: dbf0t00lpaora-vip.ch.cadhlt.org
VIP IPv4 Address: 172.27.110.95
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node dbf0t01lpbora
VIP Name: dbf0t01lpbora-vip.ch.cadhlt.org
VIP IPv4 Address: 172.27.110.97
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.crsd' on 'dbf0t01lpbora'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.OCR_VD.dg' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.OCR_VD.dg' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.DATA.dg' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.asm' on 'dbf0t01lpbora' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dbf0t01lpbora' has completed
CRS-2677: Stop of 'ora.crsd' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.evmd' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.storage' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.crf' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dbf0t01lpbora'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.storage' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.crf' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'dbf0t01lpbora' succeeded
CRS-2677: Stop of 'ora.asm' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.cssd' on 'dbf0t01lpbora' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dbf0t01lpbora'
CRS-2677: Stop of 'ora.gipcd' on 'dbf0t01lpbora' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbf0t01lpbora' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2016/01/14 11:07:08 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2016/01/14 11:07:31 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

From a node that is to remain a member of the Oracle RAC, run the following command from the Grid_home/bin directory as root to update the Clusterware configuration to delete the node from the cluster.

root@dbf0t00lpaora:$ $GRID_HOME/bin/crsctl delete node -n dbf0t01lpbora
CRS-4661: Node dbf0t01lpbora successfully deleted.
root@dbf0t00lpaora:$ $GRID_HOME/bin/olsnodes -s -t
dbf0t00lpaora   Active  Unpinned

As the Grid Infrastructure owner, execute runInstaller from Grid_home/oui/bin on the node being removed to update the inventory.
grid@dbf0t01lpbora:$ cd /u01/app/12.1.0/grid/oui/bin/
grid@dbf0t01lpbora:$ pwd
/u01/app/12.1.0/grid/oui/bin
root@dbf0t01lpbora:$ su - grid
dbf0t01lpbora:~ # PS1="\u@\h:$ "
grid@dbf0t01lpbora:$

grid@dbf0t01lpbora:$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.1.0/grid "CLUSTER_NODES={dbf0t01lpbora}" CRS=TRUE -silent -local
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16079 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

Run deinstall as the Grid Infrastructure software owner from the node to be removed in order to delete the Oracle Grid Infrastructure software.

Please pay extra care while responding to the prompts. When supplying the values to listener, give only local listener value and don't specify scan_listener for deletion.


grid@dbf0t01lpbora:$ cd /u01/app/12.1.0/grid/deinstall/
grid@dbf0t01lpbora:$ ./deinstall -local
Checking for required files and bootstrapping ...
Please wait ...

On Node1

grid@dbf0t00lpaora:$ cd /u01/app/12.1.0/grid/oui/bin
grid@dbf0t00lpaora:$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.1.0/grid "CLUSTER_NODES={dbf0t00lpaora}" CRS=TRUE -silent
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16070 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.


2. D Verification:

grid@dbf0t00lpaora:$ cd /u01/app/oraInventory/ContentsXML/
grid@dbf0t00lpaora:$ ls
comps.xml  inventory.xml  libs.xml
grid@dbf0t00lpaora:$ cat inventory.xml
<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2014, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<INVENTORY>
<VERSION_INFO>
   <SAVED_WITH>12.1.0.2.0</SAVED_WITH>
   <MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME="OraGI12Home1" LOC="/u01/app/12.1.0/grid" TYPE="O" IDX="4" CRS="true">
   <NODE_LIST>
      <NODE NAME="dbf0t00lpaora"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraDB12Home1" LOC="/u02/app/oracle/product/12.1.0/dbhome_1" TYPE="O" IDX="5">
   <NODE_LIST>
      <NODE NAME="dbf0t00lpaora"/>
   </NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome2" LOC="/u01/app/11.2.0.4/grid" TYPE="O" IDX="2" REMOVED="T"/>
<HOME NAME="Ora11g_gridinfrahome1" LOC="/u01/app/11.2.0/grid" TYPE="O" IDX="1" REMOVED="T"/>
<HOME NAME="OraDb11g_home1" LOC="/u01/app/oracle/product/11.2.0.4/dbhome_1" TYPE="O" IDX="3" REMOVED="T"/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

grid@dbf0t00lpaora:$ /u01/app/12.1.0/grid/bin/cluvfy stage -post nodedel -n dbf0t01lpbora -verbose

Performing post-checks for node removal

Checking CRS integrity...
The Oracle Clusterware is healthy on node "dbf0t00lpaora"

CRS integrity check passed

Clusterware version consistency passed.
Result:
Node removal check passed

Post-check for node removal was successful.

grid@dbf0t00lpaora:$ olsnodes -s -t
dbf0t00lpaora   Active  Unpinned
grid@dbf0t00lpaora:$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.FRA.dg
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.OCR_VD.dg
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.asm
               ONLINE  ONLINE       dbf0t00lpaora            Started,STABLE
ora.net1.network
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.ons
               ONLINE  ONLINE       dbf0t00lpaora            STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       dbf0t00lpaora            169.254.28.162 172.2
                                                             7.117.125,STABLE
ora.cvu
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.dbf0t00lpaora.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.m.db
      1        ONLINE  ONLINE       dbf0t00lpaora            Open,STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       dbf0t00lpaora            Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       dbf0t00lpaora            STABLE
ora.test.db
      1        ONLINE  ONLINE       dbf0t00lpaora            Open,STABLE
grid@dbf0t00lpaora:$ crsctl status res -t | grep -i dbf0t01lpbora

2. E Removing remaining components:
Remove asmlib if you are using asmlib for ASM storage
Revmoce udev rules, if you are using udev rules for ASM storage
Remove oracle and grid users and also corresponding groups.
 groups