Oracle Cluster Registry (OCR) in RAC

Oracle Cluster Registry (OCR) is the critical component in Oracle RAC.

Ø  OCR records cluster configuration information.  If it fails, the entire clustered environment of Oracle RAC is affected and a possible outage is a result.

Ø  It is the central repository for CRS, which stores its metadata, configuration and state information for all clusters defined in the clusterware.

Ø  It is the cluster registry maintains application resources and their availability within the RAC environment.

Ø  It also stores information of CRS daemons and cluster managed applications.

What is stored in OCR

We have the introduction of OCR, now we will see what are stored in OCR file.

Ø  Node membership information, i.e, which nodes are part of the cluster.

Ø  Software version

Ø  Location of 11g voting disk

Ø  Server pools

Ø  Status of cluster resources such as RAC databases, listeners, instances and services.

·         Server up/down

·         Network up/down

·         Database up/down

·         Instance up/down

·         Listener up/down

Ø  Configuration of the cluster resources like RAC databases, listeners, instances and services.

·         Dependencies,

·         Management Policies (automatic/manual)

·         Callout scripts

·         Retries

Ø  Cluster database instance to node mapping

Ø  ASM instance, disk groups, etc

Ø  CRS application resource profiles such as VIP address, service, etc.

Ø  Database services characteristics eg., preferred/available nodes, TAF policy, load balancing goal, etc

Ø  Information about clusterware processes

Ø  Information about interaction and management of 3^rd party applications controlled by CRS

Ø  Communication settings where the clusterware daemons or background process listen.

Ø  Information about OCR backups.

Let us see the contents in OCR file.

[root@rac1 ~]# ocrconfig -manualbackup

rac2     2014/01/18 01:03:40     /u01/app/grid/product/11.2.0.3/grid_1/cdata/cluster01/backup_20140118_010340.ocr

[root@rac2 ~]#  strings /u01/app/grid/product/11.2.0.3/grid_1/cdata/cluster01/backup_20140118_010340.ocr| grep -v type |grep ora!

ora!LISTENER!lsnr

ora!host02!vip

rora!rac1!vip

;ora!oc4j

6ora!LISTENER_SCAN3!lsnr

ora!LISTENER_SCAN2!lsnr

ora!LISTENER_SCAN1!lsnr

ora!scan3!vip

ora!scan2!vip

ora!scan1!vip

ora!gns

ora!gns!vip

ora!registry!acfs

ora!DATA!dg

dora!asm

_ora!eons

ora!ons

ora!gsd

ora!net1!network

Who is updating OCR

It is updated and maintained by many client applications.

Ø  CSSd during startup of cluster – to update the status of the servers.

Ø  CSSd during node addition/deletion – to add/delete nodes

Ø  CRSd about status of nodes during failure/reconfiguration

Ø  OUI  - Oracle universal Installer during installation/upgradation/deletion/addition

Ø  Srvctl – ( to manage clusters and rac databases)

Ø  Cluster control utility – crsctl (to manage cluster /local resources)

Ø  Enterprise Manager (EM)

Ø  Database Configuration Assistant (DBCA)

Ø  Database upgrade Assistant (DBUA)

Ø  Network Configuration Assistant (NETCA)

Ø  ASM configuration assistant (ASMCA)

How the update is performed in OCR.

1.       Each node in the cluster will have the copy of OCR in memory for better performance and each node is responsible for updating the OCR if required.

2.       CRSd process is responsible for reading and writing to the OCR files as well as refreshing the local OCR cache and caches on the other nodes in the cluster.

3.       Oracle uses distributed shared cache architecture during cluster management to optimize queries against the cluster repository and at the same time, each node maintains a copy of the OCR in memory.

4.       Oracle clusterware uses the background process to access the OCR cache.

5.       Only one CRSd process (designated as master) in the cluster performs any read/write activity. If any new information read by the CRSd master process, then it refresh the local OCR cache and the OCR cache on the others nodes of the cluster.

6.       As the OCR cache is distributed across all nodes in the cluster, OCR clients like srvctl,crsctl,etc will communicate directly with the OCR process on the node to get required information.

7.       Clients communicate via the local CRSd process for any updates on the physical OCR binary file.

During the above process, OCRCONFIG command cannot modify the OCR configuration information for nodes that are shut down or for nodes on which oracle clusterware is not running.  So, we need to avoid shutting down the nodes while modifying the OCR using the ocrconfig command.  We need to perform a repair on the stopped node before it can bring online to join the cluster.

OCRCONFIG –repair command change the OCR configuration only on the node from which we are executing this command.

 Example :  if the OCR mirror was relocated to a disk in rac2 which is down from /dev/raw/raw2  in rac1, then use ocrconfig –repair ocrmirror /dev/raw/raw2 command on rac2 while the CRS stack is down on that node to repair its OCR configuration.

Purpose of OCR

Ø  Oracle clusterware reads the ocr.loc for the location of the registry and to know which application resources needs to be started and the nodes on which to start them.

Ø  Used for bootstrap the CSS with port info, nodes in the cluster and  other similar informations.

Ø  CRSd and other clusterware daemons function is to define and manage resources managed by clusterware. Resources have profiles that define metadata about them. This metadata is stored in OCR. The CRS reads the OCR and manages the application resources, starts, stops, and monitor and manages their failover.

Ø  Maintains and tracks the information pertaining to the definition, availability and current state of the services.

Ø  Implements the workload balancing and continuous availability features of services.

Ø  Generates event during state changes.

Ø  Maintain configuration profiles of resources in OCR.

Ø  Records the currently known state of the cluster at regular intervals and provides the same when reqeuested by client application like srvctl,crsctl,etc.

How the information is stored in OCR

Ø  OCR uses a file-based repository to store configuration information in a series of key-value pairs, using a directory tree-like structure.

Ø  It contains information pertaining to all tiers in the clustered database.

Ø  Various parameters are stored as name-value pairs used and maintained at different levels of the architecture.

Each tier is managed and administered by daemon process at different levels with appropriate privileges.

Eg. All system level resources or application definitions would require root, or superuser to start or stop.

Those defined at the database level will require dba privileges.

Where and how should OCR be stored ?

Ø  Location of the OCR is found in a file on each individual node of the cluster. It varies depending on the flavors of unix. In Linux , /etc/oracle/ocr.loc

Ø  OCR must reside on a shared disk that is accessible by all nodes in the cluster. It cannot be stored on raw filesystems as in 10g as it is deprecated.  So, we are left with cluster filesystem (CFS) and ASM. But CFS will cost high and may not be an option. So, better to go for ASM.  OCR and voting disks can be exclusively available on any disksgroups .

Ø  OCR is striped and mirrored (if we have redundancy in the external) similar to other database files.

Ø  OCR is replicated across all the underlying disks of the diskgroup, so failure of the disk does not bring the failure of the diskgroup.

Ø  In 11gR2, we can have upto 5 OCR copies.

Ø  Since it is in shared location, it can be administered from any nodes irrespective of the node on which the registry was created.

Ø  A small disk of around 300MB-500MB is a good choice.

Utilities to manage OCR

Add an OCR file

—————-

Add an OCR file to an ASM diskgroup called +DATA

ocrconfig –add  +DATA

Moving the OCR

————–

Move an existing OCR file to another location :

ocrconfig –replace /u01/app/oracle/ocr –replacement +DATA

Removing an OCR location

————————

- requires that at least one other OCR file must remain online.

ocrconfig –delete +DATA

OCR Backups

ü  Oracle Clusterware 11g Release 2 backs up the OCR automatically every four hours on a schedule that is dependent on when the node started (not clock time).

ü  OCR backups are made available in $GRID_HOME/cdata/<cluster name> directory on the node performing the backups.

ü  One node known as the master node is dedicated to these backups, but in case master node is down, some other node may become the master. Hence, backups could be spread across nodes due to outages.

These backups are named as follows:

-4-hour backups   (3 max) –backup00.ocr, backup01.ocr, and backup02.ocr.

-Daily backups     (2 max) – day.ocr and day_.ocr

-Weekly backups (2 max) – week.ocr and week_.ocr

ü  It is recommended that OCR backups may be placed on a shared location which can be configured using ocrconfig -backuploc <new location> command.

ü  Oracle Clusterware maintains the last three backups, overwriting the older backups. Thus, you will have 3 4-hour backups, the current one, one four hours old and one eight hours old.

ü  Therefore no additional clean-up tasks are required of the DBA.

ü  Oracle Clusterware will also take a backup at the end of the day. The last two of these backups are retained.

ü  Finally, at the end of each week Oracle will perform another backup, and again the last two of these backups are retained. You should make sure that your routine file system backups backup the OCR location.

ü  Note that RMAN does not backup the OCR.

You can use the ocrconfig command to view the current OCR backups:

#ocrconfig –showbackup auto                                           

Voting Disks in RAC

Voting disks are important component of Oracle Clusterware. Clusterware uses voting disk to determine which nodes are members of the cluster.  After ASM is introduced to store these files, these are called as VOTING FILES.
Primary function of voting disks is to manage node membership and prevent SPLITBRAIN Syndrome in which 2 or more instances attempt to control the RAC database. These files can be stored either in ASM or on shared storage.

•   If it is stored in ASM, no need to configure manually as the files will be created depending on the redundancy in ASM.

•   In shared storage system, we need to manually configure these files with redundancy setup for high availability.

•   We must have odd number of disks.

•  Oracle recommends minimum of 3 and maximum of 5. In 10g, Clusterware can supports 32 voting disks but in 11gR2 supports 15 voting disks.

•  A node must be able to access more than half of the voting disks at any time.  For eg, if you have 5 voting disks, a node must be able access atleast 3 of the voting disks. If it cannot access the minimum of voting disks, then it is evicted/removed from the cluster.

•  All nodes in the RAC cluster register their heartbeat information in the voting disks/files.  RAC heartbeat is the polling mechanism that is sent over the cluster interconnect to ensure all RAC nodes are available.

How Voting Happens

The CKPT process updates the control file every 3 seconds in an operation known as heartbeat. CKPT writes to a single block that is local to the node/each instance and intra instance coordination is not required.  This block is called checkpoint progress record.

All members of the cluster attempt to lock on the controlfile record for updating. The instance which obtains the locks tallies the votes from all members.  Then, the group membership must conform to the decided(voted) membership before allowing GCS/GES to proceed for reconfiguration. The control file record is then stored in the same block as the heartbeat in the controlfile checkpoint progress record.


What is NETWORK and DISK HEARTBEAT and how it registers in VOTING DISKS/FILES

            1.       All nodes in the RAC cluster register their heartbeat information in the voting disks/files.  RAC heartbeat is the polling mechanism that is sent over the cluster interconnect to ensure all RAC
a.       nodes are available.
b.      Voting disks/files are just like attendance register where you have nodes mark their attendance (heartbeats).

            2.       CSSD process on every node makes entries in the voting disk to ascertain the membership of the node.  While marking their own presence, all the nodes also register the information about their communicability with other nodes in the voting disk.  This is called NETWORK HEARTBEAT.

            3.       CSSD  process in each RAC maintains the heart beat in a block of size  1 OS block in the hot block of voting disk at a specific offset. The written block has a header area with the node name.  The heartbeat counter increments every second on every write call.  Thus heartbeat of various nodes is recorded at different offsets in the voting disk. This process is called DISK HEARTBEAT.

            4.       In addition of maintaining its own disk block, CSSD processes also monitors the disk block maintained by the CSSD processes of other nodes in cluster. Healthy nodes will have continuous network & disk heartbeats exchanged between the nodes.  Break in heartbeats indicates a possible error scenario.

            5.       If the disk is not updated in a short timeout period, the node is considered unhealthy and may be rebooted to protect the database.  In this case, a message to this effect is written in the KILL BLOCK of node. Each nodes reads its KILL BLOCK once per second, if the kill block is not overwritten, node commits suicide.

            6.       During reconfig (leaving or joining), CSSD monitors all nodes heartbeat information and determines whether the nodes has a disk heartbeat including those with no network heartbeat.  If no disk heartbeat is detected, then node is considered as dead.



What Information is stored in VOTING DISK/FILE
It contains 2 types of data .

Static data : Info about the nodes in cluster
Dynamic data: Disk heartbeat logging

It contains the important details of the cluster nodes membership like
a.       Which node is part of the cluster.
b.      Which node is leaving the cluster and
c.       Which node is joining the cluster.

 Purpose of Voting disk or Why is Voting disk needed

Voting disks are used by clusterware for health check.

Ø  Used by CSS to determine which nodes are currently members of the cluster.
Ø  In concert with other cluster components like CRS to shutdown, fence or reboot either single or multiple nodes whenever network communication is lost between any node within the cluster, to prevent to split-brain condition in which 2 or more instances attempt  to control the RAC database and thus protecting the database.
Ø  Will be used by CSS to arbitrate (to take an authorized decision) with peers that it is not able to see over the private interconnect in the event of an outage, allowing it to salvage (rescue from loss) the largest fully  connected sub-cluster for further operation.  During this operation , node membership (NM) will make an entry in the voting disk to inform its vote on availability. Other instances in the cluster too do similar actions.  The 3 voting disks configured also provide a method to determine who in the cluster should survive.
Example : if eviction of one of the node is necessiated due to unresponsive action, then the node that has 2 voting disks with start evicting the other node. NM alternates it action between the heartbeat and the voting disk to determine the availability of the other nodes in cluster.

Possible scenarios in Voting disks

As we know now that voting disks is used by CSSD. It contains both network & disk heartbeat from all nodes and if any break in heartbeat will result in eviction of the node from cluster. There are possible scenarios with missing heartbeats.

1.       Network heart beat is successful, but disk heart beat is missed.
2.       Disk heart beat is successful, but network heart beat is missed.
3.       Both heart beats failing.

When a cluster is involved with many nodes, then few more scenarios are possible.

1.       Nodes have a split into N sets of nodes., communicating within the sets, but not with the members in other set.
2.       Just one node going unhealthy.  Nodes with  quorum (minimum number of nodes to make cluster valid) will maintain active membership of the cluster and other node(s) will be fenced/rebooted.

Why should we have ODD number of voting diks ??

A node must be able to access more than half of the voting disks at any time.
Example.

a.       Let us consider 2 node cluster with even number of voting disks say 2.
b.      Let node 1 is able to access Voting disk 1.
c.       Node 2 is able to access voting disk 2.
d.      From the above steps, we see that we don’t any common file where clusterware can check the heartbeat of both the nodes.
e.      If we have 3 voting disks and both the nodes are able to access more than half ie., 2 voting disks, there will be atleast one disk which will be accessed by both the nodes. The clusterware can use this disk to check the heartbeat of the nodes.
f.        A node not able to do so will be evicted from the cluster by another node that has more than half the voting disks to maintain the integrity of the cluster.

Where voting disks are stored

It can be stored in
a.       Raw devices
b.      Cluster file system supported by Oracle RAC such as OCFS,Sun cluster or Veritas Cluster Filesystem
c.       ASM disks (in 11gR2).

When voting disk is stored in ASM, a question is arised how the voting file on ASM can be accessed when we want to add a new node to a cluster.

The answer is.
Oracle ASM reserves several blocks at the fixed location for every Oracle ASM disk used for storing the voting files. As a result, Oracle clusterware can access the voting disks present in ASM even if the ASM instance is down and CSS can continue to maintain the Oracle cluster even if the ASM has failed.  The physical location of the voting files in ASM disks are fixed i.e., the cluster stack does not rely on a running ASM instance to access the files.

d.      If the ASM is stored in ASM, the multiplexing of voting disk is decided by the redundancy of the diskgroup.

Redundancy
of the diskgrou  #of copies of
voting disk
 ( Minimum # of disks
 in the diskgroup)
External
1
1
Normal
3
3
High
5
5




Commands to check the Votingdisk

Crsctl query css votedisk    - for checking the file location

When to take voting disk backup
1.       Fresh installation
2.       Adding /deleting node


Voting disk backup  (In 10g)

dd if=<voting-disk-path> of=<backup/path>

Voting disk restore (In 10g)

dd  if=<backup/path>  of=<voting disk path>


In 11gR2, the voting files are backed up automatically as part of OCR.  Oracle recommends NOT used dd command to backup or restore as this can lead to loss of the voting disk.

Add/delete vote disk

crsctl add css votedisk <path> -adds a new voting disk
crsctl delete css votedisk <path> -- deletes the voting disk

Split Brain Syndrome in RAC

In a Oracle RAC environment all the instances/servers communicate with each other using high-speed interconnects on the private network. This private network interface or interconnect are redundant and are only used for inter-instance oracle data block transfers. Now talking about split-brain concept with respect to oracle rac systems, it occurs when the instance members in a RAC fail to ping/connect to each other via this private interconnect, but the servers are all pysically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. So basically due to lack of commincation the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instance running, the sane block might get read, updated in these individual instances and there would be data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented check for the split brain syndrome. 

What does RAC do incase node becomes inactive:

In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node which first detects that one of the node is not accessible, it will evict that node from the RAC group. e.g. there are 4 nodes in a rac instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning.
The split brain concepts can become more complicated in large RAC setups. For example there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10 node RAC cluster ( one group of 4 nodes and other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking controlfile, then the node that lock the controlfile will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that if thats how it work by experience, but this is the theory behind it.
When we see that the node is evicted, usually oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node.
You will see oracle error: ORA-29740, when there is a node eviction in RAC. There are many reasons for a node eviction like heart beat not received by the controlfile, unable to communicate with the clusterware etc.
A good metalink note on understanding node eviction and how to address is Note ID: 219361.1

The CSS (Cluster Synchronization Service) daemon in the clusterware maintains the heart beat to the voting disk.

Cache Fusion in RAC

Now before going into Cache Fusion – first let’s first memorize once again how actually a normal database instance behaves when there is a request for data block.

Let us suppose a user process is requesting a data block, and since this process cannot directly read from the disk, first the requested block must be read(Physical Read) into Buffer Cache of SGA. Once it is read into buffer cache it will remains in the Buffer Cache for further requests. Whenever there is a request for the same data block, since it is already in the buffer, it can be directly read (Buffer Read) from the buffer, thus avoiding another Physical Read. If a data block is found in buffer cache it is called a ‘Cache Hit’ and if it is not found, then it is called a ‘Cache Miss’.

In order to maintain data integrity, when there are concurrent requests for the same data block, Oracle uses Locking Mechanism and multi-version consistency control. A data block can reside in various buffers with different versions, for example a dirtied block, where the previous version of the data block will be maintained in UNDO and the copy of the current version in REDO. Whenever a user request for the block that was already in buffer and dirtied, the UNDO segment provides the required information to construct the read-consistent (CR) image of the data blocks. So, Multi-version data blocks help to achieve read consistency.

The read consistency model guarantees that the data block seen by a statement is consistent with respect to a single point in time and does not change during the statement execution. Readers of data do not wait for other writer’s data or for other readers of the same data. At the same time, writers do not wait for other readers for the same data. Only writers wait for other writers if they attempt to write.

In a single instance the following happens when reading a block

* When a reader reads a recently modified block, it might find an
active transaction in the block.
* The reader will need to read the undo segment header to decide
whether the transaction has been committed or not.
* If the transaction is not committed, the process creates a
consistent read (CR) version of the block in the buffer cache
using the data in the block and the data stored in the undo segment.
* If the undo segment shows the transaction is committed, the process
has to revisit the block and clean out the block and generate the redo
for the changes.
Now let us see how it goes in a RAC environment –

In RAC, there are 2 or more instances accessing same database files that are residing in shared storage area. Each instance has its own SGA and background processes, which means each instance has its own buffer cache (local to each instance). These buffer cache’s act individually at instance level and fuse together at database level to form a single entity (Global Cache) so as to share the data blocks between them. This is what we called ‘Cache Fusion’. Cache Fusion uses a high-speed IPC interconnect to provide cache-to-cache transfers of data blocks between instances in a cluster. This data block shipping eliminates the disk I/O and optimizes read/write concurrency.


Now the question is how the integrity of the data is maintained in a RAC environment, if there are concurrent requests for the same data block – Here too Oracle uses locking and queuing mechanisms to coordinate lock resources, data and inter-instance data requests.

Cache Fusion was implemented by a controlling mechanism called Global Cache Service (GCS), which is responsible for block transfers between instances. The Global Cache Service is implemented by various background processes, such as

Global Cache Service Processes (LMSn)
Global Enqueue Service Daemon (LMD)
[Before going into those processes, let us see how oracle treats the data blocks and how it manages them –

Oracle treats the data blocks as resources. Each of these resources can be held in different modes, which is important mechanism to maintain data integrity. These modes are classified into 3 types depending on whether resource holder intends to modify the data or read the data. They are –

Null (N) mode —Null mode is usually held as a placeholder.
Shared (S) mode — In this mode, data block is not modified by another session, but will allow concurrent shared access.
Exclusive (X) mode — This level grants the holding process exclusive access. Other processes cannot write to the resource. It may have consistent read blocks.

Furthermore, these resources act in one of 2 roles – Local (L) and Global (G).

A resource (data block) is assigned a local role, when a block is first read into the cache and no other instance request for the same data block.

A resource is assigned a Global role, when block is dirtied locally and transmitted to another instance.]

Now let us see what those daemon’s do –

Global Cache Service Daemon (LMSn)

Upon a request from an Instance GCS organizes the block shipping to other instances by retaining block copies in memory. Each such copy is called a past image (PI), which in the event of a node failure, Oracle can reconstruct the current version of a block by using a saved PI. It is also possible to have more than 1 PI of the data block; depending on how many times the block was requested in dirty stage.

Do not confuse read-consistent (CR) image with past image (PI), they are not same as they appear. PI is not a read consistent image of the data block, to make it so, you need to apply UNDO, which in turn converts into a CR image.

Keep in mind that if you want to read a data block, it must be in read consistent state. You are not allowed to read the changes made by others.

Global Enqueue Service Daemon (LMD)

The global enqueue service (GES) tracks the status of all Oracle enqueuing mechanisms. The GES performs concurrency control on dictionary cache locks, library cache locks, and transactions. It performs this operation for resources that are accessed by more than one instance. The GES controls access to data files and control files but not for the data blocks. GES processing includes the coordination for enqueues other than the data blocks. The resources managed by the GES include the following:

Transaction locks – It is acquired in the exclusive mode when a transaction initiates its first row level change. The lock is held until the transaction is committed or rolled back.
Library Cache locks – When a database object (such as a table, view, procedure, function, package, package body, trigger, index, cluster, or synonym) is referenced during parsing or compiling of a SQL, DML or DDL, PL/SQL, or Java statement, the process parsing or compiling the statement acquires the library cache lock in the correct mode.
Dictionary Cache Locks – Global enqueues are used in the cluster database mode. The data dictionary structure is the same for all Oracle instances in a cluster database, as it is for instances in a single-instance database. However, in real application clusters, Oracle synchronizes all the dictionary caches throughout the cluster. Real application clusters use latches to do this, just as in the case of a single-instance Oracle database.
Table locks – These are the GES locks that protect the entire table(s). A transaction acquires a table lock when a table is modified. A table lock can be held in any of several modes: null (N), row share (RS), row exclusive (RX), share lock (S), share row exclusive (SRX), or exclusive (X).

GCS (LMSn + LMD) keeps track of the resources, location and their statuses (mode, role) and this information is recorded in Global Resource Directory (GRD). Each instance maintains its own GRD and manages a portion of the directory. Whenever a block is transferred out of a local cache to another instance’s cache the GRD is updated. A GRD knows where exactly a recent version of the data block is available.

To perform any operation on a data block we need to know the current state of the particular data block. To know its current state, it requires 3 things –

1. What is its current role?(Local(L) or Global(G))
2. What is its current mode?(Null (N) or Shared (S) or Exclusive (E))
3. Whether the requesting block has any Past Images (PI)? (0 or 1)
But where can you get this information from? Yes – you are right – from GRD.

The state of the data block is represented in a 3 letter code – (mode,role,PI) – NL0, SL0, XL1 etc.,

Now let me show the different scenarios of data block transfer –

1. Reading the data block from the disk – In this scenario, initially a data block is read (disk read) from the data file, since no copy of this data block is currently available in any of the instances. Once the block is read into the buffer cache, it is in the state of SL0. This indicates that the block now is in shared mode with local role and doesn’t have any past images. The resource information will be updated accordingly in GRD.

2. Reading the data block from the cache – In this scenario, a data block is currently available in one of the instances buffer cache, so there is no need to read from the disk, thus avoiding a physical read. Let us say, Instance 2 request for a data block, in that regard it sends a request to GCS. GCS in turn passes the request to the owning instance (instance 1). Upon receiving the request, instance 1 forwards the data block to the requesting instance (instance 2) keeping the data block in shared mode and also retains its Local role. No past image is created on instance 1 as the data block was not dirtied yet. Now the state of the data block in instance 2 is SL0 (similar to that of reading from a disk).

3. Modifying the data block – In this scenario, Instance 2 requests for a block to modify, and pass the request to GCS. GCS in turn pass the request to the Instance 1 (owner of that data block). Instance 1 modified this data block, but not committed yet. Upon receiving the request, instance 1 sends the data block to instance 2. Before sending, the resource is downgraded to NULL mode and keeps a copy of current version of the block (PI). Now the role becomes Global, since it is dirtied. It also informs Instance 2 that it retained a PI copy and a NULL resource, which specifies that instance 2 can held the block in exclusive mode (X) with a global role (G). Upon receipt of the block, instance 2 informs GCS about the mode and role of the block (X, G).

4. Writing dirty buffers to disk – In this scenario, Instance 1 wants to write the buffer to disk, so a request is send to GCS. GCS forwards the request to instance 2(current holder of block). Upon receiving the request, instance 2 writes the block to the disk and informs GCS. Instance 2 also informs GCS that the resource role now become local because the instance has completed write of the current block. Upon receiving the message GCS orders all PI holders to discard their PI’s and they no longer need for recovery as the current block is written and buffer is released.

That’s the brief about Cache Fusion – will discuss another topic in the next post.

Scan name, Scan IP/ScanVIP, node VIP, Node IP, Scan listener, local listener, remote listene

SCAN NAME concept was introduced in Oracle 11g and it's still unsolved mystery for few of the Database Administrators. The main purpose for using SCAN NAME was to remove each Cluster Node entry from Tnsnames.ora file from client machine and to make it more dynamic. Though, Oracle successfully achieve his goal but make it complicated for newbies or sometimes for experienced DBA's as well. Each term SCAN NAME, SCAN VIP, SCAN Listener, Node VIP, Local_Listener and Remote_Listener has its own meaning and role to play in RAC environment.


In this post, I will explain meaning and use of each of above discussed Terms. Let's Start with SCAN NAME.


SCAN NAME: SCAN NAME was introduced in Oracle 11g R2 version. This a name which can resolve upto maximum three IP address and minimum one is required for Oracle 11g R2 Real Application Cluster installation. SCAN NAME must be unique in Entire Organization network, This name is used in Tnsname.ora file of client system, all database connection are made thorugh this name. DBA can use below commands to file SCAN NAME details.

[oracle@database bin]$ srvctl config scan
SCAN name: orarac-scan, Network: 1/10.141.132.0/255.255.254.0/eth8
SCAN VIP name: scan1, IP: /orarac-scan/10.191.111.57
SCAN VIP name: scan2, IP: /orarac-scan/10.191.111.55
SCAN VIP name: scan3, IP: /orarac-scan/10.191.111.56


[oracle@database bin]$ nslookup orarac-scan
Server:         111.12.128.13
Address:        111.12.128.13#53

Name:   orarac-scan.oracle.com
Address: 10.191.111.55

Name:   orarac-scan.oracle.com
Address: 10.191.111.56

Name:   orarac-scan.oracle.com
Address: 10.191.111.57


SCAN VIP: SCAN NAME resolves to one or more than one IP addresses, these IP address are called as SCAN VIP or SCAN IP. Each Scan VIP has a SCAN Listener generated corresponding to it. If there is one SCAN IP one SCAN Listener will be generated, if there are three SCAN IP's three SCAN Listeners will be generated. These SCAN Listener runs on any of three nodes on the RAC environment or it could be two SCAN Listener on one node if there are three SCAN Listener and only two nodes.

SCAN VIP and SCAN Listener works as a pair when SCAN VIP fails over to other node, the corresponding SCAN listener will also be failed over to the same node. Whenever SCAN VIP fails over happens, it will always select a node with least running SCAN VIP, i.e., if SCAN VIP runs on node1, node2 and node3 of a 4-node cluster, if node3 goes down, the SCAN VIP and corresponding SCAN listener will be failed over to node4 as the other two nodes already have one SCAN VIP running on each node.

Database Administrator can use below command to find SCAN Listener running on a node and SCAN VIP corresponding to it.

[oracle@database ~]$ ps -ef | grep tnslsnr
oracle   18181 10705  0 15:07 pts/0    00:00:00 grep tnslsnr
grid     22438     1  0 Jul24 ?        00:00:39 /u01/app/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     25927     1  0 Jul24 ?        00:01:31 /u01/app/11.2.0.3/grid/bin/tnslsnr LISTENER -inherit
grid     29211     1  0 Jul28 ?        00:00:27 /u01/app/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN3 -inherit


[oracle@database bin]$ lsnrctl status LISTENER_SCAN3
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 06-AUG-2013 16:16:22
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN3
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                28-JUL-2013 08:38:01
Uptime                    9 days 7 hr. 38 min. 21 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0.3/grid/log/diag/tnslsnr/node2/listener_scan3/alert/log.xml
Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.191.111.57)(PORT=1521)))
Services Summary...
Service "newdb" has 2 instance(s).
  Instance "newdb1", status READY, has 1 handler(s) for this service...
  Instance "newdb2", status READY, has 1 handler(s) for this service...
Service "newdbXDB" has 2 instance(s).
  Instance "newdb1", status READY, has 1 handler(s) for this service...
  Instance "newdb2", status READY, has 1 handler(s) for this service...
The command completed successfully

From the above grep command DBA can see, this system has two SCAN Listener running at this moment. SCAN Listener having name LISTENER_SCAN3 is running on SCAN VIP 10.191.111.57 as shown in HOST parameter of lsnrctl status output.


SCAN Listener: As explained above Each SCAN VIP has SCAN Listener associated with each other and both works as a pair. One SCAN Listener is created corresponding to one SCAN VIP.
SCAN Listener used to run on database nodes. At max there could be three SCAN Listeners in the whole cluster. To file where all these SCAN Listeners are running use

[oracle@database ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node node1

SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node node2

SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node node2

In this example, This is a two node RAC having three SCAN Listeners, Node1 has LISTENER_SCAN1 running while Node2 has LISTENER_SCAN2 and LISTENER_SCAN3 running.


Node VIP: Each Database node in Real Application cluster environment has one node IP and one Node VIP address, The main difference between these two is Node VIP can move to any other system in case if current owning system is down but Node IP can't do that. When ever a new connection request is made the SCAN listener listening on a SCAN IP address and the SCAN port is contracted on a client's behalf. Because all services on the cluster are registered with the SCAN listener, the SCAN listener replies with the address of the local listener as Node VIP address on the least-loaded node (Each scan listener keeps updated cluster load statistics) and connection is routed to that node.

Local Listener on the Database server is registered with Node VIP and Node IP address.

[oracle@database ~]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 07-AUG-2013 05:12:02
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                24-JUL-2013 07:14:10
Uptime                    13 days 21 hr. 57 min. 52 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/node2/listener/alert/log.xml
Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.21.360)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.21.366)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "newdb" has 1 instance(s).
  Instance "newdb2", status READY, has 1 handler(s) for this service...
Service "newdbXDB" has 1 instance(s).
  Instance "newdb2", status READY, has 1 handler(s) for this service...
The command completed successfully

Here, DBA can see Listner is registered with two IP addresses 10.141.21.360 is Node IP address and 10.141.21.366 is node VIP address.


Local_Listener: This is a database parameter which is used to provide detail of local listener running on database node itself. In Real Application Cluster environment this has Node VIP address as value.

SQL> show parameter local_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string       (DESCRIPTION=(ADDRESS_LIST=(A
                                                 DDRESS=(PROTOCOL=TCP)(HOST=10.
                                                 141.21.366)(PORT=1521))))

When a new connection request is made my SCAN Listener, This address is returned to the SCAN Listener and then connection is made to database local listener. The difference between SCAN Listener and Local Listener is SCAN listener runs corosponding to SCAN VIP's while Local Listener runs with Node VIP or Node IP address.SCAN Listener can move to another database node in case of node failure but local Listener doesn't have this kind of behavior.


Remote_Listener: Each database instance has this parameter set to SCAN NAME of the cluster environment. SCAN NAME in turn has one or more IP address further called SCAN VIP and each SCAN VIP has SCAN Listener running on it, so finally each database instance register itself with all SCAN Listeners running across the clusterware.

SQL> show parameter remote_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      orarac-scan:1521

The solo purpose of remote_listener parameter is to register each database instance with all SCAN Listeners in RAC. This provide information on what services are being provided by the instance,the current load, and a recommendation on how many incoming connections should be directed to the instance.


Step by step Database Connection Process In Oracle 11g R2 RAC environment

1. SCAN Name is used in tnsname.ora file of client machine, So for new connection DNS server return to one of the IP address of SCAN Name to client machine called SCAN VIP.
2. SCAN VIP must be using by a Database Node and a SCAN Listener should be running on this SCAN VIP on that node.



3. SCAN Listener has all database instances registered with it, it check internally about load and availability and return Node VIP address to connect for.
4. Each Node VIP has a Local Listener running on database node for it and has a Node IP as well. This returns listener detail and database connection is made with the Local database listener.

In conclusion, This is a complicated process but each and every part has it's own role to play which make it complete.