Oracle 10g RAC Fast Application Notification - Server side callouts

On the database tier, you can implement FAN server-side callouts.


FAN events are generated in response to a state change, which can include the following:
• Starting or stopping the database
• Starting or stopping an instance
• Starting or stopping a service
• A node leaving the cluster


You can also implement server-side callouts, which can be configured to perform actions such as logging fault tickets or notifying the database administrator.

When a state change occurs, the RAC HA framework posts a FAN event to ONS immediately. When a node receives an event through ONS, it will asynchronously execute all executables in the
server-side callouts directory, which is $ORA_CRS_HOME/racg/usrco. Server-side callouts must be stored in this directory; otherwise, they will not be executed.

Create the following callout sample script as Oracle user in $ORA_CRS_HOME/racg/usrco on
Node 1(coltdb01)

[oracle@coltdb01 usrco]$ cat callout.sh
#! /bin/ksh
FAN_LOGFILE=/home/oracle/log/rac_`hostname`.log
echo $* "reported="`date` >> $FAN_LOGFILE &


All the events are recorded and appended to the log file - home/oracle/log/rac_<hostname>.log

You must set the execute permission for the callout

[oracle@coltdb01 usrco]$ chmod 744 /u01/crs/oracle/product/10.2.0/crs/racg/usrco/callout.sh

In addition, you must remember to copy the script to each node in the cluster. The script will automatically be executed whenever the HA framework generates a RAC event.

Example - 1

Node 1 (coltdb01) - shutdown the node-1(coltdb01) instance

[oracle@coltdb01 ~]$ srvctl stop instance -d sendb -i sendb1

Now check the logfile mentioned in callout script on node 1

[oracle@coltdb01 log]$ pwd
/home/oracle/log
[oracle@coltdb01 log]$ ls -ltr
total 4
-rw-rw-r-- 1 oracle oinstall 182 Jun 26 17:20 rac_coltdb01.log
[oracle@coltdb01 log]$ more  rac_coltdb01.log
INSTANCE VERSION=1.0 service=sendb.cms.colt database=sendb instance=sendb1 host=coltdb01 status=down reason=user timestamp=26-Jun-2012 17:20:59 reported=Tue Jun 26 17:20:59 IST 2012

One node-2 & node-3 there are no logfile created as we shutdown only node-1 instance


Example - 2

Node -1 (coltdb1) - shutdown the node-2(coltdb02) instance


Check the logfile on node 1 - Nothing is appended as we shutdown node-2 instance

Check the logfile on node 2
[oracle@coltdb02 log]$ more rac_coltdb02.log
INSTANCE VERSION=1.0 service=sendb.cms.colt database=sendb instance=sendb2 host=coltdb02 status=down reason=user timestamp=26-Jun-2012 17:26:48 reported=Tue Jun 26 17:26:48 IST 2012


Node-3 there are no logfile created as we shutdown only node-2 instance


Example - 3

Node 1 (coltdb01) - Start the database

Logfiles in all the nodes are updated with the Event

Node -1
[oracle@coltdb01 log]$ more  rac_coltdb01.log
INSTANCE VERSION=1.0 service=sendb.cms.colt database=sendb instance=sendb1 host=coltdb01 status=up reason=user timestamp=26-Jun-2012 17:32:44 reported=Tue Jun 26 17:32:44 IST 2012

Node - 2
[oracle@coltdb02 log]$ more rac_coltdb02.log
INSTANCE VERSION=1.0 service=sendb.cms.colt database=sendb instance=sendb2 host=coltdb02 status=up reason=user timestamp=26-Jun-2012 17:33:11 reported=Tue Jun 26 17:33:11 IST 2012

Node-3 
[oracle@coltdb03 log]$ more rac_coltdb03.log
INSTANCE VERSION=1.0 service=sendb.cms.colt database=sendb instance=sendb3 host=coltdb03 status=up reason=user timestamp=26-Jun-2012 17:33:07 reported=Tue Jun 26 17:33:07 IST 2012