Solaris Useful Commands

find command

 

1.- Sort the 40 largest files :

#find /apps/ora11/product -xdev -ls | sort -nrk 7 | head -40

2.-Find a file name with a specific name :

#find . -name “*.jar" -print

3.-Another variant could be without specifying the file extension :

find . -name  jar  -print

du command

du provides output in a readable format, easier to be understood. You will be able to see the information in Bytes, Kilobytes, Megabytes, Gigabytes, etc.

1.- Check all directories and files size in gigas.

#du -sh *
1.5G   agent11g
819M   agent12c
120M   apex
294M   assistants
414M   bin
27M   ccr
3K   cdata
2.4M   cfgtoollogs
42K   clone
1K   config
7.8M   crs
17K   csmig
156K   css
326M   ctx
2.8M   cv

2.-check size for 20 biggest files ordered by size

#du -a | sort -n -r| head -20

iostat command

 

1.-Check disk performance usage in solaris :

#iostat
tty        md10          md11          md12          md20           cpu
tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
0  153  14   2   12    9   2   10    9   2    8    0   0   11   11  2  0 86

The command display all the devices

#iostat -xn 5

extended device statistics
r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
81.0   12.5 10131.5  847.7  0.0  2.4    0.0   25.3   0  42 ssd163
0.4    0.3   30.7    6.1  0.0  0.0    0.0   14.6   0   0 ssd164
54.0    8.9 6738.1  532.1  0.0  0.7    0.0   11.3   0  16 ssd165
0.9    2.5   12.6   14.4  0.0  0.0    0.0    1.9   0   0 ssd166
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 ssd167
2.3    3.0  149.9   29.9  0.0  0.0    0.0    5.0   0   1 ssd168
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 ssd169
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.2   0   0 ssd170
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 ssd171
162.6   16.3 20423.9 1181.9  0.0  1.8    0.0   10.3   0  29 ssd172
133.9   15.8 16853.3 1128.4  0.0  1.2    0.0    8.0   0  22 ssd173
162.6   16.3 20423.9 1181.9  0.0  1.8    0.0   10.3   0  29 ssd172
133.9   15.8 16853.3 1128.4  0.0  1.2    0.0    8.0   0  22 ssd173
85.8    8.5 10808.2  510.2  0.0  1.0    0.0   11.0   0  19 ssd198
44.9    8.1 5599.1  467.6  0.0  0.6    0.0   11.5   0  14 ssd199
44.6    8.1 5558.4  467.7  0.0  0.6    0.0   12.0   0  15 ssd200
126.5   22.9 15817.3 1806.7  0.0  1.2    0.0    7.9   0  21 ssd201

Important Pay special attention to the %w  %b devicecolumns.
%w : Is the wait is the expected rate and should not exceed
%b : Is the disk busy percentage. The most important is that if to a high percentaje, it get free quickly.
device : Is the device disk name.

#iostat -n
tty        md10          md11          md12          md20           cpu
tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
0  153  14   2   12    9   2   10    9   2    8    0   0   11   11  2  0 86

Print logical disk names

#iostat -en

—- errors —
s/w h/w trn tot device
0   0   0   0 md10
0   0   0   0 md11
0   0   0   0 md12
0   0   0   0 md20
0   0   0   0 md21
0   0   0   0 md22
0   0   0   0 md50
0   0   0   0 md51
0   0   0   0 md52
0   0   0   0 sd0
0   0   0   0 sd1
0   0   0   0 sd2
0   0   0   0 ssd72
0   0   0   0 ssd194
0   0   0   0 ssd195
0   0   0   0 ssd196
0   0   0   0 ssd197
0   0   0   0 ssd198
0   0   0   0 ssd199
0   0   0   0 ssd200
0   0   0   0 ssd201

Soft error      : A disk sector fails the CRC check and needs to be re-read
Hard error      : Re-read fails several times for CRC check
Transport error : Errors reported by I/O bus
Total errors    : Soft error + Hard error + Transport errors

#iostat -E

sd0       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: HITACHI  Product: H106030SDSUN300G Revision: A2B0 Serial No: 1219NV1M6B
Size: 300.00GB <300000000000 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
sd1       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: TEAC     Product: DV-W28SS-V       Revision: 1.0B Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
sd2       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

The errors details can be displayed with -e option.

prstat command

prstat -a :  The prstat commands show statistics and projects that currently are running on your system. With -a option shows the sum the total for each column for each user group by number of processes.
#prstat -a
NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
1016 ora11      62G   38G    30% 171:39:20  17%
13 remedyad 3517M 3104M   2.4%  62:36:47 0.6%
30 root      819M  509M   0.4%  16:47:34 0.0%
7 xymon    4608K 6696K   0.0%   0:01:32 0.0%
1 nagios    536K 3616K   0.0%   0:08:11 0.0%
1 noaccess   86M   98M   0.1%   1:01:23 0.0%
1 smmsp    2264K 7400K   0.0%   0:00:24 0.0%
6 daemon   8608K 8976K   0.0%   0:01:27 0.0%

Total: 1075 processes, 12238 lwps, load averages: 7.16, 6.77, 6.56

At the end, you will be able to see the load averages, That is an average for 5, 10 and 15 minutes respectively.

#prstat -J : Same above but group by project :

PROJID    NPROC  SWAP   RSS MEMORY      TIME  CPU PROJECT
100     1011   62G   38G    30% 171:52:09  17% group.dba
3       20 3522M 3109M   2.4%  62:39:44 0.8% default
0       39  916M  613M   0.5%  17:59:02 0.0% system

#prstat -T :  Group by task id :

TASKID    NPROC  SWAP   RSS MEMORY      TIME  CPU PROJECT
225      941   61G   38G    29% 148:11:16  16% group.dba
59726       13 3517M 3104M   2.4%  62:38:24 0.7% default
222224        3 5256K   13M   0.0%   0:00:06 0.6% group.dba
10542       37   14G 7049M   5.4%  12:42:34 0.0% group.dba
216        2  148M  106M   0.1%   4:44:04 0.0% system
162        5  556M  294M   0.2%   9:42:22 0.0% system
855       32   16G   11G   8.5%  11:02:48 0.0% group.dba
177        1 9264K 9264K   0.0%   0:50:29 0.0% system
201        1 8808K   11M   0.0%   0:00:52 0.0% system
206        2 1872K 4568K   0.0%   0:00:16 0.0% system
194        1  816K 2352K   0.0%   0:00:00 0.0% system

Total: 1072 processes, 12209 lwps, load averages: 7.31, 6.75, 6.57

id Command

The id command display the OS current username and group and with -p option prints the current project ID in addition to the user and group IDs

#id -p
uid=1000(ora11) gid=1000(dba) projid=100(group.dba)

 

http://unixaddiction.blogspot.com/2015/09/useful-commands-in-unix.html

 

 

hpux check disk

Check bad sector

more /var/adm/syslog/syslog.log
And most important: make sure the online diagnostics are loaded and running. They will monitor disk error messages and can be configured to notify you when there are error messages. The file: /var/adm/syslog/syslog.log will also have disk errors logged.

ioscan –fCdisk
Use the ioscan command to check the S/W state of the disk. Only disks in state CLAIMED are currently accessible by the system. Disks in other states such as NO_HW or disks that are completely missing from the ioscan output are suspicious. If the disk is marked as CLAIMED, then its controller is responding. For example: # ioscan –fCdisk 
Class I H/W Path Driver S/W State H/W Type Description
======
disk 0 8/4.5.0 sdisk CLAIMED DEVICE SEAGATE ST34572WC
disk 1 8/4.8.0 sdisk UNCLAIMED UNKNOWN SEAGATE ST34572WC
disk 2 8/16/5.2.0 sdisk CLAIMED DEVICE TOSHIBA CD-ROM XM-5401TA
In this example the disk at hardware path 8/4.8.0 is not accessible. If the disk has multiple hardware paths, be sure to check all the paths.


diskinfo –v /dev/rdsk/cXtYdZ
The diskinfo for this disk showing 0 bytes , it is probably bad.

dd if=dev/rdsk/cXtYdZ of=/dev/null bs=1024k
if this exits without I/O error, the disk is assumed OK
For example, the following shows a successful read of the first 64 megabytes of the disk:
dd if=/dev/rdsk/c0t5d0 of=/dev/null bs=1024k count=64
64+0 records in
64+0 records out
The following shows an unsuccessful read of the whole disk:
dd if=/dev/rdsk/c1t3d0 of=/dev/null bs=1024k
dd read error: I/O error
0+0 records in
0+0 records out



Check LVM Errors 
Sometimes LVM commands, such as vgdisplay, return an error suggesting that a disk has problems. For example:
vgdisplay –v | more 

— Physical volumes —
PV Name /dev/dsk/c0t3d0
PV Status unavailable 
Total PE 1023
Free PE 173

The physical volume status of unavailable indicates that LVM is having problems with the disk. You can get the same status information from pvdisplay.
The next two examples are warnings from vgdisplay and vgchange indicating that LVM has no contact with a disk:
vgdisplay -v vg vgdisplay: Warning: couldn’t query physical volume “/dev/dsk/c0t3d0″: The specified path does not correspond to physical volume attached to this volume group vgdisplay: Warning: couldn’t query all of the physical volumes.
vgchange -a y /dev/vg01 vgchange: Warning: Couldn’t attach to the volume group physical volume “/dev/dsk/c0t3d0″: A component of the path of the physical volume does not exist. Volume group “/dev/vg01″ has been successfully changed.
Another sign that you might have a disk problem is seeing stale extents in the output from lvdisplay. If you have stale extents on a logical volume even after running the vgsync or lvsync commands, you might have an issue with an I/O path or one of the disks used by the logical volume, but not necessarily the disk showing stale extents. For example:
lvdisplay –v /dev/vg01/lvol3 | more 

LV Status available/stale 

— Logical extents — LE PV1 PE1 Status 1 PV2 PE2 Status 2 0000 /dev/dsk/c0t3d0 0000 current /dev/dsk/c1t3d0 0100 current 0001 /dev/dsk/c0t3d0 0001 current /dev/dsk/c1t3d0 0101 current 0002 /dev/dsk/c0t3d0 0002 current /dev/dsk/c1t3d0 0102 stale 0003 /dev/dsk/c0t3d0 0003 current /dev/dsk/c1t3d0 0103 stale … 
All LVM error messages tell you which device file is associated with the problematic disk


Check File system corrupt(no disk problem)
umount /fs
fsck -F vxfs -y -o full,nolog /dev/vg01/rlvol1
mount /dev/vg01/rlvol1

check file system consistency check and interactive repair and not fix any errors found
http://kmit4u.blogspot.hk/2010/06/unix-hp-ux-how-to-check-bad-disk.html

 

Datafile Size

SET PAUSE ON
SET PAUSE 'Press Return to Continue'
SET PAGESIZE 60
SET LINESIZE 300
COLUMN "Tablespace Name" FORMAT A20
COLUMN "File Name" FORMAT A80
 
SELECT  Substr(df.tablespace_name,1,20) "Tablespace Name",
        Substr(df.file_name,1,80) "File Name",
        Round(df.bytes/1024/1024,0) "Size (M)",
        decode(e.used_bytes,NULL,0,Round(e.used_bytes/1024/1024,0)) "Used (M)",
        decode(f.free_bytes,NULL,0,Round(f.free_bytes/1024/1024,0)) "Free (M)",
        decode(e.used_bytes,NULL,0,Round((e.used_bytes/df.bytes)*100,0)) "% Used"
FROM    DBA_DATA_FILES DF,
       (SELECT file_id,
               sum(bytes) used_bytes
        FROM dba_extents
        GROUP by file_id) E,
       (SELECT Max(bytes) free_bytes,
               file_id
        FROM dba_free_space
        GROUP BY file_id) f
WHERE    e.file_id (+) = df.file_id
AND      df.file_id  = f.file_id (+)
ORDER BY df.tablespace_name,
         df.file_name
/

ASM Datafiles Commands

Datafiles

Extending an ASM Datafile

The following example shows how to extend a datafile to the APP_DATA tablespace in the +DATA ASM group.

1. Identify the datafiles for the tablespace you want to increase.

SQL> select file_name,bytes/1024/1024 mb
aaaaaafrom dba_data_files
aaaaaawhere tablespace_name = ‘APP_DATA’
aaaaaaorder by file_name;

FILE_NAMEaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaMB
———————————————————— ——-
+DATA/SID/datafile/app_data.259.669898683aaaaaaaaaaaaaaaa20000
+DATA/SID/datafile/app_data.267.636216519aaaaaaaaaaaaaaaa28100

2. Extened the Datafile

SQL> alter database datafile ‘+DATA/SID/datafile/app_data.259.669898683’ resize 30000m;

Tablespace altered.


Adding an ASM Datafile

The following example shows how to add a datafile to the APP_DATA tablespace in the +DATA ASM group.

1. Find out where the current datafiles are defined. In this example the ASM group is +STORE1.

SQL> select file_name,bytes/1024/1024 mb
aaaaaafrom dba_data_files
aaaaaawhere tablespace_name = ‘APP_DATA’
aaaaaaorder by file_name;

2. Add the datafile.

SQL> alter tablespace APP_DATA add datafile ‘+DATA’ size 20000m;

Tablespace altered.

Oracle RAC Logs

The Cluster Ready Services Daemon (crsd) Log Files

Log files for the CRSD process (crsd) can be found in the following directories:

                 CRS home/log/hostname/crsd

Oracle Cluster Registry (OCR) Log Files

The Oracle Cluster Registry (OCR) records log information in the following location:

                CRS Home/log/hostname/client

Cluster Synchronization Services (CSS) Log Files

You can find CSS information that the OCSSD generates in log files in the following locations:

                CRS Home/log/hostname/cssd

Event Manager (EVM) Log Files

Event Manager (EVM) information generated by evmd is recorded in log files in the following locations:

                CRS Home/log/hostname/evmd

RACG Log Files

The Oracle RAC high availability trace files are located in the following two locations:

CRS home/log/hostname/racg

$ORACLE_HOME/log/hostname/racg

Core files are in the sub-directories of the log directories. Each RACG executable has a sub-directory assigned exclusively for that executable. The name of the RACG executable sub-directory is the same as the name of the executable.

You can follow below table which define locations of logs files:

Oracle Clusterware log files

Cluster Ready Services Daemon (crsd) Log Files:
$CRS_HOME/log/hostname/crsd

Cluster Synchronization Services (CSS):
$CRS_HOME/log/hostname/cssd

Event Manager (EVM) information generated by evmd:
$CRS_HOME/log/hostname/evmd

Oracle RAC RACG:
$CRS_HOME/log/hostname/racg
$ORACLE_HOME/log/hostname/racg

Oracle RAC 11g Release 2 log files

Clusterware alert log:
$GRID_HOME/log/<host>/alert<host>.log

Disk Monitor daemon:
$GRID_HOME/log/<host>/diskmon

OCRDUMP, OCRCHECK, OCRCONFIG, CRSCTL:
$GRID_HOME/log/<host>/client

Cluster Time Synchronization Service:
$GRID_HOME/log/<host>/ctssd

Grid Interprocess Communication daemon:
$GRID_HOME/log/<host>/gipcd

Oracle High Availability Services daemon:
$GRID_HOME/log/<host>/ohasd

Cluster Ready Services daemon:
$GRID_HOME/log/<host>/crsd

Grid Plug and Play daemon:
$GRID_HOME/log/<host>/gpnpd:

Mulitcast Domain Name Service daemon:
$GRID_HOME/log/<host>/mdnsd

Event Manager daemon:
$GRID_HOME/log/<host>/evmd

RAC RACG (only used if pre-11.1 database is installed):
$GRID_HOME/log/<host>/racg

Cluster Synchronization Service daemon:
$GRID_HOME/log/<host>/cssd

Server Manager:
$GRID_HOME/log/<host>/srvm

HA Service Daemon Agent:
$GRID_HOME/log/<host>/agent/ohasd/oraagent_oracle11

HA Service Daemon CSS Agent:
$GRID_HOME/log/<host>/agent/ohasd/oracssdagent_root

HA Service Daemon ocssd Monitor Agent:
$GRID_HOME/log/<host>/agent/ohasd/oracssdmonitor_root

HA Service Daemon Oracle Root Agent:
$GRID_HOME/log/<host>/agent/ohasd/orarootagent_root

CRS Daemon Oracle Agent:
$GRID_HOME/log/<host>/agent/crsd/oraagent_oracle11

CRS Daemon Oracle Root Agent:
$GRID_HOME/log/<host> agent/crsd/orarootagent_root

Grid Naming Service daemon:
$GRID_HOME/log/<host>/gnsd

RAC Cheatsheet

RAC Cheatsheet

This is a quick and dirty cheatsheet on Oracle RAC 10g, as my experience with RAC grows I will update this section, below is a beginners guide on the commands and information that you will require to administer Oracle RAC.

Acronyms

Acronyms
GCS Global Cache Services in memory database containing current locks and awaiting locks, also known as PCM
GES Global Enqueue Services coordinates the requests of all global enqueues uses the GCS, also known as non-PCM
GRD Global Resource Directory all resources available to the cluster (formed and managed by GCS and GES), see GRDfor more details
GRM Global Resource Manager helps to coordinate and communicate the locks requests between Oracle processes
GSD Global Services Daemon runs on each node with one GSD process per node. The GSD coordinates with the cluster manager to receive requests from clients such as the DBCA, EM, and the SRVCTL utility to execute administrative job tasks such as instance startup or shutdown. The GSD is not an Oracle instance background process and is therefore not started with the Oracle instance
PCM (IDLM) Parallel Cache Management formly know as (integrated) Distributed Lock Manager, its another name for GCS
Resource n/a it is a identifiable entity it basically has a name or a reference, it can be a area in memory, a disk file or an abstract entity
Resource (Global) n/a a resource that can be accessed by all the nodes within the cluster examples would be the following

  • Data Buffer Cache Block
  • Transaction Enqueue
  • Database Data Structures
LVB Lock Value Block contains a small amount of data regarding the lock
TRFC Traffic Controller controls the DLM traffic between instances (messaging tickets)

Files and Directories

Files and Directories
$ORA_CRS_HOME/cdata/<cluster_name> OCR backups (default location)
$ORA_HOME/log/<hostname>/client/ocrconfig_<pid>.log OCR command log file
$ORA_CRS_HOME/crs/log contains trace files for the CRS resources
$ORA_CRS_HOME/crs/init contains trace files for the CRS daemon during startup, a good place to start
$ORA_CRS_HOME/css/log contains cluster reconfigurations, missed check-ins, connects and disconnects from the client CSS listener. Look here to obtain when reboots occur
$ORA_CRS_HOME/css/init contains core dumps from the cluster synchronization service daemon (OCSd)
$ORA_CRS_HOME/evm/log logfiles for the event volume manager and eventlogger daemon
$ORA_CRS_HOME/evm/init pid and lock files for EVM
$ORA_CRS_HOME/srvm/log logfiles for Oracle Cluster Registry (OCR)
$ORA_CRS_HOME/log log fles for Oracle clusterware which contains diagnostic messages at the Oracle cluster level

Useful Views/Tables

GCS and Cache Fusion Diagnostics
v$cache contains information about every cached block in the buffer cache
v$cache_transfer contains information from the block headers in SGA that have been pinged at least once
v$instance_cache_transfer contains information about the transfer of cache blocks through the interconnect
v$cr_block_server contains statistics about CR block transfer across the instances
v$current_block_server contains statistics about current block transfer across the instances
v$gc_element contains one-to-one information for each global cache resource used by the buffer cache
GES diagnostics
v$lock contains information about locks held within a database and outstanding requests for locks and latches
v$ges_blocking_enqueue contains information about locks that are being blocked or blocking others and locks that are known to the lock manager
v$enqueue_statistics contains details about enqueue statistics in the instance
v$resource_limits display enqueue statistics
v$locked_object contains information about DML locks acquired by different transactions in databases with their mode held
v$ges_statistics contains miscellaneous statistics for GES
v$ges_enqueue contains information about all locks known to the lock manager
v$ges_convert_local contains information about all local GES operations
v$ges_convert_remote contains information about all remote GES operations
v$ges_resource contains information about all resources known to the lock manager
v$ges_misc contains information about messaging traffic information
v$ges_traffic_controller contains information about the message ticket usage
Dynamic Resource Remastering
v$hvmaster_info contains information about current and previous master instances of GES resources in relation to hash value ID of resource
v$gcshvmaster_info the same as above but globally
v$gcspfmaster_info conatins information about current and previous masters about GCS resources belonging to files mapped to a particular master, including the number of times the resource has remastered
Cluster Interconnect
v$cluster_interconnects contains information about interconnects that are being used for cluster communication
v$configured_interconnects same as above but also contains interconnects that AC is aware off that are not being used
Miscellanous
v$service services running on an instance
x$kjmsdp display LMS daemon statistics
x$kjmddp display LMD daemon statistics

Useful Parameters

Parameters
cluster_interconnects specify a specific IP address to use for the inetrconnect
_gcs_fast_config enables fast reconfiguration for gcs locks (true|false)
_lm_master_weight controls which instance will hold or (re)master more resources than others
_gcs_resources controls the number of resources an instance will master at a time
_lm_tickets controls the number of message tickets
_lm_ticket_active_sendback controls the number of message tickets (aggressive messaging)
_db_block_max_cr_dba limits the number of CR copies per DBA on the buffer cache (see grd)
_fairness_threshold used when too many CR requested arrive for a particular buffer and the block becomes disowned (see grd)
_gc_affinity_time specifies interval minutes for reamstering
_gc_affinity_limit defines the number of times a instance access the resource before remastering
_gc_affinity_minimum defines the minimum number of times a instance access the resource before remastering
_lm_file_affinity disables dynamic remastering for the objects belonging to those files
_lm_dynamic_remastering enable or disable remastering
_gc_defer_time define the time by which an instance deferred downgrading a lock (see Cache Fusion)
_lgwr_async_broadcast change the SCN boardcast method (see troubleshooting)

Processes

Oracle RAC Daemons and Processes
OPROCd Process Monitor provides basic cluster integrity services
EVMd Event Management spawns a child process event logger and generates callouts
OCSSd Cluster Synchronization Services basic node membership, group services, basic locking
CRSd Cluster Ready Services resource monitoring, failover and node recovery
LMSn
Lock Manager Server process – GCS this is the cache fusion part, it handles the consistent copies of blocks that are tranferred between instances. It receives requests from LMD to perform lock requests. I rools back any uncommitted transactions. There can be upto ten LMS processes running and can be started dynamically if demand requires it.

they manage lock manager service requests for GCS resources and send them to a service queue to be handled by the LMSn process. It also handles global deadlock detection and monitors for lock conversion timeouts.

LMON
Lock Monitor Process – GES this process manages the GES, it maintains consistency of GCS memory in case of process death. It is also responsible for cluster reconfiguration and locks reconfiguration (node joining or leaving), it checks for instance deaths and listens for local messaging.

A detailed log file is created that tracks any reconfigurations that have happened.

LMD
Lock Manager Daemon – GES this manages the enqueue manager service requests for the GCS. It also handles deadlock detention and remote resource requests from other instances.
LCK0
Lock Process – GES manages instance resource requests and cross-instance call operations for shared resources. It builds a list of invalid lock elements and validates lock elements during recovery.
DIAG
Diagnostic Daemon This is a lightweight process, it uses the DIAG framework to monitor the healt of the cluster. It captures information for later diagnosis in the event of failures. It will perform any neccessary recovery if an operational hang is detected.

General Administration

Managing the Cluster
starting /etc/init.d/init.crs start

crsctl start crs

stopping /etc/init.d/init.crs stop

crsctl stop crs

enable/disable at boot time /etc/init.d/init.crs enable
/etc/init.d/init.crs disable

crsctl enable crs
crsctl disable crs

Managing the database configuration with SRVCTL
start all instances srvctl start database -d <database> -o <option>

Note: starts listeners if not already running, you can use the -o option to specify startup/shutdown options

force
open
mount
nomount

stop all instances srvctl stop database -d <database> -o <option>

Note: the listeners are not stopped, you can use the -o option to specify startup/shutdown options

immediate
abort
normal
transactional

start/stop particular instance srvctl [start|stop] database -d <database> -i <instance>,<instance>
display the registered databases srvctl config database
status srvctl status database -d <database>
srvctl status instance -d <database> -i <instance>,<instance>
srvctl status service -d <database>
srvctl status nodeapps -n <node>
srvctl status asm -n <node>
stopping/starting srvctl stop database -d <database>
srvctl stop instance -d <database> -i <instance>,<instance>
srvctl stop service -d <database> -s <service>,<service> -i <instance>,<instance>
srvctl stop nodeapps -n <node>
srvctl stop asm -n <node>

srvctl start database -d <database>
srvctl start instance -d <database> -i <instance>,<instance>
srvctl start service -d <database> -s <service>,<service> -i <instance>,<instance>
srvctl start nodeapps -n <node>
srvctl start asm -n <node>

adding/removing srvctl add database -d <database> -o <oracle_home>
srvctl add instance -d <database> -i <instance> -n <node>
srvctl add service -d <database> -s <service> -r <preferred_list>
srvctl add nodeapps -n <node> -o <oracle_home> -A <name|ip>/network
srvctl add asm -n <node> -i <asm_instance> -o <oracle_home>

srvctl remove database -d <database> -o <oracle_home>
srvctl remove instance -d <database> -i <instance> -n <node>
srvctl remove service -d <database> -s <service> -r <preferred_list>
srvctl remove nodeapps -n <node> -o <oracle_home> -A <name|ip>/network
srvctl asm remove -n <node>

OCR utilities
log file $ORA_HOME/log/<hostname>/client/ocrconfig_<pid>.log
checking ocrcheck

Note: will return the OCR version, total space allocated, space used, free space, location of each device and the result of the integrity check

dump contents ocrdump -backupfile <file>

Note: by default it dumps the contents into a file named OCRDUMP in the current directory

export/import ocrconfig -export <file>

ocrconfig -restore <file>

backup/restore # show backups
ocrconfig -showbackup

# to change the location of the backup, you can even specify a ASM disk
ocrconfig -backuploc <path|+asm>

# perform a backup, will use the location specified by the -backuploc location
ocrconfig -manualbackup

# perform a restore
ocrconfig -restore <file>

# delete a backup
orcconfig -delete <file>

Note: there are many more option so see the ocrconfig man page

add/remove/replace ## add/relocate the ocrmirror file to the specified location
ocrconfig -replace ocrmirror ‘/ocfs2/ocr2.dbf’

## relocate an existing OCR file
ocrconfig -replace ocr ‘/ocfs1/ocr_new.dbf’

## remove the OCR or OCRMirror file
ocrconfig -replace ocr
ocrconfig -replace ocrmirror

CRS Administration

CRS Administration
starting ## Starting CRS using Oracle 10g R1
not possible

## Starting CRS using Oracle 10g R2
$ORA_CRS_HOME/bin/crsctl start crs

stopping ## Stopping CRS using Oracle 10g R1
srvctl stop database -d <database>
srvctl stop asm -n <node>
srvctl stop nodeapps -n <node>
/etc/init.d/init.crs stop

## Stopping CRS using Oracle 10g R2
$ORA_CRS_HOME/bin/crsctl stop crs

disabling/enabling ## use to stop CRS restarting after a reboot

## Oracle 10g R1
/etc/init.d/init.crs [disable|enable]

## Oracle 10g R2
$ORA_CRS_HOME/bin/crsctl [disable|enable] crs

checking $ORA_CRS_HOME/bin/crsctl check crs
$ORA_CRS_HOME/bin/crsctl check evmd
$ORA_CRS_HOME/bin/crsctl check cssd
$ORA_CRS_HOME/bin/crsctl check crsd
$ORA_CRS_HOME/bin/crsctl check install -wait 600
Resource Applications (CRS Utilities)
status $ORA_CRS_HOME/bin/crs_stat
create profile $ORA_CRS_HOME/bin/crs_profile
register/unregister application $ORA_CRS_HOME/bin/crs_register
$ORA_CRS_HOME/bin/crs_unregister
Start/Stop an application $ORA_CRS_HOME/bin/crs_start
$ORA_CRS_HOME/bin/crs_stop
Resource permissions $ORA_CRS_HOME/bin/crs_getparam
$ORA_CRS_HOME/bin/crs_setparam
Relocate a resource $ORA_CRS_HOME/bin/crs_relocate
Nodes
member number/name olsnodes -n
local node name olsnodes -l
activates logging olsnodes -g
Oracle Interfaces
display oifcfg getif
delete oicfg delig -global
set oicfg setif -global <interface name>/<subnet>:public
oicfg setif -global <interface name>/<subnet>:cluster_interconnect
Global Services Daemon Control
starting gsdctl start
stopping gsdctl stop
status gsdctl status
Cluster Configuration (clscfg is used during installation)
create a new configuration clscfg -install
upgrade or downgrade and existing configuration clscfg -upgrade
clscfg -downgrade
add or delete a node from the configuration clscfg -add
clscfg -delete
create a special single-node configuration for ASM clscfg -local
brief listing of terminology used in the other nodes clscfg -concepts
used for tracing clscfg -trace
help clscfg -h
Cluster Name Check
print cluster name cemulto -n

Note: in Oracle 9i the ulity was called “cemutls"

print the clusterware version cemulto -w

Note: in Oracle 9i the ulity was called “cemutls"

Node Scripts
Add Node addnode.sh

Note: see adding and deleting nodes

Delete Node deletenode.sh

Note: see adding and deleting nodes

Enqueues
displaying statistics SQL> column current_utilization heading current
SQL> column max_utilization heading max_usage
SQL> column initial_allocation heading initial
SQL> column resource_limit format a23;

SQL> select * from v$resource_limit;

Messaging (tickets)
ticket usage select local_nid local, remote_nid remote, tckt_avail avail, tckt_limit limit, snd_q_len send_queue, tckt_wait waiting from v$ges_traffic_controller;
dump ticket information SQL> oradebug setmypid
SQL> oradebug unlimit
SQL> oradebug lkdebug -t
Lighwork Rule and Fairness Threshold
downconvert select cr_requests, light_works, data_requests, fairness_down_converts from v$cr_block_server;

Note: lower the _fairness_threshold if the ratio goes above 40%, set to 0 if the instance is a query only instance.

Remastering
force dynamic remastering (DRM) ## Obtain the OBJECT_ID form the below table
SQL> select * from v$gcspfmaster_info;

## Determine who masters it
SQL> oradebug setmypid
SQL> oradebug lkdebug -a <OBJECT_ID>

## Now remaster the resource
SQL> oradebug setmypid
SQL> oradebug lkdebug -m pkey <OBJECT_ID>

GRD, SRVCTL, GSD and SRVCONFIG Tracing
Enable tracing  $ export SRVM_TRACE=true
Disable tracing  $ export SRVM_TRACE=""

Voting Disk

adding crsctl add css votedisk <file>
deleting crsctl delete css votedisk <file>
querying crsctl query css votedisk

Commonly used srvctl commands

Commonly used srvctl commands

What is srvctl? it is the Server Control Utility, we use SRVCTL to start and stop the database and instances, manage configuration information, and to add, move or remove instances and services.

These are some of the srvctl commands I frequently use, this is not a complete reference guide.

I organized the commands as follow:

  • To start a rac database
  • To stop a rac database
  • To check status and configurations
  • To start and stop instances
  • To start, stop and manage services

Start a rac database (order: nodeapps – asm – database)

srvctl start nodeapps -n nodename
srvctl start asm -n nodename
srvctl start database -d dbname
options are: srvctl start database -d dbname -o open | -o mount | -o nomount

Stop a rac database (order: database – asm – nodeapps)
srvctl stop database -d dbname -o immediate
options are: srvctl stop database -d dbname -o normal | -o transactional | -o immediate | -o abort

srvctl stop asm -n nodename
options are: srvctl stop asm -n nodename -o immediate

srvctl stop nodeapps -n nodename

To check status and configurations
Nodeapps:
srvctl status nodeapps -n nodename

srvctl config nodeapps -n nodename

ASM:
srvctl status asm -n nodename
srvctl config asm -n nodename

Database:
srvctl status database -d dbname
srvctl config database -d dbname (shows instances name, node and oracle home)

Instance:
srvctl status instance -d dbname -i instancename

Services:
srvctl status service -d dbname

To start and stop instances

srvctl start instance -d dbname -i instancename
srvctl stop instance -d dbname -i instancename

To start, stop and manage services
srvctl status service -d dbname
srvctl config service -d dbname
srvctl start service -d dbname -s servicename
srvctl stop service -d dbname -s servicename

srvctl relocate service -d dbname -s servicename -i instancename -t newinstancename [-f]

srvctl documentation for Oracle 10G R2 and 11G

Oracle 10g RAC – CRS commands

This paper will discuss and demonstrate some of the basic RAC management commands you might use to manage your Oracle 10g RAC components. The material presented will be applicable to most if not all Linux and Unix platforms. It will not cover RAC on the Microsoft Windows operating system. We will begin with the basics of checking out a RAC system to identify whether or not the appropriate services and resources are running or not. Then we will go through the basic startup and shutdown commands for all services and resources. Lastly, we will go over the startup of individual resources and checking the status of each step as we progress. The shutdown of individual resources can be done by reversing the individual startup order.

About our Environment

The RAC environment that we will reference throughout this paper is composed of two nodes running Red Hat Enterprise Linux 4 ES with a shared disk, and has two ASM instances named ASM1 and ASM2, two database instances named orcl1 and orcl2, and a service named RAC that is used for transparent application failover (TAF) and load balancing.

Overview of Basic RAC Management Commands

The commands we will use are listed below. Remember that this document is a quick reference, and not an exhaustive list of all commands for managing your RAC environment.

Cluster Related Commands
crs_stat -t Shows HA resource status (hard to read)
crsstat Ouptut of crs_stat -t formatted nicely
ps -ef|grep d.bin crsd.bin evmd.bin ocssd.bin
crsctl check crs CSS,CRS,EVM appears healthy
crsctl stop crs Stop crs and all other services
crsctl disable crs* Prevents CRS from starting on reboot
crsctl enable crs* Enables CRS start on reboot
crs_stop -all Stops all registered resources
crs_start -all Starts all registered resources

* These commands update the file /etc/oracle/scls_scr/<node>/root/crsstart which contains the string “enable” or “disable” as appropriate.

Database Related Commands
srvctl start instance -d <db_name> -i <inst_name> Starts an instance
srvctl start database -d <db_name> Starts all instances
srvctl stop database -d <db_name> Stops all instances, closes database
srvctl stop instance -d <db_name> -i <inst_name> Stops an instance
srvctl start service -d <db_name> -s <service_name> Starts a service
srvctl stop service -d <db_name> -s <service_name> Stops a service
srvctl status service -d <db_name> Checks status of a service
srvctl status instance -d <db_name> -i <inst_name> Checks an individual instance
srvctl status database -d <db_name> Checks status of all instances
srvctl start nodeapps -n <node_name> Starts gsd, vip, listener, and ons
srvctl stop nodeapps -n <node_name> Stops gsd, vip and listener

Keep in mind that some resources will not start unless other resources are already online. We will now look at the general dependency list in greater detail.

There are three main background processes you can see when doing a ps –ef|grep d.bin.  They are normally started by init during the operating system boot process.  They can be started and stopped manually by issuing the command /etc/init.d/init.crs {start|stop|enable|disable}

  1. /etc/rc.d/init.d/init.evmd
  2. /etc/rc.d/init.d/init.cssd
  3. /etc/rc.d/init.d/init.crsd

Once the above processes are running, they will automatically start the following services in the following order if they are enabled.  This list assumes you are using ASM and have a service set up for TAF/load balancing.

  1. The nodeapps (gsd, VIP, ons, listener) are brought online.
  2. The ASM instances are brought online.
  3. The database instances are brought online.
  4. Any defined services are brought online.

Basic RAC Management Commands

Now that we know the dependency tree and have some commands at our disposal, let’s have a look at them one at a time, starting with the cluster commands and processes.

crs_stat -t

This command shows us the status of each registered resource in the cluster. I generally avoid this command because its output is hard to read since the names are truncated as you can see in the sample output below.  You can download a helpful script called crsstat from http://www.dbspecialists.com/specialists/specialist2007-05.html to make it easy on your eyes.

[oracle@green ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    green
ora....EN.lsnr application    ONLINE    ONLINE    green
ora.green.gsd  application    ONLINE    ONLINE    green
ora.green.ons  application    ONLINE    ONLINE    green
ora.green.vip  application    ONLINE    ONLINE    green
ora.....RAC.cs application    ONLINE    ONLINE    red
ora....cl1.srv application    ONLINE    ONLINE    green
ora....cl2.srv application    ONLINE    ONLINE    red
ora.orcl.db    application    ONLINE    ONLINE    red
ora....l1.inst application    ONLINE    ONLINE    green
ora....l2.inst application    ONLINE    ONLINE    red
ora....SM2.asm application    ONLINE    ONLINE    red
ora....ED.lsnr application    ONLINE    ONLINE    red
ora.red.gsd    application    ONLINE    ONLINE    red
ora.red.ons    application    ONLINE    ONLINE    red
ora.red.vip    application    ONLINE    ONLINE    red
[oracle@green ~]$

crsstat

The output of this script is much better. You can learn more about this script and download it at http://www.dbspecialists.com/specialists/specialist2007-05.html.

[oracle@green ~]$ crsstat
HA Resource                                 Target     State
-----------                                 ------     -----
ora.green.ASM1.asm                          ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr               ONLINE     ONLINE on green
ora.green.gsd                               ONLINE     ONLINE on green
ora.green.ons                               ONLINE     ONLINE on green
ora.green.vip                               ONLINE     ONLINE on green
ora.orcl.RAC.cs                             ONLINE     ONLINE on red
ora.orcl.RAC.orcl1.srv                      ONLINE     ONLINE on green
ora.orcl.RAC.orcl2.srv                      ONLINE     ONLINE on red
ora.orcl.db                                 ONLINE     ONLINE on red
ora.orcl.orcl1.inst                         ONLINE     ONLINE on green
ora.orcl.orcl2.inst                         ONLINE     ONLINE on red
ora.red.ASM2.asm                            ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                   ONLINE     ONLINE on red
ora.red.gsd                                 ONLINE     ONLINE on red
ora.red.ons                                 ONLINE     ONLINE on red
ora.red.vip                                 ONLINE     ONLINE on red
[oracle@green ~]$

ps -ef|grep d.bin

We can use this command to verify that the CRS background processes are actually running.  It is implicit that they are running if the crs_stat command and crsstat script work. If they do not work, you will want to verify the background processes are really running.

[root@green ~]# ps -ef|grep d.bin
oracle    5335  3525  0 Jul11 ?        00:00:05 /u01/app/oracle/product/10.2.0/crs/bin/evmd.bin
root      5487  3817  0 Jul11 ?        00:00:00 /u01/app/oracle/product/10.2.0/crs/bin/crsd.bin reboot
oracle    5932  5392  0 Jul11 ?        00:00:00 /u01/app/oracle/product/10.2.0/crs/bin/ocssd.bin
root     30486 30177  0 18:23 pts/1    00:00:00 grep d.bin
[root@green ~]#

crsctl check crs

This command verifies that the above background daemons are functioning.

[oracle@green ~]$ crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[oracle@green ~]$

crsctl stop crs

We’ll need to be logged onto the server as the root user to run this command. It will stop all HA resources on the local node, and it will also stop the above mentioned background daemons.

[oracle@green ~]$ crsctl stop crs
Insufficient user privileges.
[oracle@green ~]$ su
Password:
[root@green oracle]# crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@green oracle]#

crsctl disable crs

This command will prevent CRS from starting on a reboot.  Note there is no return output from the command.

[root@green oracle]# crsctl disable crs
[root@green oracle]#

We did a reboot after this and verified that CRS did not come back online because we wanted to do some operating system maintenance. Let’s check the status by running some of the commands we’ve just discussed.

[oracle@green ~]$ crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
error connecting to CRSD at [(ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))] clsccon 184

[oracle@green ~]$ crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

[oracle@green ~]$ ps -ef|grep d.bin
oracle    6149  5582  0 15:54 pts/1    00:00:00 grep d.bin
[oracle@green ~]$

Everything appears to be down on this node as expected.

Now let’s start everything back up. We will need to be root for this, unless you have been given permissions or sudo to run crsctl start crs.

[root@green oracle]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@green oracle]#

After a few minutes the registered resources for this node should come online. Let’s check to be sure:

[oracle@green ~]$ crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
ora.green.ASM1.asm                            ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr                 ONLINE     ONLINE on green
ora.green.gsd                                 ONLINE     ONLINE on green
ora.green.ons                                 ONLINE     ONLINE on green
ora.green.vip                                 ONLINE     ONLINE on green
ora.orcl.RAC.cs                               ONLINE     ONLINE on red
ora.orcl.RAC.orcl1.srv                        ONLINE     ONLINE on green
ora.orcl.RAC.orcl2.srv                        ONLINE     ONLINE on red
ora.orcl.db                                   ONLINE     ONLINE on red
ora.orcl.orcl1.inst                           ONLINE     ONLINE on green
ora.orcl.orcl2.inst                           ONLINE     ONLINE on red
ora.red.ASM2.asm                              ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                     ONLINE     ONLINE on red
ora.red.gsd                                   ONLINE     ONLINE on red
ora.red.ons                                   ONLINE     ONLINE on red
ora.red.vip                                   ONLINE     ONLINE on red
[oracle@green ~]$

Let’s not forget to enable CRS on reboot:

[root@green oracle]# crsctl enable crs

crs_stop -all

This is a handy script that stops the registered resources and leaves the CRS running.  This includes all services in the cluster, so it will bring down all registered resources on all nodes.

[oracle@green ~]$ crs_stop -all
Attempting to stop `ora.green.gsd` on member `green`
Attempting to stop `ora.orcl.RAC.orcl2.srv` on member `red`
Stop of `ora.orcl.TEST.orcl1.srv` on member `green` succeeded.
Attempting to stop `ora.orcl.RAC.orcl1.srv` on member `green`
Attempting to stop `ora.green.ons` on member `green`
Attempting to stop `ora.orcl.RAC.cs` on member `red`
Stop of `ora.green.gsd` on member `green` succeeded.
Stop of `ora.orcl.RAC.orcl1.srv` on member `green` succeeded.
Stop of `ora.orcl.RAC.orcl2.srv` on member `red` succeeded.
Stop of `ora.orcl.TEST.orcl2.srv` on member `red` succeeded.
Stop of `ora.green.ons` on member `green` succeeded.
--snip--
CRS-0216: Could not stop resource 'ora.orcl.orcl2.inst'.
[oracle@green ~]$

Occasionally you will get the CRS-0216 error message shown above.  This is usually bogus, but you should re-check with crsstat and ps –ef|grep smon or similar to be sure everything has died off.

Let’s verify that crs_stop -all worked as expected:

[root@green oracle]# crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
ora.green.ASM1.asm                            OFFLINE    OFFLINE
ora.green.LISTENER_GREEN.lsnr                 OFFLINE    OFFLINE
ora.green.gsd                                 OFFLINE    OFFLINE
ora.green.ons                                 OFFLINE    OFFLINE
ora.green.vip                                 OFFLINE    OFFLINE
ora.orcl.RAC.cs                               OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                        OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                        OFFLINE    OFFLINE
ora.orcl.db                                   OFFLINE    OFFLINE
ora.orcl.orcl1.inst                           OFFLINE    OFFLINE
ora.orcl.orcl2.inst                           OFFLINE    OFFLINE
ora.red.ASM2.asm                              OFFLINE    OFFLINE
ora.red.LISTENER_RED.lsnr                     OFFLINE    OFFLINE
ora.red.gsd                                   OFFLINE    OFFLINE
ora.red.ons                                   OFFLINE    OFFLINE
ora.red.vip                                   OFFLINE    OFFLINE
[root@green oracle]#

Let’s move on to working with srvctl and managing individual resources. We will begin with the crs background daemons already running, and all registered resources being offline from the last step above.  We will first start the nodeapps, then the ASM instances, followed by the database instances, and lastly the services for TAF and load balancing.  This is the dependency order in our particular environment.  You may or may not have ASM or TAF and load balancing services to start in your environment.

srvctl start nodeapps -n (node)

This will bring up the gsd, ons, listener, and vip. The same command can shut down the nodeapps by replacing start with stop.

[oracle@green ~]$ srvctl start nodeapps -n green
[oracle@green ~]$ srvctl start nodeapps -n red

Now we will check with crsstat again to be sure the nodeapps have started.

[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        OFFLINE    OFFLINE
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                    OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                    OFFLINE    OFFLINE
ora.orcl.db                               OFFLINE    OFFLINE
ora.orcl.orcl1.inst                       OFFLINE    OFFLINE
ora.orcl.orcl2.inst                       OFFLINE    OFFLINE
ora.red.ASM2.asm                          OFFLINE    OFFLINE
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[root@green oracle]#

Now we need to start our ASM instances before we bring up our database and services.

srvctl start asm -n (node)

This will bring up our ASM instances on nodes green and red. Again, the same command will stop the ASM instances by replacing start with stop.

[oracle@green ~]$ srvctl start asm -n green
[oracle@green ~]$ srvctl start asm -n red
[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                    OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                    OFFLINE    OFFLINE
ora.orcl.db                               OFFLINE    OFFLINE
ora.orcl.orcl1.inst                       OFFLINE    OFFLINE
ora.orcl.orcl2.inst                       OFFLINE    OFFLINE
ora.red.ASM2.asm                          ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[oracle@green ~]$

Now let’s bring up our orcl1 and orcl2 instances, and verify they are up with crsstat. Once more we can replace start with stop and shutdown an individual instance if we so choose.

srvctl start instance -d (database) -I (instance)

[oracle@green ~]$ srvctl start instance –d orcl –i orcl1
[oracle@green ~]$ srvctl start instance -d orcl -i orcl2
[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                    OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                    OFFLINE    OFFLINE
ora.orcl.db                               ONLINE     ONLINE on red
ora.orcl.orcl1.inst                       ONLINE     ONLINE on green
ora.orcl.orcl2.inst                       ONLINE     ONLINE on red
ora.red.ASM2.asm                          ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[oracle@green ~]$

srvctl start service -d (database) -s (service)

Now we will finish up by bringing our load balanced/TAF service named RAC online.

[oracle@green ~]$ srvctl start service -d orcl -s RAC
[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           ONLINE     ONLINE on red
ora.orcl.RAC.orcl1.srv                    ONLINE     ONLINE on green
ora.orcl.RAC.orcl2.srv                    ONLINE     ONLINE on red
ora.orcl.db                               ONLINE     ONLINE on red
ora.orcl.orcl1.inst                       ONLINE     ONLINE on green
ora.orcl.orcl2.inst                       ONLINE     ONLINE on red
ora.red.ASM2.asm                          ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[oracle@green ~]$

There we have it; all of our resources are now online. The next steps would be to verify you can connect via SQL*Plus or your favorite application.

Oracle 10g RAC useful commands

Check status of Oracle RAC 

http://dbaquest.blogspot.hk/2014/02/some-useful-commands-to-manage-oracle.html

$ srvctl status database -d RAC
Instance RAC1 is running on node orarac1
Instance RAC2 is running on node orarac2

$srvctl status instance -d RAC -i RAC1
Instance RAC1 is running on node orarac1

$srvctl status asm -n orarac1
ASM instance +ASM1 is running on node orarac1

$srvctl status nodeapps -n orarac1
VIP is running on node: orarac1
GSD is running on node: orarac1
PRKO-2016 : Error in checking condition of listener on node: orarac1
ONS daemon is running on node: orarac1

Please refer PRKO-2016 to Oracle Bug No.4625482 “CHECKING/STARTING/STOPING NAMED LISTENER WITH SRVCTL NODEAPPS FAILS /W PRKO-2016″. It’s a known issue on Oracle 10.2.0.1. Ignore if the listener works fine.

Stop/Start Oracle RAC

1. Stop Oracle 10g on one of RAC nodes.
$ export ORACLE_SID=RAC1
$ srvctl stop instance -d RAC -i RAC1
$ srvctl stop asm -n orarac1 
$ srvctl stop nodeapps -n orarac1 

2. Start Oracle 10g on one of RAC nodes.
$ export ORACLE_SID=RAC1
$ srvctl start nodeapps -n orarac1
$ srvctl start asm -n orarac1 
$ srvctl start instance -d RAC -i RAC1

3. Stop/start Oracle 10g on all RAC nodes.
$ srvctl stop database -d RAC
$ srvctl start database -d RAC

Check Oracle Listener on nodes

$ srvctl config listener -n orarac1
orarac1 LISTENER_ORARAC1
$ lsnrctl start LISTENER_ORARAC1

Check, backup, restore OCR

$ocrconfig -showbackup

$ocrconfig -export /data/backup/rac/ocrdisk.bak

!To restore OCR, it must stop Clusterware on all nodes before.
$ocrconfig -import /data/backup/rac/ocrdisk.bak

$cluvfy comp ocr -n all //verification

$ocrcheck   //check OCR disk usage

Check configuration of ORACLE RAC

$srvctl config database
RAC

$srvctl config database -d RAC
orarac1 RAC1 /space/oracle/product/10.2.0/db_2
orarac2 RAC2 /space/oracle/product/10.2.0/db_2

$srvctl config asm -n orarac2
+ASM2 /space/oracle/product/10.2.0/db_2

$srvctl config nodeapps -n orarac1 -a -g -s -l
VIP exists.: /orarac1-vip.banctecmtl.lan/10.21.51.13/255.255.224.0/Public
GSD exists.
ONS daemon exists.
Listener exists.

SQL> column HOST_NAME format a10;
SQL> select INSTANCE_NAME, HOST_NAME,STARTUP_TIME,STATUS,PARALLEL,DATABASE_STATUS,ACTIVE_STATE from gv$instance;

INSTANCE_NAME   HOST_NAME  STARTUP_T STATUS        PAR DATABASE_STATUS   ACTIVE_ST
---------------- ---------- --------- ------------ --- ----------------- ---------
rac2            ORARAC2    16-SEP-08 OPEN          YES ACTIVE            NORMAL
rac1            ORARAC1    19-SEP-08 OPEN          YES ACTIVE            NORMAL