Pages

Wednesday, July 27, 2016

Tuning Node Evictions 11gR2

Tuning # disktimeout,reboottime,misscount 

[root@tnc1 ~]# crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services
The maximum amount of time, allowed for css writing to voting file to join cluster is defined with “disktimeout” parameter. Default value is 200 seconds  

[root@tnc1 ~]# crsctl get css reboottime
CRS-4678: Successful get reboottime 3 for Cluster Synchronization Services
Default value is 3 seconds, (The amount of time allowed for a node to complete a reboot after the css daemon has been evicted 
This indicates, how long machine will completely shutdown when you do a reboot)

[root@tnc1 ~]# crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services
misscount #  
Misscount can be considered as network latency in seconds
One node will interact with another node through interconnect,
If the node is unable to communicate until “misscount time” the node evicts from cluster,
which we consider as failure in network heartbeat.
The value of css misscount should be lessthan disktimeout. 
In 11GR2 , misscount is to set 30 seconds 

Before 11gR2, we used to shut down cluster, and tuning the disktimeout, misscount and reboottime. From 11gR2, we can change values without shutting down RAC cluster # using (Doc ID 284752.1)
1) Execute crsctl as root to modify the misscount:
$CRS_HOME/bin/crsctl set css misscount <n>    #### where <n> is the maximum private network latency in seconds
$CRS_HOME/bin/crsctl set css reboottime <r> [-force]  #### (<r> is seconds)
$CRS_HOME/bin/crsctl set css disktimeout <d> [-force] #### (<d> is seconds) 

--Nikhil Tatineni--
--11gR2, 12c Cluster -- 

Tuesday, July 26, 2016

RAC ORAchk 12.1.0.4

## orachk  scans and validates across the varoius layers of stack, It helps us to reconfigure environment and finds out potential risks for RAC Environment. This can be configured in the way to send email notifications when it detect problems 

[oracle@tnc1 utl]$ ./orachk 
This version of orachk was released on 07-Feb-2016 and its older than 120 days. No new version of orachk is available in RAT_UPGRADE_LOC. It is highly recommended that you download the latest version of orachk from my oracle support to ensure the highest level of accuracy of the data contained within the report.

Do you want to download latest version from my oracle support? [y/n][y]n
Do you want to continue running this version? [y/n][y]y
Running older version...

CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0.4/grid?[y/n][y]y

Checking ssh user equivalency settings on all nodes in cluster
Node tnc2 is configured for ssh user equivalency for oracle user

Searching for running databases . . . . .
List of running databases registered in OCR
1. mipd09
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].1
. . . . 
Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS
. . . . . . . . . . . . . . .. . . . . . . . . . . .. . . . . . 
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status                            
-------------------------------------------------------------------------------------------------------
Host Name  CRS Installed  RDBMS Installed  CRS UP    ASM UP    RDBMS UP  DB Instance Name
-------------------------------------------------------------------------------------------------------
tnc1        Yes             Yes             Yes        Yes      Yes      mipd091   
tnc2        Yes             Yes             Yes        Yes      Yes      mipd092   
-------------------------------------------------------------------------------------------------------
Copying plug-ins
. . . . . . . . . . 
172 of the included audit checks require root privileged data collection . If sudo is not configured or the root password is not available, audit checks which require root privileged data collection can be skipped.

1. Enter 1 if you will enter root password for each  host when prompted
2. Enter 2 if you have sudo configured for oracle user to execute root_orachk.sh script 
3. Enter 3 to skip the root privileged collections 
4. Enter 4 to exit and work with the SA to configure sudo  or to arrange for root access and run the tool later.

Please indicate your selection from one of the above options for root access[1-4][1]:- 1
*** Checking Best Practice Recommendations (PASS/WARNING/FAIL) ***

Collections and audit checks log file is 
/u01/stage/utl/orachk_tnc1_mipd09_072516_215651/log/orachk.log

Running orachk in serial mode because expect(/usr/bin/expect) is not available to supply root passwords on remote nodes
NOTICE:  Installing the expect utility (/usr/bin/expect) will allow orachk to gather root passwords at the beginning of the process and execute orachk on all nodes in parallel speeding up the entire process. For more info - http://www.nist.gov/el/msid/expect.cfm.  Expect is available for all major platforms.  See User Guide for more details.

Checking for prompts in /home/oracle/.bash_profile on tnc1 for oracle user...
Checking for prompts in /home/oracle/.bash_profile on tnc2 for oracle user...
. . 

=======================================================
                    Node name - tnc1                                
=============================================================
. . . . . 

Collecting - ASM Disk Groups 
Collecting - ASM Disk I/O stats 
Collecting - ASM Diskgroup Attributes 
Collecting - ASM disk partnership imbalance 
Collecting - ASM diskgroup attributes 
Collecting - ASM diskgroup usable free space 
Collecting - ASM initialization parameters 
Collecting - Active sessions load balance for mipd09 database
Collecting - Archived Destination Status for mipd09 database
Collecting - Cluster Interconnect Config for mipd09 database
Collecting - Database Archive Destinations for mipd09 database
Collecting - Database Files for mipd09 database
Collecting - Database Instance Settings for mipd09 database
Collecting - Database Parameters for mipd09 database
Collecting - Database Parameters for mipd09 database
Collecting - Database Properties for mipd09 database
Collecting - Database Registry for mipd09 database
Collecting - Database Sequences for mipd09 database
Collecting - Database Undocumented Parameters for mipd09 database
Collecting - Database Undocumented Parameters for mipd09 database
Collecting - Database Workload Services for mipd09 database
Collecting - Dataguard Status for mipd09 database
Collecting - Files not opened by ASM 
Collecting - Log Sequence Numbers for mipd09 database
Collecting - Percentage of asm disk  Imbalance 
Collecting - Process for shipping Redo to standby for mipd09 database
Collecting - RDBMS Feature Usage for mipd09 database
Collecting - Redo Log information for mipd09 database
Collecting - Standby redo log creation status before switchover for mipd09 database
Collecting - /proc/cmdline
Collecting - /proc/modules
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - CSS Reboot time
Collecting - CSS disktimout
Collecting - Cluster interconnect (clusterware)
Collecting - Clusterware OCR healthcheck 
Collecting - Clusterware Resource Status
Collecting - DiskFree Information
Collecting - DiskMount Information
Collecting - Huge pages configuration
Collecting - Interconnect network card speed
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - NUMA Configuration
Collecting - Network Interface Configuration
Collecting - Network Performance
Collecting - Network Service Switch
Collecting - OS Packages
Collecting - OS version
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure 
Collecting - Patches for RDBMS Home 
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Verify ioctl to advm [ACFS]
Collecting - Voting disks (clusterware)
Collecting - number of semaphore operations per semop system call

Preparing to run root privileged commands  tnc1.  Please enter root password when prompted.
root@tnc1's password: 
Collecting - ACFS Volumes status 
Collecting - ACFS and ASM driver version comparison [ACFS] 
Collecting - Broadcast Requirements for Networks 
Collecting - CRS Opatch version 
Collecting - CRS user time zone check 
Collecting - Custom rc init scripts (rc.local) 
Collecting - Disk Information 
Collecting - Generic ACFS health [ACFS] 
Collecting - Grid Infastructure user shell limits configuration 
Collecting - Health of the mounted ACFS file systems [ACFS] 
Collecting - Health of unmounted ACFS file systems [ACFS] 
Collecting - Interconnect interface config 
Collecting - Network interface stats 
Collecting - OCFS2 disks 
/home/oracle/root_orachk.sh: line 80: /sbin/mounted.ocfs2: No such file or directory
Collecting - OLR Integrity 
Collecting - Root user limits 
Collecting - Verify no database server kernel out of memory errors  
Collecting - Volume list for unmount ACFS file system [ACFS] 
Collecting - ocsf status 
/home/oracle/root_orachk.sh: line 105: /etc/init.d/o2cb: No such file or directory
Collecting - root time zone check 
Collecting - slabinfo 
Collecting - umask setting for GI owner 


Data collections completed. Checking best practices on tnc1.
--------------------------------------------------------------------------------------


 WARNING => Cluster Health Monitor (CHM) repository does not provide recommended level of retention
 FAIL =>    Database DB_CREATE_FILE_DEST and DB_RECOVERY_FILE_DEST are NOT in different diskgroups for mipd09
 INFO =>    Important Automatic Storage Management (ASM) Notes and Technical White Papers
 INFO =>    Oracle Data Pump Best practices.
 FAIL =>    Bash is vulnerable to code injection (CVE-2014-6271)
 WARNING => os kernel version value not met the minimum requirement
 WARNING => free physical memory is not at sufficient level on system
 INFO =>    Important Storage Minimum Requirements for Grid & Database Homes
 FAIL =>    DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for mipd09
 INFO =>    Most recent ADR incidents for /u01/app/oracle/product/11.2.0.4/db_1
 WARNING => Consider unsetting database Parameter DB_FILE_MULTIBLOCK_READ_COUNT for mipd09
 INFO =>    Direct NFS Client is NOT enabled for mipd09
 INFO =>    Oracle GoldenGate failure prevention best practices
 WARNING => Local archive destination does not alternate destination configured for mipd09
 WARNING => OCR is NOT being backed up daily
 WARNING => Package libaio-devel-0.3.106-x86_64 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-i386 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-x86_64 is recommended but NOT installed
 WARNING => Package unixODBC-devel-2.2.11-x86_64 is recommended but NOT installed
 WARNING => Package sysstat-7.0.2-x86_64 is recommended but NOT installed
 INFO =>    Some data or temp files are not autoextensible for mipd09
 INFO =>    oracleasm (asmlib) module is not loaded
 INFO =>    Number of SCAN listeners is NOT equal to the recommended number of 3.
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU >= 9000) are not configured for interconnect
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on mipd091 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on mipd091 instance
 WARNING => CSS log file size has not been increased from default (50 MB) .
 WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for mipd09
 FAIL =>    Flashback on PRIMARY is not configured for mipd09
 INFO =>    Operational Best Practices
 INFO =>    Database Consolidation Best Practices
 INFO =>    Computer failure prevention best practices
 INFO =>    Data corruption prevention best practices
 INFO =>    Logical corruption prevention best practices
 INFO =>    Database/Cluster/Site failure prevention best practices
 INFO =>    Client failover operational best practices
 WARNING => fast_start_mttr_target should be greater than or equal to 300. on mipd091 instance

 INFO =>    IMPORTANT: Oracle Database Patch 17478514 PSU is NOT applied to RDBMS Home /u01/app/oracle/product/11.2.0.4/db_1
 INFO =>    Information about hanganalyze and systemstate dump
 WARNING => Value of remote_listener parameter is not able to tnsping for mipd09
 FAIL =>    Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for mipd09
 INFO =>    Database failure prevention best practices
 FAIL =>    Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for mipd09
 WARNING => avahi-daemon process is running
 WARNING => loopback address is NOT configured as recommended in /etc/hosts
 INFO =>    Parallel Execution Health-Checks and Diagnostics Reports for mipd09
 WARNING => vm.min_free_kbytes should be set as recommended.
 WARNING => OS parameter vm.swappiness is NOT set to the recommended value
 FAIL =>    control_file_record_keep_time is set to zero for mipd09
 FAIL =>    FRA space management problem file types are present without an RMAN backup completion within the last 7 days. for mipd09
 INFO =>    Oracle recovery manager(rman) best practices
 WARNING => RMAN controlfile autobackup should be set to ON for mipd09
 WARNING => Linux Disk I/O Scheduler should be configured to Deadline
 INFO =>    Consider increasing the COREDUMPSIZE size
 WARNING => Consider investigating changes to the schema objects such as DDLs or new object creation for mipd09
 WARNING => Consider investigating the frequency of SGA resize operations and take corrective action for mipd09


Best Practice checking completed.Checking recommended patches on tnc1.
---------------------------------------------------------------------------------
Collecting patch inventory on  CRS HOME /u01/app/11.2.0.4/grid
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0.4/db_1 
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112040 from /u01/app/11.2.0.4/grid on tnc1
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description                            
---------------------------------------------------------------------------------
22191577  no          no  /u01/app/oracle/product/11.2.0.4/db_1GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0.4/db_1 on tnc1
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description                       
---------------------------------------------------------------------------------
22191577  no             merge               GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM 
---------------------------------------------------------------------------------
1              0              0                0              
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME    
---------------------------------------------------------------------------------
 1              0              0                /u01/app/oracle/product/11.2.0.4/db_1
---------------------------------------------------------------------------------



=============================================================
                    Node name - tnc2                                
=============================================================
. . . . . 

Collecting - /proc/cmdline
Collecting - /proc/modules
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - Cluster interconnect (clusterware)
Collecting - DiskFree Information
Collecting - DiskMount Information
Collecting - Huge pages configuration
Collecting - Interconnect network card speed
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - NUMA Configuration
Collecting - Network Interface Configuration
Collecting - Network Performance
Collecting - Network Service Switch
Collecting - OS Packages
Collecting - OS version
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure 
Collecting - Patches for RDBMS Home 
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Verify ioctl to advm [ACFS]
Collecting - number of semaphore operations per semop system call



Preparing to run root privileged commands  tnc2.  Please enter root password when prompted.
root@tnc2's password: 
Collecting - ACFS Volumes status 
Collecting - ACFS and ASM driver version comparison [ACFS] 
Collecting - Broadcast Requirements for Networks 
Collecting - CRS Opatch version 
Collecting - CRS user time zone check 
Collecting - Disk Information 
Collecting - Generic ACFS health [ACFS] 
Collecting - Grid Infastructure user shell limits configuration 
Collecting - Health of the mounted ACFS file systems [ACFS] 
Collecting - Health of unmounted ACFS file systems [ACFS] 
Collecting - Interconnect interface config 
Collecting - Network interface stats 
Collecting - OCFS2 disks 
/home/oracle/root_orachk.sh: line 75: /sbin/mounted.ocfs2: No such file or directory
Collecting - OLR Integrity 
Collecting - Root user limits 
Collecting - Verify no database server kernel out of memory errors  
Collecting - Volume list for unmount ACFS file system [ACFS] 
Collecting - ocsf status 
/home/oracle/root_orachk.sh: line 100: /etc/init.d/o2cb: No such file or directory
Collecting - root time zone check 
Collecting - slabinfo 
Collecting - umask setting for GI owner 


Data collections completed. Checking best practices on tnc2.
--------------------------------------------------------------------------------------


 FAIL =>    Bash is vulnerable to code injection (CVE-2014-6271)
 WARNING => system is not started with runlevel 3 or 5
 WARNING => os kernel version value not met the minimum requirement
 WARNING => free physical memory is not at sufficient level on system
 INFO =>    Important Storage Minimum Requirements for Grid & Database Homes
 FAIL =>    Hidden ASM initialization parameters should not be set per best practice recommendations
 INFO =>    Hidden database initialization parameters should not be set per best practice recommendations for mipd09
 INFO =>    Most recent ADR incidents for /u01/app/oracle/product/11.2.0.4/db_1
 WARNING => Non-default database Services are not configured for mipd09
 INFO =>    $CRS_HOME/log/hostname/client directory has too many older log files.
 WARNING => The number of async IO descriptors is too low (/proc/sys/fs/aio-max-nr)
 INFO =>    core_dump_dest has too many older core dump files for mipd09
 WARNING => Package libaio-devel-0.3.106-x86_64 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-i386 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-x86_64 is recommended but NOT installed
 WARNING => Package unixODBC-devel-2.2.11-x86_64 is recommended but NOT installed
 WARNING => Package sysstat-7.0.2-x86_64 is recommended but NOT installed
 INFO =>    oracleasm (asmlib) module is not loaded
 WARNING => /tmp is not on a dedicated filesystem
 WARNING => ACFS driver is not online[ACFS]
 WARNING => Linux Swap Configuration does NOT meet Recommendation
 WARNING => Free space in /tmp directory does NOT meet recommendation of minimum 1GB
 WARNING => ACFS registry is not online [ACFS]
 WARNING => kernel.shmmax parameter is NOT configured according to recommendation
 WARNING => ip_local_port_range is NOT configured according to recommendation
 INFO =>    Public interface is not configured in OCR
 INFO =>    Number of SCAN listeners is NOT equal to the recommended number of 3.
 WARNING => ASM shared_pool_size is < recommended value
 WARNING => crsd Log Ownership is NOT Correct (should be root root)
 WARNING => crsd/orarootagent_root Log Ownership is NOT Correct (should be root root)
 WARNING => ohasd/orarootagent_root Log Ownership is NOT Correct (should be root root)
 WARNING => ohasd Log Ownership is NOT Correct (should be root root)
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU >= 9000) are not configured for interconnect
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on mipd092 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on mipd092 instance
 WARNING => fast_start_mttr_target should be greater than or equal to 300. on mipd092 instance
 WARNING => Local listener init parameter is not set to local node VIP for mipd09
 WARNING => Server-side TNSNAMES.ora alias resolves using SCAN EZ syntax for mipd09
 WARNING => Remote listener is not set to SCAN name for mipd09
 WARNING => avahi-daemon process is running
 WARNING => Redo log write time is more than 500 milliseconds for mipd09
 WARNING => loopback address is NOT configured as recommended in /etc/hosts
 WARNING => TFA Collector is either not installed or not running
 WARNING => System clock is not synchronized to hardware clock at system shutdown
 WARNING => OS parameter vm.swappiness is NOT set to the recommended value
 FAIL =>    ASM_DISKSTRING parameter is either null or set to /dev/*
 WARNING => Linux Disk I/O Scheduler should be configured to Deadline
 WARNING => ADVM Volume(s) should not be configured with redundancy as UNPROT when the underlying ASM Disk Group(s) have a redundancy of MIRROR or HIGH.
 INFO =>    Consider increasing the COREDUMPSIZE size
 FAIL =>    Online (hot) patches are applied to ORACLE_HOME. for mipd09
 WARNING => ASM memory_target is not set to recommended value


Best Practice checking completed.Checking recommended patches on tnc2.
---------------------------------------------------------------------------------
Collecting patch inventory on  CRS HOME /u01/app/11.2.0.4/grid 
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0.4/db_1 
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112040 from /u01/app/11.2.0.4/grid on tnc2
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description                            
---------------------------------------------------------------------------------
22191577  no          no  /u01/app/oracle/product/11.2.0.4/db_1GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0.4/db_1 on tnc2
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description                       
---------------------------------------------------------------------------------
22191577  no             merge               GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM 
---------------------------------------------------------------------------------
1              0              0                0              
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME    
---------------------------------------------------------------------------------
 1              0              0                /u01/app/oracle/product/11.2.0.4/db_1
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
                      CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
 WARNING => DNS server config file does not match across database servers
 WARNING => RDBMS software version does not match across cluster.
 WARNI[oracle@tnc1 utl]$ ls -ltr
total 67680
-rw-r--r-- 1 oracle oinstall     2888 Feb  8 00:30 user_defined_checks.xsd
-rw-r--r-- 1 oracle oinstall    40052 Feb  8 00:30 sample_user_defined_checks.xml
-rwxr-xr-x 1 oracle oinstall  2653265 Feb  8 00:30 orachk
-rw-r--r-- 1 oracle oinstall  4690680 Feb  8 00:30 CollectionManager_App.sql
-rw-r--r-- 1 oracle oinstall      425 Feb  8 00:31 UserGuide.txt
-rw-r--r-- 1 oracle oinstall     3869 Feb  8 00:31 readme.txt
-rw-r--r-- 1 oracle oinstall  5613338 Feb  8 00:31 rules.dat
-rw-r--r-- 1 oracle oinstall 44243042 Feb  8 00:31 collections.dat
-rw-r--r-- 1 oracle oinstall  2077055 Feb  8 12:13 Audit_Checks_Report_Orachk.html
-rw-r--r-- 1 oracle oinstall  9860069 Jul 17 22:01 orachk.zip
[oracle@tnc1 utl]$ 
[oracle@tnc1 utl]$ 
[oracle@tnc1 utl]$ ./orachk 
This version of orachk was released on 07-Feb-2016 and its older than 120 days. No new version of orachk is available in RAT_UPGRADE_LOC. It is highly recommended that you download the latest version of orachk from my oracle support to ensure the highest level of accuracy of the data contained within the report.

Do you want to download latest version from my oracle support? [y/n][y]n

Do you want to continue running this version? [y/n][y]y
Running older version...

CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0.4/grid?[y/n][y]y

Checking ssh user equivalency settings on all nodes in cluster
Node tnc2 is configured for ssh user equivalency for oracle user

Searching for running databases . . . . .
. . 
List of running databases registered in OCR
1. mipd09
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].1
. . . . 

Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . . . . . . . .. . . . . . . . . . . .. . . . . . 
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status                            
-------------------------------------------------------------------------------------------------------
Host Name  CRS Installed  RDBMS Installed  CRS UP    ASM UP    RDBMS UP  DB Instance Name
-------------------------------------------------------------------------------------------------------
tnc1        Yes             Yes             Yes        Yes      Yes      mipd091   
tnc2        Yes             Yes             Yes        Yes      Yes      mipd092   
-------------------------------------------------------------------------------------------------------
Copying plug-ins
. . . . . . . . . . 
172 of the included audit checks require root privileged data collection . If sudo is not configured or the root password is not available, audit checks which require root privileged data collection can be skipped.

1. Enter 1 if you will enter root password for each  host when prompted
2. Enter 2 if you have sudo configured for oracle user to execute root_orachk.sh script 
3. Enter 3 to skip the root privileged collections 
4. Enter 4 to exit and work with the SA to configure sudo  or to arrange for root access and run the tool later.

Please indicate your selection from one of the above options for root access[1-4][1]:- 1
*** Checking Best Practice Recommendations (PASS/WARNING/FAIL) ***

Collections and audit checks log file is 
/u01/stage/utl/orachk_tnc1_mipd09_072516_215651/log/orachk.log

Running orachk in serial mode because expect(/usr/bin/expect) is not available to supply root passwords on remote nodes
NOTICE:  Installing the expect utility (/usr/bin/expect) will allow orachk to gather root passwords at the beginning of the process and execute orachk on all nodes in parallel speeding up the entire process. For more info - http://www.nist.gov/el/msid/expect.cfm.  Expect is available for all major platforms.  See User Guide for more details.

Checking for prompts in /home/oracle/.bash_profile on tnc1 for oracle user...
Checking for prompts in /home/oracle/.bash_profile on tnc2 for oracle user...
. . 
=======================================================
                    Node name - tnc1                                
=============================================================
. . . . . 

Collecting - ASM Disk Groups 
Collecting - ASM Disk I/O stats 
Collecting - ASM Diskgroup Attributes 
Collecting - ASM disk partnership imbalance 
Collecting - ASM diskgroup attributes 
Collecting - ASM diskgroup usable free space 
Collecting - ASM initialization parameters 
Collecting - Active sessions load balance for mipd09 database
Collecting - Archived Destination Status for mipd09 database
Collecting - Cluster Interconnect Config for mipd09 database
Collecting - Database Archive Destinations for mipd09 database
Collecting - Database Files for mipd09 database
Collecting - Database Instance Settings for mipd09 database
Collecting - Database Parameters for mipd09 database
Collecting - Database Parameters for mipd09 database
Collecting - Database Properties for mipd09 database
Collecting - Database Registry for mipd09 database
Collecting - Database Sequences for mipd09 database
Collecting - Database Undocumented Parameters for mipd09 database
Collecting - Database Undocumented Parameters for mipd09 database
Collecting - Database Workload Services for mipd09 database
Collecting - Dataguard Status for mipd09 database
Collecting - Files not opened by ASM 
Collecting - Log Sequence Numbers for mipd09 database
Collecting - Percentage of asm disk  Imbalance 
Collecting - Process for shipping Redo to standby for mipd09 database
Collecting - RDBMS Feature Usage for mipd09 database
Collecting - Redo Log information for mipd09 database
Collecting - Standby redo log creation status before switchover for mipd09 database
Collecting - /proc/cmdline
Collecting - /proc/modules
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - CSS Reboot time
Collecting - CSS disktimout
Collecting - Cluster interconnect (clusterware)
Collecting - Clusterware OCR healthcheck 
Collecting - Clusterware Resource Status
Collecting - DiskFree Information
Collecting - DiskMount Information
Collecting - Huge pages configuration
Collecting - Interconnect network card speed
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - NUMA Configuration
Collecting - Network Interface Configuration
Collecting - Network Performance
Collecting - Network Service Switch
Collecting - OS Packages
Collecting - OS version
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure 
Collecting - Patches for RDBMS Home 
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Verify ioctl to advm [ACFS]
Collecting - Voting disks (clusterware)
Collecting - number of semaphore operations per semop system call

Preparing to run root privileged commands  tnc1.  Please enter root password when prompted.
root@tnc1's password: 
Collecting - ACFS Volumes status 
Collecting - ACFS and ASM driver version comparison [ACFS] 
Collecting - Broadcast Requirements for Networks 
Collecting - CRS Opatch version 
Collecting - CRS user time zone check 
Collecting - Custom rc init scripts (rc.local) 
Collecting - Disk Information 
Collecting - Generic ACFS health [ACFS] 
Collecting - Grid Infastructure user shell limits configuration 
Collecting - Health of the mounted ACFS file systems [ACFS] 
Collecting - Health of unmounted ACFS file systems [ACFS] 
Collecting - Interconnect interface config 
Collecting - Network interface stats 
Collecting - OCFS2 disks 
/home/oracle/root_orachk.sh: line 80: /sbin/mounted.ocfs2: No such file or directory
Collecting - OLR Integrity 
Collecting - Root user limits 
Collecting - Verify no database server kernel out of memory errors  
Collecting - Volume list for unmount ACFS file system [ACFS] 
Collecting - ocsf status 
/home/oracle/root_orachk.sh: line 105: /etc/init.d/o2cb: No such file or directory
Collecting - root time zone check 
Collecting - slabinfo 
Collecting - umask setting for GI owner 

Data collections completed. Checking best practices on tnc1.
--------------------------------------------------------------------------------------
 WARNING => Cluster Health Monitor (CHM) repository does not provide recommended level of retention
 FAIL =>    Database DB_CREATE_FILE_DEST and DB_RECOVERY_FILE_DEST are NOT in different diskgroups for mipd09
 INFO =>    Important Automatic Storage Management (ASM) Notes and Technical White Papers
 INFO =>    Oracle Data Pump Best practices.
 FAIL =>    Bash is vulnerable to code injection (CVE-2014-6271)
 WARNING => os kernel version value not met the minimum requirement
 WARNING => free physical memory is not at sufficient level on system
 INFO =>    Important Storage Minimum Requirements for Grid & Database Homes
 FAIL =>    DB_UNIQUE_NAME on primary has not been modified from the default, confirm that database name is unique across your Oracle enterprise. for mipd09
 INFO =>    Most recent ADR incidents for /u01/app/oracle/product/11.2.0.4/db_1
 WARNING => Consider unsetting database Parameter DB_FILE_MULTIBLOCK_READ_COUNT for mipd09
 INFO =>    Direct NFS Client is NOT enabled for mipd09
 INFO =>    Oracle GoldenGate failure prevention best practices
 WARNING => Local archive destination does not alternate destination configured for mipd09
 WARNING => OCR is NOT being backed up daily
 WARNING => Package libaio-devel-0.3.106-x86_64 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-i386 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-x86_64 is recommended but NOT installed
 WARNING => Package unixODBC-devel-2.2.11-x86_64 is recommended but NOT installed
 WARNING => Package sysstat-7.0.2-x86_64 is recommended but NOT installed
 INFO =>    Some data or temp files are not autoextensible for mipd09
 INFO =>    oracleasm (asmlib) module is not loaded
 INFO =>    Number of SCAN listeners is NOT equal to the recommended number of 3.
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU >= 9000) are not configured for interconnect
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on mipd091 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on mipd091 instance
 WARNING => CSS log file size has not been increased from default (50 MB) .
 WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for mipd09
 FAIL =>    Flashback on PRIMARY is not configured for mipd09
 INFO =>    Operational Best Practices
 INFO =>    Database Consolidation Best Practices
 INFO =>    Computer failure prevention best practices
 INFO =>    Data corruption prevention best practices
 INFO =>    Logical corruption prevention best practices
 INFO =>    Database/Cluster/Site failure prevention best practices
 INFO =>    Client failover operational best practices
 WARNING => fast_start_mttr_target should be greater than or equal to 300. on mipd091 instance

 INFO =>    IMPORTANT: Oracle Database Patch 17478514 PSU is NOT applied to RDBMS Home /u01/app/oracle/product/11.2.0.4/db_1
 INFO =>    Information about hanganalyze and systemstate dump
 WARNING => Value of remote_listener parameter is not able to tnsping for mipd09
 FAIL =>    Table AUD$[FGA_LOG$] should use Automatic Segment Space Management for mipd09
 INFO =>    Database failure prevention best practices
 FAIL =>    Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for mipd09
 WARNING => avahi-daemon process is running
 WARNING => loopback address is NOT configured as recommended in /etc/hosts
 INFO =>    Parallel Execution Health-Checks and Diagnostics Reports for mipd09
 WARNING => vm.min_free_kbytes should be set as recommended.
 WARNING => OS parameter vm.swappiness is NOT set to the recommended value
 FAIL =>    control_file_record_keep_time is set to zero for mipd09
 FAIL =>    FRA space management problem file types are present without an RMAN backup completion within the last 7 days. for mipd09
 INFO =>    Oracle recovery manager(rman) best practices
 WARNING => RMAN controlfile autobackup should be set to ON for mipd09
 WARNING => Linux Disk I/O Scheduler should be configured to Deadline
 INFO =>    Consider increasing the COREDUMPSIZE size
 WARNING => Consider investigating changes to the schema objects such as DDLs or new object creation for mipd09
 WARNING => Consider investigating the frequency of SGA resize operations and take corrective action for mipd09

Best Practice checking completed.Checking recommended patches on tnc1.
---------------------------------------------------------------------------------
Collecting patch inventory on  CRS HOME /u01/app/11.2.0.4/grid
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0.4/db_1 
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112040 from /u01/app/11.2.0.4/grid on tnc1
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description                        
---------------------------------------------------------------------------------
22191577  no          no  /u01/app/oracle/product/11.2.0.4/db_1GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0.4/db_1 on tnc1
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description                       
---------------------------------------------------------------------------------
22191577  no             merge               GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM 
---------------------------------------------------------------------------------
1              0              0                0              
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME    
---------------------------------------------------------------------------------
 1              0              0                /u01/app/oracle/product/11.2.0.4/db_1
---------------------------------------------------------------------------------
=============================================================
                    Node name - tnc2                                
=============================================================
. . . . . 

Collecting - /proc/cmdline
Collecting - /proc/modules
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - Cluster interconnect (clusterware)
Collecting - DiskFree Information
Collecting - DiskMount Information
Collecting - Huge pages configuration
Collecting - Interconnect network card speed
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - NUMA Configuration
Collecting - Network Interface Configuration
Collecting - Network Performance
Collecting - Network Service Switch
Collecting - OS Packages
Collecting - OS version
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure 
Collecting - Patches for RDBMS Home 
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Verify ioctl to advm [ACFS]
Collecting - number of semaphore operations per semop system call

Preparing to run root privileged commands  tnc2.  Please enter root password when prompted.
root@tnc2's password: 
Collecting - ACFS Volumes status 
Collecting - ACFS and ASM driver version comparison [ACFS] 
Collecting - Broadcast Requirements for Networks 
Collecting - CRS Opatch version 
Collecting - CRS user time zone check 
Collecting - Disk Information 
Collecting - Generic ACFS health [ACFS] 
Collecting - Grid Infastructure user shell limits configuration 
Collecting - Health of the mounted ACFS file systems [ACFS] 
Collecting - Health of unmounted ACFS file systems [ACFS] 
Collecting - Interconnect interface config 
Collecting - Network interface stats 
Collecting - OCFS2 disks 
/home/oracle/root_orachk.sh: line 75: /sbin/mounted.ocfs2: No such file or directory
Collecting - OLR Integrity 
Collecting - Root user limits 
Collecting - Verify no database server kernel out of memory errors  
Collecting - Volume list for unmount ACFS file system [ACFS] 
Collecting - ocsf status 
/home/oracle/root_orachk.sh: line 100: /etc/init.d/o2cb: No such file or directory
Collecting - root time zone check 
Collecting - slabinfo 
Collecting - umask setting for GI owner 

Data collections completed. Checking best practices on tnc2.
--------------------------------------------------------------------------------------
 FAIL =>    Bash is vulnerable to code injection (CVE-2014-6271)
 WARNING => system is not started with runlevel 3 or 5
 WARNING => os kernel version value not met the minimum requirement
 WARNING => free physical memory is not at sufficient level on system
 INFO =>    Important Storage Minimum Requirements for Grid & Database Homes
 FAIL =>    Hidden ASM initialization parameters should not be set per best practice recommendations
 INFO =>    Hidden database initialization parameters should not be set per best practice recommendations for mipd09
 INFO =>    Most recent ADR incidents for /u01/app/oracle/product/11.2.0.4/db_1
 WARNING => Non-default database Services are not configured for mipd09
 INFO =>    $CRS_HOME/log/hostname/client directory has too many older log files.
 WARNING => The number of async IO descriptors is too low (/proc/sys/fs/aio-max-nr)
 INFO =>    core_dump_dest has too many older core dump files for mipd09
 WARNING => Package libaio-devel-0.3.106-x86_64 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-i386 is recommended but NOT installed
 WARNING => Package libaio-devel-0.3.106-3.2-x86_64 is recommended but NOT installed
 WARNING => Package unixODBC-devel-2.2.11-x86_64 is recommended but NOT installed
 WARNING => Package sysstat-7.0.2-x86_64 is recommended but NOT installed
 INFO =>    oracleasm (asmlib) module is not loaded
 WARNING => /tmp is not on a dedicated filesystem
 WARNING => ACFS driver is not online[ACFS]
 WARNING => Linux Swap Configuration does NOT meet Recommendation
 WARNING => Free space in /tmp directory does NOT meet recommendation of minimum 1GB
 WARNING => ACFS registry is not online [ACFS]
 WARNING => kernel.shmmax parameter is NOT configured according to recommendation
 WARNING => ip_local_port_range is NOT configured according to recommendation
 INFO =>    Public interface is not configured in OCR
 INFO =>    Number of SCAN listeners is NOT equal to the recommended number of 3.
 WARNING => ASM shared_pool_size is < recommended value
 WARNING => crsd Log Ownership is NOT Correct (should be root root)
 WARNING => crsd/orarootagent_root Log Ownership is NOT Correct (should be root root)
 WARNING => ohasd/orarootagent_root Log Ownership is NOT Correct (should be root root)
 WARNING => ohasd Log Ownership is NOT Correct (should be root root)
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU >= 9000) are not configured for interconnect
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on mipd092 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on mipd092 instance
 WARNING => fast_start_mttr_target should be greater than or equal to 300. on mipd092 instance
 WARNING => Local listener init parameter is not set to local node VIP for mipd09
 WARNING => Server-side TNSNAMES.ora alias resolves using SCAN EZ syntax for mipd09
 WARNING => Remote listener is not set to SCAN name for mipd09
 WARNING => avahi-daemon process is running
 WARNING => Redo log write time is more than 500 milliseconds for mipd09
 WARNING => loopback address is NOT configured as recommended in /etc/hosts
 WARNING => TFA Collector is either not installed or not running
 WARNING => System clock is not synchronized to hardware clock at system shutdown
 WARNING => OS parameter vm.swappiness is NOT set to the recommended value
 FAIL =>    ASM_DISKSTRING parameter is either null or set to /dev/*
 WARNING => Linux Disk I/O Scheduler should be configured to Deadline
 WARNING => ADVM Volume(s) should not be configured with redundancy as UNPROT when the underlying ASM Disk Group(s) have a redundancy of MIRROR or HIGH.
 INFO =>    Consider increasing the COREDUMPSIZE size
 FAIL =>    Online (hot) patches are applied to ORACLE_HOME. for mipd09
 WARNING => ASM memory_target is not set to recommended value

Best Practice checking completed.Checking recommended patches on tnc2.
---------------------------------------------------------------------------------

Collecting patch inventory on  CRS HOME /u01/app/11.2.0.4/grid 
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0.4/db_1 
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112040 from /u01/app/11.2.0.4/grid on tnc2
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description                            
---------------------------------------------------------------------------------
22191577  no          no  /u01/app/oracle/product/11.2.0.4/db_1GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112040 from /u01/app/oracle/product/11.2.0.4/db_1 on tnc2
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description                       
---------------------------------------------------------------------------------
22191577  no             merge               GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM 
---------------------------------------------------------------------------------
1              0              0                0              
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME    
---------------------------------------------------------------------------------
 1              0              0                /u01/app/oracle/product/11.2.0.4/db_1
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
                      CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
 WARNING => DNS server config file does not match across database servers
 WARNING => RDBMS software version does not match across cluster.
 WARNING => OS Kernel version(uname -r) does not match across cluster.
 WARNING => Time zone does not match for GI/CRS software owner across cluster
--------------------------------------------------------------------------------
mv: cannot stat `/u01/stage/utl/orachk_tnc1_mipd09_072516_215651/1ACCA8C6291F9C87E0530C98EB0A6EFA.html': No such file or directory
Detailed report (html) - /u01/stage/utl/orachk_tnc1_mipd09_072516_215651/orachk_tnc1_mipd09_072516_215651.html

UPLOAD(if required) - /u01/stage/utl/orachk_tnc1_mipd09_072516_215651.
NG => OS Kernel version(uname -r) does not match across cluster.
 WARNING => Time zone does not match for GI/CRS software owner across cluster
---------------------------------------------------------------------------------
mv: cannot stat `/u01/stage/utl/orachk_tnc1_mipd09_072516_215651/1ACCA8C6291F9C87E0530C98EB0A6EFA.html': No such file or directory
Detailed report (html) - /u01/stage/utl/orachk_tnc1_mipd09_072516_215651/orachk_tnc1_mipd09_072516_215651.html
UPLOAD(if required) - /u01/stage/utl/orachk_tnc1_mipd09_072516_215651.zip

I strongly recommend you to use "ORAchk" in clustered environment 
It will help DBA's to know about potential risks in RAC Cluster and helps us to maintain RAC cluster in much efficient manner :) 

--for more info please go through #
https://blogs.oracle.com/UPGRADE/entry/orachk_12_1_0_4

--Nikhil Tatineni--
--Oracle-In-Memory--