If you manually upgrade an Oracle Clusterware 12c (12.1) target that is registered as a Fleet Patching and Provisioning target to a later version, then the result will be an incompatible Fleet Patching and Provisioning Client resource.

1. Stop the Fleet Patching and Provisioning Client resource on the target if it is running.
2. Run the following commands on the Fleet Patching and Provisioning Server:
   Update the Fleet Patching and Provisioning Client credential on the Fleet Patching and Provisioning Server:

   $ rhpctl modify client -client client_name -password

   Export data from the repository to the Fleet Patching and Provisioning Client data file:

   $ rhpctl export client -client client_name -clienetdata file_name.xml

3. Copy the xml file you created in the previous step to a node in the Fleet Patching and Provisioning Client cluster, and run the following command to update the client credential:

   $ srvctl modify rhpclient -clientdata file_name.xml

4. Start the Fleet Patching and Provisioning Client:

   $ srvctl start rhpclient

5. Verify that the Fleet Patching and Provisioning Client started successfully:

   $ srvctl status rhpclient

6. To verify that the Fleet Patching and Provisioning Client is properly registered with the Fleet Patching and Provisioning Server, run the following command and search for the Fleet Patching and Provisioning Client-registered port reported in the command output:

   $ rhpctl query client -client client_cluster

The cluster resource activity log provides precise and specific information about a resource failure, separate from diagnostic logs.

If an Oracle Clusterware-managed resource fails, then Oracle Clusterware logs messages about the failure in the cluster resource activity log located in the Grid Infrastructure Management Repository. Failures can occur as a result of a problem with a resource, a hosting node, or the network. The cluster resource activity log provides a unified view of the cause of resource failure.

Writes to the cluster resource activity log are tagged with an activity ID and any related data gets the same parent activity ID, and is nested under the parent data. For example, if Oracle Clusterware is running and you run the crsctl stop clusterware -all command, then all activities get activity IDs, and related activities are tagged with the same parent activity ID. On each node, the command creates sub-IDs under the parent IDs, and tags each of the respective activities with their corresponding activity ID. Further, each resource on the individual nodes creates sub-IDs based on the parent ID, creating a hierarchy of activity IDs. The hierarchy of activity IDs enables you to analyze the data to find specific activities.

For example, you may have many resources with complicated dependencies among each other, and with a database service. On Friday, you see that all of the resources are running on one node but when you return on Monday, every resource is on a different node, and you want to know why. Using the crsctl query calog command, you can query the cluster resource activity log for all activities involving those resources and the database service. The output provides a complete flow and you can query each sub-ID within the parent service failover ID, and see, specifically, what happened and why.

You can query any number of fields in the cluster resource activity log using filters. For example, you can query all the activities written by specific operating system users such as root. The output produced by the crsctl query calog command can be displayed in either a tabular format or in XML format.

The cluster resource activity log is an adjunct to current Oracle Clusterware logging and alert log messages.

Use the following commands to manage and view the contents of the cluster resource activity log:

Review this content to understand clusterware-specific aspects of how Oracle Clusterware uses ADR.

Oracle Clusterware uses Oracle Database fault diagnosability infrastructure to manage diagnostic data and its alert log. As a result, most diagnostic data resides in the Automatic Diagnostic Repository (ADR), a collection of directories and files located under a base directory that you specify during installation.

Oracle Clusterware ADR data is written under a root directory known as the ADR base. Because components other than ADR use this directory, it may also be referred to as the Oracle base. You specify the file system path to use as the base during Oracle Grid Infrastructure installation and can only be changed if you reinstall the Oracle Grid Infrastructure.

ADR files reside in an ADR home directory. The ADR home for Oracle Clusterware running on a given host always has this structure:

ORACLE_BASE/diag/crs/host_name/crs

In the preceding example, ORACLE_BASE is the Oracle base path you specified when you installed the Oracle Grid Infrastructure and host_name is the name of the host. On a Windows platform, this path uses backslashes (\) to separate directory names.

Under the ADR home are various directories for specific types of ADR data. The directories of greatest interest are incident. The trace directory contains all normal (non-incident) trace files written by Oracle Clusterware daemons and command-line programs as well as the simple text version of the Oracle Clusterware alert log. This organization differs significantly from versions prior to Oracle Clusterware 12c release 1 (12.1.0.2), where diagnostic log files were written under distinct directories per daemon.

To change the log level, edit the ORACLE_BASE/crsdata/host_name/crsdiag/ocrcheck.ini file.

origin_executable_user_name

When an Oracle Clusterware error occurs, run the diagcollection.pl diagnostics collection script to collect diagnostic information from Oracle Clusterware into trace files. The diagnostics provide additional information so My Oracle Support can resolve problems. Run this script as root from the Grid_home/bin directory.

Use the diagcollection.pl script with the following syntax:

diagcollection.pl {--collect [--crs | --acfs | -all] [--chmos [--incidenttime time [--incidentduration time]]]
   [--adr location [--aftertime time [--beforetime time]]]
   [--crshome path | --clean | --coreanalyze}]

Table K-3 lists and describes the parameters used with the diagcollection.pl script.

--collect

--clean

--coreanalyze

A storage split occurs when the private network between two or more disparate sites is available and online, but the storage network has failed.

When Oracle Automatic Storage Management (Oracle ASM) detects a storage split in a typical extended cluster configuration with three sites (two data sites and a quorum site), one of the data sites terminates and quarantines itself and the nodes it contains from the rest of the cluster. If Oracle ASM attempts to start on the quarantined site, then error CRS-2971 occurs.

Resolve the issue, as follows:

1. Resolve the inter-site connectivity issue that resulted in the storage split.
2. Ensure that all Oracle ASM disk groups are mounted on the site that is not quarantined, as follows:

   SELECT group_number, name, state FROM v$asm_diskgroup_stat;

3. Obtain a list of online disks belonging to these disk groups by running the following command for each disk group:

   SELECT path FROM v$asm_disk_stat WHERE group_number=group_number AND state = 'NORMAL' AND mode_status = 'ONLINE';
   

4. For each of the paths from you obtained in the previous step, ensure that the disk is accessible from the quarantined site, as follows:

   asmcmd lsdsk -I --member 'path'

5. If the preceding verification succeeds, then rejuvenate the quarantined site, as follows:

   crsctl modify cluster site site_name -s rejuvenate

During an upgrade, while running the Oracle Clusterware root.sh script, you may see the following messages:

• ACFS-9427 Failed to unload ADVM/ACFS drivers. A system restart is recommended.

• ACFS-9428 Failed to load ADVM/ACFS drivers. A system restart is recommended.

If you see these error messages during the upgrade of the initial (first) node, then do the following:

1. Complete the upgrade of all other nodes in the cluster.

2. Restart the initial node.

3. Run the root.sh script on initial node again.

4. Run the Grid_home/gridSetup -executeConfigTools script as the user that installed Oracle Grid Infrastructure to complete the upgrade.

For nodes other than the initial node (the node on which you started the installation):

1. Restart the node where the error occurs.

2. Run the orainstRoot.sh script as root on the node where the error occurs.

3. Change directory to the Grid home, and run the root.sh script on the node where the error occurs.

Use the following procedure to test zone delegation:

1. Start the GNS VIP by running the following command as root:

   # crsctl start ip -A IP_name/netmask/interface_name
   

   The interface_name should be the public interface and netmask of the public network.

2. Start the test DNS server on the GNS VIP by running the following command (you must run this command as root if the port number is less than 1024):

   # crsctl start testdns -address address [-port port]
   

   This command starts the test DNS server to listen for DNS forwarded packets at the specified IP and port.

3. Ensure that the GNS VIP is reachable from other nodes by running the following command as root:

   crsctl status ip -A IP_name
   

4. Query the DNS server directly by running the following command:

   crsctl query dns -name name -dnsserver DNS_server_address
   

   This command fails with the following error:

   CRS-10023: Domain name look up for name asdf.foo.com failed. Operating system error: Host name lookup failure

   Look at Grid_home/log/host_name/client/odnsd_*.log to see if the query was received by the test DNS server. This validates that the DNS queries are not being blocked by a firewall.

5. Query the DNS delegation of GNS domain queries by running the following command:

   crsctl query dns -name name

   Note:

   The only difference between this step and the previous step is that you are not giving the -dnsserver DNS_server_address option. This causes the command to query name servers configured in /etc/resolv.conf. As in the previous step, the command fails with same error. Again, look at odnsd*.log to ensure that odnsd received the queries. If step 5 succeeds but step 6 does not, then you must check the DNS configuration.

6. Stop the test DNS server by running the following command:

   crsctl stop testdns -address address
   

7. Stop the GNS VIP by running the following command as root:

   crsctl stop ip -A IP_name/netmask/interface_name

Oracle Clusterware writes messages to the ADR alert log file (as previously described) for various important events. Alert log messages generally are localized (translated) and carry a message identifier that can be used to look up additional information about the message. The alert log is the first place to look if there appears to be problems with Oracle Clusterware.

The following is an example of alert log messages from two different CRS daemon processes:

2014-07-16 00:27:22.074 [CTSSD(12817)]CRS-2403:The Cluster Time Synchronization Service on host stnsp014 is in observer mode.
2014-07-16 00:27:22.146 [CTSSD(12817)]CRS-2407:The new Cluster Time Synchronization Service reference node is host stnsp013.
2014-07-16 00:27:22.753 [CTSSD(12817)]CRS-2401:The Cluster Time Synchronization Service started on host stnsp014.
2014-07-16 00:27:43.754 [CRSD(12975)]CRS-1012:The OCR service started on node stnsp014.
2014-07-16 00:27:46.339 [CRSD(12975)]CRS-1201:CRSD started on node stnsp014.

The following example shows the start of the Oracle Cluster Time Synchronization Service (OCTSS) after a cluster reconfiguration:

2014-07-15 23:51:17.532 [CTSSD(12813)]CRS-2403:The Cluster Time Synchronization Service on host stnsp014 is in observer mode.
2014-07-15 23:51:18.292 [CTSSD(12813)]CRS-2407:The new Cluster Time Synchronization Service reference node is host stnsp013.
2014-07-15 23:51:18.961 [CTSSD(12813)]CRS-2401:The Cluster Time Synchronization Service started on host stnsp014.