1. Clusterware Administration and Deployment Guide
  2. Oracle Flex Clusters

4 Oracle Flex Clusters

An Oracle Flex Cluster scales Oracle Clusterware to large numbers of nodes.

This chapter includes the following topics:

Overview of Oracle Flex Clusters

Oracle Grid Infrastructure installed in an Oracle Flex Cluster configuration is a scalable, dynamic, robust network of nodes.

Oracle Flex Clusters provide a platform for a variety of applications, including Oracle Real Application Clusters (Oracle RAC) databases with large numbers of nodes. Oracle Flex Clusters also provide a platform for other service deployments that require coordination and automation for high availability.

All nodes in an Oracle Flex Cluster belong to a single Oracle Grid Infrastructure cluster. This architecture centralizes policy decisions for deployment of resources based on application needs, to account for various service levels, loads, failure responses, and recovery.

Oracle Flex Clusters contain Hub Nodes and any number of other supported nodes. The number of Hub Nodes in an Oracle Flex Cluster can be as many as 64. The number of other nodes can be many more. Hub Nodes can host different types of applications.

Hub Nodes are similar to Oracle Grid Infrastructure nodes in an Oracle Clusterware standard Cluster configuration: they are tightly connected, and have direct access to shared storage. Use Hub Nodes to host read-write database instances.

Other supported nodes in an Oracle Flex Cluster are different from standard Oracle Grid Infrastructure nodes, in that they do not require direct access to shared storage, but instead request data through Hub Nodes. Use other nodes to host read-only database instances.

Note:

Read-write and read-only database instances of the same primary database can coexist in an Oracle Flex Cluster.

Hub Nodes can run in an Oracle Flex Cluster configuration without having any other nodes as cluster member nodes, but other nodes must be members of a cluster that includes at least one Hub Node.

Note:

If you upgrade an Oracle Flex Cluster, then Oracle recommends that you upgrade the Hub Nodes first, and that you also have any upgraded Hub Nodes up and running as part of the upgrade process.

Reader Nodes

You can use other nodes to host Oracle RAC database instances that run in read-only mode, which become reader nodes. You can optimize these nodes for parallel query operations by provisioning nodes with a large amount of memory so that data is cached in the hosting node.

A hosting node sends periodic heartbeat messages to its associated Hub Node, which is different from the heartbeat messages that occur between Hub Nodes. During planned shutdown of the Hub Nodes, a hosting node attempts to connect to another Hub Node, unless the hosting node is connected to only one Hub Node. If the Hub Node is evicted, then the hosting node is also evicted from the cluster.

Managing Oracle Flex Clusters

Use CRSCTL to manage Oracle Flex Clusters after successful installation of Oracle Grid Infrastructure for either a small or large cluster.

This section includes the following topics:

Related Topics

Changing the Cluster Mode

You can change the mode of an existing Oracle Clusterware standard Cluster to be an Oracle Flex Cluster.

Note:

  • Changing the cluster mode requires cluster downtime.

  • Oracle does not support changing an Oracle Flex Cluster to an Oracle Clusterware standard Cluster.

  • Oracle Flex Cluster requires Grid Naming Service (GNS).

  • Zone delegation is not required.

Changing an Oracle Clusterware Standard Cluster to an Oracle Flex Cluster

Use CRSCTL to change an existing Oracle Clusterware standard Cluster to an Oracle Flex Cluster.

Perform the following steps:

  1. Run the following command to determine the current mode of the cluster:

    $ crsctl get cluster mode status

  2. Run the following command to ensure that the Grid Naming Service (GNS) is configured with a fixed VIP:

    $ srvctl config gns

    This procedure cannot succeed unless GNS is configured with a fixed VIP. If there is no GNS, then, as root, create one, as follows: 

    # srvctl add gns -vip vip_name | ip_address

    Run the following command as root to start GNS: 

    # srvctl start gns

  3. Use the Oracle Automatic Storage Management Configuration Assistant (ASMCA) to enable Oracle Flex ASM in the cluster before you change the cluster mode.

  4. Run the following command as root to change the mode of the cluster to be an Oracle Flex Cluster: 

    # crsctl set cluster mode flex

  5. Stop Oracle Clusterware by running the following command as root on each node in the cluster: 

    # crsctl stop crs

  6. Start Oracle Clusterware by running the following command as root on each node in the cluster: 

    # crsctl start crs -wait

    Note:

    Use the -wait option to display progress and status messages.

Oracle Extended Clusters

You can extend an Oracle RAC cluster across two, or more, geographically separate sites, each equipped with its own storage. In the event that one of the sites fails, the other site acts as an active standby.

Both Oracle ASM and the Oracle Database stack, in general, are designed to use enterprise-class shared storage in a data center. Fibre Channel technology, however, enables you to distribute compute and storage resources across two or more data centers, and connect them through ethernet cables and Fibre Channel, for compute and storage needs, respectively.

While you can configure Oracle Extended Clusters when you install Oracle Grid Infrastructure, you can also do so post installation using the ConvertToExtended script. You manage your Oracle Extended Cluster using CRSCTL.

Configuring Oracle Extended Clusters

This procedure is only supported for clusters that have been installed with or upgraded to Oracle Grid Infrastructure 12c release 2 (12.2), or later, which are typically configured with one site (default site).

Note:

This procedure requires that all nodes in the cluster be accessible. There will also be a cluster outage during which time database access is disrupted.
You can configure an Oracle Extended Cluster with one or many disk groups and with multiple failure groups. Using the ConvertToExtended script you can create multiple data sites and associate a node with each data site. All Oracle Flex ASM storage remains associated with the default cluster site because there is no mechanism to convert an existing disk group to an extended disk group. After you convert your cluster to an Oracle Extended Cluster, the voting file membership remains flat, and not hierarchical.
You must also add an extended disk group, and migrate the voting files to the extended disk group to take advantage of a site-specific hierarchical voting file algorithm.
Use CRSCTL to query the cluster, as follows, to determine its extended status:
$ crsctl get cluster extended
CRS-6579: The cluster is 'NOT EXTENDED'
$ crsctl query cluster site -all
Site 'crsclus' identified by '7b7b3bef4c1f5ff9ff8765bceb45433a' in state 'ENABLED',
 and contains nodes 'node1,node2,node3,node4', and disks ''.

The preceding example identifies a cluster called crsclus that has four nodes—node1, node2, node3, and node4—and a disk group—datadg. The cluster has one site configured.

Shut down the Oracle Clusterware stack to prevent some Oracle Flex ASM instances, cssd, and crsd from considering the cluster mode is not extended, and other components to consider that it is extended. The advantage to keeping the Grid Plug and Play daemon (gpnpd) online is that the profile gets updated on the remote nodes. When you next start the Oracle Clusterware stack, the cluster mode will be extended.
# crsctl stop cluster -all
  1. On the first node in the cluster, run the following command:
    $ rootcrs.sh -converttoextended -firstnode -sites site_list -site site_name
  2. Run crsctl check css to ensure that Cluster Synchronization Servcies (CSS) is not running on any remote nodes.
  3. Generate the site names and the site GUIDs, using the following command:
    $ crsctl query cluster site -all
    The preceding command displays output similar to the following:
    Site 'SiteA' identified by GUID '7b7b3bef4c1f5ff9ff8765bceb45433a' in state
  'ONLINE' contains nodes 'node1,node2,node3', and disks 'disk1, disk2, disk3'.
Site 'SiteB' identified by GUID '23453bef4c1f5ff9ff8765bceb45433a' in state
  'QUARANTINED' contains nodes 'node4, node5, node6', and disks 'disk4, disk5, disk6'.
Site 'SiteQ' identified by GUID '98763bef4c1f5ff9ff8765bceb45433a' in state
  'ONLINE' contains no nodes and disk 'disk7'.
Site 'SiteD' identified by GUID '4abcd453c6bc6fc1ffd3a58849d5ba41' in state
  'ONLINE' contains nodes 'cuj1234' and no disk

    Update the checkpoint with this information, and copy it to the rest of the nodes in the cluster.

  4. Update the Grid Plug and Play profile and set ext_mode to TRUE using the following commands:
    $ cd $Grid_home/gpnp/host_name/profiles/peer

$ gpnptool getpval -p=profile.xml -prf_sq -o=temp_file // run_gpnptool_getpval()
$ gpnptool edit -p=profile.xml -asm_ext="" prf_sq=seq+1 -o=profile_ext.xml

    Note:

    The value of prf_sq must be greater than what is in the current profile for the ProfileSequence key. 
    $ gpnptool sign -p=profile_ext.xml -o=profile_ext_sign.xml
$ mv profile_ext_sign.xml profile.xml
$ gpnptool put -p=profile.xml
  5. Add sites to the local configuration, as follows:
    $ crsctl add crs site site_name -guid site_guid -local
  6. Add sites to the global configuration, as follows (this requires that OCR be running):
    $ crsctl add crs site site_name -guid site_guid
  7. Update the to-site mapping in the local configuration for this node, as follows:
    $ crsctl modify cluster site site_name -n local_host -local
  8. Update the node-to-site mapping, as follows:
    $ crsctl modify cluster site site_name -n host1, host2, ...
  9. Stop and then start the Oracle High Availability Services stack, as follows:
    # crsctl stop crs
# crsctl start crs
Do the following on the rest of the nodes in the cluster:

  1. Extend the cluster to the specific node, as follows:

    $ rootcrs.sh -converttoextended -site site_name

  2. Ensure that CSS is not running on any remote nodes.

  3. Look up new sites and the site GUIDs using the previous checkpoint information.

  4. Add sites to the local configuration, as follows:

    $ crsctl add crs site site_name -guid site_guid -local

  5. Update the to-site mapping in the local configuration for this node, as follows:

    $ crsctl modify cluster site site_name -n local_host -local

  6. Stop and then start the Oracle High Availability Services stack, as follows:

    # crsctl stop crs
# crsctl start crs
After you finish configuring the Oracle Extended Cluster, run the following command to verify the configuration:
$ crsctl get cluster extended
CRS-XXXX: The cluster is 'EXTENDED'

$ crsctl query cluster site -all
Site 'crsclus' identified by '7b7b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and 
contains nodes '', and disks ''.
Site 'ny' identified by '888b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and \
contains nodes 'node1,node2', and disks ''.
Site 'nj' identified by '999b3bef4c1f5ff9ff8765bceb45433a' is 'ONLINE', and \
contains nodes 'node3,node4', and disks ''.

The output in the preceding command examples is similar to what CRSCTL displays when you run the commands.