A capture process is an optional Oracle background process that scans the database redo log to capture DML and DDL changes made to database objects.

The primary function of the redo log is to record all of the changes made to the database. A capture process captures database changes from the redo log, and the database where the changes were generated is called the source database for the capture process.

When a capture process captures a database change, it converts it into a specific message format called a logical change record (LCR). In an XStream Out configuration, the capture process sends these LCRs to an outbound server.

Figure 3-1 shows a capture process.

Figure 3-1 Capture Process

Description of "Figure 3-1 Capture Process"

A capture process can run on its source database or on a remote database. When a capture process runs on its source database, the capture process is a local capture process.

You can also capture changes for the source database by running the capture process on different server. When a capture process runs on a remote database, the capture process is called a downstream capture process, and the remote database is called the downstream database. The log files are written to the remote database and to the source database. In this configuration, the source logfiles must be available at the downstream capture database. The capture process on the remote database mines the logs from the source database and stages them locally. This configuration can be helpful when you want to offload the processing of capture changes from a production database to different, remote database.

A capture process can capture changes made to columns of most data types.

When capturing the row changes resulting from DML changes made to tables, a capture process can capture changes made to columns of the following data types:

• VARCHAR2

• NVARCHAR2

• NUMBER

• FLOAT

• LONG

• DATE

• BINARY_FLOAT

• BINARY_DOUBLE

• TIMESTAMP

• TIMESTAMP WITH TIME ZONE

• TIMESTAMP WITH LOCAL TIME ZONE

• INTERVAL YEAR TO MONTH

• INTERVAL DAY TO SECOND

• RAW

• LONG RAW

• UROWID

• CHAR

• NCHAR

• CLOB with BASICFILE or SECUREFILE storage

• NCLOB with BASICFILE or SECUREFILE storage

• BLOB with BASICFILE or SECUREFILE storage

• XMLType stored as CLOB, object relational, or as binary XML

• Object types

• Varrays

• REF data types

• The following Oracle-supplied types: ANYDATA, SDO_GEOMETRY, and media types

• BFILE

If XStream is replicating data for an object type, then the replication must be unidirectional, and all replication sites must agree on the names and data types of the attributes in the object type. You establish the names and data types of the attributes when you create the object type. For example, consider the following object type:

CREATE TYPE cust_address_typ AS OBJECT
     (street_address     VARCHAR2(40), 
      postal_code        VARCHAR2(10), 
      city               VARCHAR2(30), 
      state_province     VARCHAR2(10), 
      country_id         CHAR(2));
/

At all replication sites, street_address must be VARCHAR2(40), postal_code must be VARCHAR2(10), city must be VARCHAR2(30), and so on.

• ID Key LCRs
  An ID key LCR is a special type of row LCR. ID key LCRs enable an XStream client application to process changes to rows that include unsupported data types.
• ID Key LCRs Demo
  A demo is available that creates a sample client application that processes ID key LCRs.

A capture process can capture different types of DML changes.

When you specify that DML changes made to certain tables should be captured, a capture process captures the following types of DML changes made to these tables:

• INSERT

• UPDATE

• DELETE

• MERGE

• Piecewise operations

A capture process converts each MERGE change into an INSERT or UPDATE change. MERGE is not a valid command type in a row LCR.

You can configure a capture process to run locally on a source database or remotely on a downstream database.

A single database can have one or more capture processes that capture local changes and other capture processes that capture changes from a remote source database. That is, you can configure a single database to perform both local capture and downstream capture.

• Local Capture
  Local capture means that a capture process runs on the source database.
• Downstream Capture
  Downstream capture means that a capture process runs on a database other than the source database.

Enabling and disabling restricted session affects capture processes.

When you enable restricted session during system startup by issuing a STARTUP RESTRICT statement, capture processes do not start, even if they were running when the database shut down. When restricted session is disabled with an ALTER SYSTEM statement, each capture process that was running when the database shut down is started.

When restricted session is enabled in a running database by the SQL statement ALTER SYSTEM ENABLE RESTRICTED SESSION clause, it does not affect any running capture processes. These capture processes continue to run and capture changes. If a stopped capture process is started in a restricted session, then the capture process does not actually start until the restricted session is disabled.

The capture process subcomponents are a reader server, one or more preparer servers, and a builder server.

A capture process is an optional Oracle background process whose process name is CPnn, where nn can include letters and numbers. A capture process captures changes from the redo log by using the infrastructure of LogMiner. XStream configures LogMiner automatically. You can create, alter, start, stop, and drop a capture process, and you can define capture process rules that control which changes a capture process captures.

The parallelism capture process parameter controls capture process parallelism. When capture process parallelism is 0 (zero), the default for XStream Out, the capture process does not use subcomponents to perform its work. Instead, the CPnn process completes all of the tasks required to capture database changes.

When capture process parallelism is greater than 0, the capture process uses the underlying LogMiner process name is MSnn, where nn can include letters and numbers. When capture process parallelism is 0 (zero), the capture process does not use this process.

When capture process parallelism is greater than 0, the capture process consists of the following subcomponents:

• One reader server that reads the redo log and divides the redo log into regions.

• One or more preparer servers that scan the regions defined by the reader server in parallel and perform prefiltering of changes found in the redo log. Prefiltering involves sending partial information about changes, such as schema and object name for a change, to the rules engine for evaluation, and receiving the results of the evaluation. You can control the number of preparer servers using the parallelism capture process parameter.

• One builder server that merges redo records from the preparer servers. These redo records either evaluated to TRUE during partial evaluation or partial evaluation was inconclusive for them. The builder server preserves the system change number (SCN) order of these redo records and passes the merged redo records to the capture process.

• The capture process (CPnn) performs the following actions for each change when it receives merged redo records from the builder server:

  • Formats the change into an LCR

  • If the partial evaluation performed by a preparer server was inconclusive for the change in the LCR, then sends the LCR to the rules engine for full evaluation

  • Receives the results of the full evaluation of the LCR if it was performed

  • Discards the LCR if it satisfies the rules in the negative rule set for the capture process or if it does not satisfy the rules in the positive rule set

  • Enqueues the LCR into the queue associated with the capture process if the LCR satisfies the rules in the positive rule set for the capture process

Each reader server, preparer server, and builder server is a process.

The state of a capture process describes what the capture process is doing currently.

You can view the state of a capture process by querying the STATE column in the V$XSTREAM_CAPTURE dynamic performance view.

Capture process parameters control the way a capture process operates.

For example, the parallelism capture process parameter controls the number of preparer servers used by a capture process, and the time_limit capture process parameter specifies the amount of time a capture process runs before it is shut down automatically. You set capture process parameters using the DBMS_CAPTURE_ADM.SET_PARAMETER procedure. After creation, a capture process is disabled so that you can set the capture process parameters for your environment before starting it for the first time.

A checkpoint is information about the current state of a capture process that is stored persistently in the data dictionary of the database running the capture process.

A capture process tries to record a checkpoint at regular intervals called checkpoint intervals.

• Required Checkpoint SCN
  The system change number (SCN) that corresponds to the lowest checkpoint for which a capture process requires redo data is the required checkpoint SCN.
• Maximum Checkpoint SCN
  The SCN that corresponds to the last physical checkpoint recorded by a capture process is the maximum checkpoint SCN.
• Checkpoint Retention Time
  The checkpoint retention time is the amount of time, in number of days, that a capture process retains checkpoints before purging them automatically.

Specific system change number (SCN) values are important for a capture process.

You can query the ALL_CAPTURE data dictionary view to display these values for one or more capture processes.

• Captured SCN and Applied SCN
  The captured SCN is the SCN that corresponds to the most recent change scanned in the redo log by a capture process. The applied SCN for a capture process is the SCN of the most recent LCR processed by the relevant outbound server.
• First SCN and Start SCN
  The first SCN and start SCN are important for a capture process.

An outbound server is an optional Oracle background process that sends database changes to a client application.

Specifically, a client application can attach to an outbound server and extract database changes from LCRs. A client application attaches to the outbound server using OCI or Java interfaces.

A client application can create multiple sessions. Each session can attach to only one outbound server, and each outbound server can serve only one session at a time. However, different client application sessions can connect to different outbound servers or inbound servers.

Change capture can be performed on the same database as the outbound server or on a different database. When change capture is performed on a different database from the one that contains the outbound server, a propagation sends the changes from the change capture database to the outbound server database. Downstream capture is also a supported mode to reduce the load on the source database.

When both the outbound server and its capture process are enabled, data changes, encapsulated in row LCRs and DDL LCRs, are sent to the outbound server. The outbound server can publish LCRs in various formats, such as OCI and Java. The client application can process LCRs that are passed to it from the outbound server or wait for LCRs from the outbound server by using a loop.

An outbound server sends LOB, LONG, LONG RAW, and XMLType data to the client application in chunks. Several chunks comprise a single column value of LOB, LONG, LONG RAW, or XMLType data type.

Figure 3-4 shows an outbound server configuration.

Figure 3-4 XStream Out Outbound Server

Description of "Figure 3-4 XStream Out Outbound Server"

The client application can detach from the outbound server whenever necessary. When the client application re-attaches, the outbound server automatically determines where in the stream of LCRs the client application was when it detached. The outbound server starts sending LCRs from this point forward.

Outbound servers support all of the data types that are supported by capture processes.

Outbound servers can send LCRs that include changes to columns of these data types to XStream client applications.

The apply user for an outbound server is the user who receives LCRs from the outbound server's capture process.

The apply user for an outbound server must match the capture user for the outbound server's capture process.

Enabling and disabling restricted session affects outbound servers.

When restricted session is enabled during system startup by issuing a STARTUP RESTRICT statement, outbound servers do not start, even if they were running when the database shut down. When the restricted session is disabled, each outbound server that was not stopped is started.

When restricted session is enabled in a running database by the SQL statement ALTER SYSTEM ENABLE RESTRICTED SESSION, it does not affect any running outbound servers. These outbound servers continue to run and send LCRs to an XStream client application. If a stopped outbound server is started in a restricted session, then the outbound server does not actually start until the restricted session is disabled.

An outbound server consists of a reader server, a coordinator process, and an apply server.

• A reader server that receives LCRs from the outbound server's capture process. The reader server is a process that computes dependencies between LCRs and assembles LCRs into transactions. The reader server then returns the assembled transactions to the coordinator process.

  You can view the state of the reader server for an outbound server by querying the V$XSTREAM_APPLY_READER dynamic performance view.

• A coordinator process that gets transactions from the reader server and passes them to apply servers. The coordinator process name is APnn, where nn can include letters and numbers. The coordinator process is an Oracle background process.

  You can view the state of a coordinator process by querying the V$XSTREAM_APPLY_COORDINATOR dynamic performance view.

• An apply server that sends LCRs to an XStream client application. The apply server is a process. If the apply server encounters an error, then it then it records information about the error in the ALL_APPLY view.

  You can view the state of the apply server for an outbound server by querying the V$XSTREAM_APPLY_SERVER dynamic performance view.

The reader server and the apply server process names are ASnn, where nn can include letters and numbers.

There are several considerations for XStream outbound servers.

The following are considerations for outbound servers:

• LCRs processed by an outbound server must be LCRs that were captured by a capture process. An outbound server does not support LCRs that were constructed by applications.

• A single outbound server can process captured LCRs from only one source database. The source database is the database where the changes encapsulated in the LCRs were generated in the redo log.

• The source database for the changes captured by a capture process must be at 10.2.0 or higher compatibility level for these changes to be processed by an outbound server.

• The capture process for an outbound server must be running on an Oracle Database 11g Release 2 (11.2) or later database.

• A single capture process cannot capture changes for both an outbound server and an apply process. However, a single capture process can capture changes for multiple outbound servers.

• Automatic split and merge of a stream is possible when the capture process and the outbound server for the stream run on different databases. However, when the capture process and outbound server for a stream run on the same database, automatic split and merge of the stream is not possible.

• An outbound server's LCRs can spill from memory to hard disk if they have been in the buffered queue for a period of time without being processed, if there are a large number of LCRs in a transaction, or if there is not enough space in memory to hold all of the LCRs. An outbound server performs best when a minimum of LCRs spill from memory. You can control an outbound server's behavior regarding spilled LCRs using the txn_age_spill_threshold and txn_lcr_spill_threshold apply parameters.

• Instantiation SCNs are not required for database objects processed by an outbound server. If an instantiation SCN is set for a database object, then the outbound server only sends the LCRs for the database object with SCN values that are greater than the instantiation SCN value. If a database object does not have an instantiation SCN set, then the outbound server skips the instantiation SCN check and sends all LCRs for that database object. In both cases, the outbound server only sends LCRs that satisfy its rule sets.

Apply parameters control the behavior of outbound servers.

You can use the following apply parameters with outbound servers:

• apply_sequence_nextval

• disable_on_limit

• grouptransops

• ignore_transaction

• max_sga_size

• maximum_scn

• startup_seconds

• time_limit

• trace_level

• transaction_limit

• txn_age_spill_threshold

• txn_lcr_spill_threshold

• write_alert_log

In LCRs that were captured by a capture process, there is additional information related to LCR position.

LCRs that were captured by a capture process contain the following additional attributes related to LCR position:

• The scn_from_position attribute contains the SCN of the LCR.

• The commit_scn_from_position attribute contains the commit SCN of the transaction to which the LCR belongs.

The processed low position is a position below which all transactions have been processed by the client application.

If the outbound server or the client application stops abnormally, then the connection between the two is broken automatically. In this case, the client application must roll back all incomplete transactions.

The client application must maintain its processed low position to recover properly after either it or the outbound server (or both) are restarted. The processed low position indicates that the client application has processed all LCRs that are less than or equal to this value. The client application can update the processed low position for each transaction that it consumes.

When the client application attaches to the outbound server, the following conditions related to the processed low position are possible:

• The client application can pass a processed low position to the outbound server that is equal to or greater than the outbound server's processed low position. In this case, the outbound server resumes streaming LCRs from the first LCR that has a position greater than the client application's processed low position.

• The client application can pass a processed low position to the outbound server that is less than the outbound server's processed low position. In this case, the outbound server raises an error.

• The client application can pass NULL to the outbound server. In this case, the outbound server determines the processed low position automatically and starts streaming LCRs from the LCR that has a position greater than this processed low position. When this happens, the client application must suppress or discard each LCR with a position less than or equal to the client application's processed low position.

To minimize network latency, the outbound server streams LCRs to the client application with time-based acknowledgments. For example, the outbound server might send an acknowledgment every 30 seconds.

This streaming protocol fully utilizes the available network bandwidth, and the performance is unaffected by the presence of a wide area network (WAN) separating the sender and the receiver. The outbound server extends the underlying Oracle Replication infrastructure, and the outbound server maintains the streaming performance rate.

Using OCI, you can control the time period of the interval by setting the OCI_ATTR_XSTREAM_ACK_INTERVAL attribute through the OCI client application. The default is 30 seconds.

Using Java, you can control the time period of the interval by setting the batchInterval parameter in the attach method in the XStreamOut class. The client application can specify this interval when it invokes the attach method.

If the interval is large, then the outbound server can stream out more LCRs for each acknowledgment interval. However, a longer interval delays how often the client application can send the processed low position to the outbound server. Therefore, a longer interval might mean that the processed low position maintained by the outbound server is not current. In this case, when the outbound server restarts, it must start processing LCRs at an earlier position than the one that corresponds to the processed low position maintained by the client application. Therefore, more LCRs might be retransmitted, and the client application must discard the ones that have been applied.

There are security considerations for the client application because XStream Out allows it to receive LCRs.

After an XStream Out application receives LCRs, the application might save the contents of LCRs to a file or generate the SQL statements to execute the LCRs on a non-Oracle database.

Java and OCI client applications must connect to an Oracle database before attaching to an XStream outbound server created on that database. The connected user must be the same as the connect_user configured for the outbound server. Otherwise, an error is raised. XStream does not assume that the connected user to the outbound server is trusted.

The XStream Java layer API relies on Oracle JDBC security because XStream accepts the Oracle JDBC connection instance created by client application in the XStream attach method in the XStreamOut class. The connected user is validated as an XStream user.

All the components of the XStream Out configuration run as the same XStream administrator. This user can be either a trusted user with a high level of privileges, or it can be an untrusted user that has only the privileges necessary for performing certain tasks.

The security model of the XStream administrator also determines the data dictionary views that this user can query to monitor the XStream configuration. The trusted administrator can monitor XStream with DBA_ views. The untrusted administrator can monitor XStream with ALL_ views.

You create an XStream administrator using the GRANT_ADMIN_PRIVILEGE procedure in the DBMS_XSTREAM_AUTH package. When you run this procedure to create an XStream administrator for XStream Out, the privilege_type parameter determines the type of privileges granted to the user:

• Specify CAPTURE for the privilege_type parameter if the XStream administrator manages only an XStream Out configuration on the database.

• Specify * for the privilege_type parameter if the XStream administrator manages both an XStream Out and an XStream In configuration on the database.

The GRANT_ADMIN_PRIVILEGE procedure grants privileges for Oracle-supplied views and packages that are required to run components in an XStream Out or XStream In configuration. This procedure does not grant privileges on database objects owned by users. If such privileges are required, then they must be granted separately.

Changes are captured in the security domain of the capture user for a capture process. The capture user captures all changes that satisfy the capture process rule sets. The capture user must have the necessary privileges to perform these actions.

An outbound server sends LCRs to an XStream client application in the security domain of its connect user.

• Capture Processes and Oracle Real Application Clusters
  A capture process can capture changes in an Oracle Real Application Clusters (Oracle RAC) environment.
• Queues and Oracle Real Application Clusters
  You can configure queues in an Oracle Real Application Clusters (Oracle RAC) environment.
• Propagations and Oracle Real Application Clusters
  A propagation can propagate LCRs from one queue to another in an Oracle Real Application Clusters (Oracle RAC) environment. A propagation job running on an instance propagates logical change records (LCRs) from any queue owned by that instance to destination queues.
• Outbound Servers and Oracle Real Application Clusters
  You can configure an outbound server in an Oracle Real Application Clusters (Oracle RAC) environment provided you have set use_rac_service to Y in the procedure DBMS_CAPTURE_ADM.SET_PARAMETER.

XStream Out can work with Transparent Data Encryption.

• Capture Processes and Transparent Data Encryption
  Capture processes can capture changes to columns that have been encrypted using Transparent Data Encryption.
• Propagations and Transparent Data Encryption
  A propagation can propagate row logical change records (row LCRs) that contain changes to columns that were encrypted using Transparent Data Encryption.
• Outbound Servers and Transparent Data Encryption
  An outbound server can process implicitly captured row logical change records (row LCRs) that contain columns encrypted using Transparent Data Encryption.

XStream Out supports tables in a flashback data archive.

Capture processes can capture data manipulation language (DML) and data definition language (DDL) changes made to these tables. Outbound servers can process the captured LCRs.

XStream Out also support the following DDL statements:

• CREATE FLASHBACK ARCHIVE

• ALTER FLASHBACK ARCHIVE

• DROP FLASHBACK ARCHIVE

• CREATE TABLE with a FLASHBACK ARCHIVE clause

• ALTER TABLE with a FLASHBACK ARCHIVE clause

RMAN deletion policies can affect capture processes.

Some RMAN deletion policies and commands delete archived redo log files. If one of these RMAN policies or commands is used on a database that generates redo log files for one or more capture processes, then ensure that the RMAN commands do not delete archived redo log files that are required by a capture process.

• RMAN and Local Capture Processes
  When a local capture process is configured, RMAN does not delete archived redo log files that are required by the local capture process unless there is space pressure in the fast recovery area.
• RMAN and Downstream Capture Processes
  When a downstream capture process captures database changes made at a source database, ensure that no RMAN deletion policy or command deletes an archived redo log file until after it is transferred from the source database to the downstream capture process database.

XStream Out supports distributed transactions.

You can perform distributed transactions using either of the following methods:

• Modify tables in multiple databases in a coordinated manner using database links.

• Use the XA interface, as exposed by the DBMS_XA supplied PL/SQL package or by the OCI or JDBC libraries. The XA interface implements X/Open Distributed Transaction Processing (DTP) architecture.

A capture process captures changes made to the source database during a distributed transaction using either of these two methods and sends the changes to an outbound server. An outbound server sends the changes in a transaction to a client application after the transaction has committed.

However, the distributed transaction state is not sent. The client application does not inherit the in-doubt or prepared state of such a transaction. Also, the outbound server does not send the changes using the same global transaction identifier used at the source database for XA transactions.

XA transactions can be performed in two ways:

• Tightly coupled, where different XA branches share locks

• Loosely coupled, where different XA branches do not share locks

XStream Out supports changes made by loosely coupled XA branches regardless of the COMPATIBLE initialization parameter value. XStream Out supports replication of changes made by tightly coupled branches on an Oracle RAC source database only if the COMPATIBLE initialization parameter set to 11.2.0.0 or higher.

A multitenant environment enables a database to contain a portable set of schemas, objects, and related structures that appears logically to an application as a separate database.

This self-contained collection is called a pluggable database (PDB). A multitenant container database (CDB) contains PDBs. In a CDB, XStream Out functions much the same as it does in a non-CDB.

A CDB can also contain application containers. An application container is an optional component of a CDB that consists of an application root and all application PDBs associated with it. An application container stores data for one or more applications. An application container shares application metadata and common data. In a CDB, each of the following is a container: the CDB root, each PDB, each application root, and each application PDB.

The main differences in the way XStream Out functions in a CDB and non-CDB are:

• XStream Out must be configured only in the CDB root.

• XStream Out can see changes made to any container within the CDB.

• XStream Out capture rules can limit the LCRs to those that are needed for the client application. The system-generated capture rules select the appropriate LCRs based on the parameters that were passed to the ADD_OUTBOUND and CREATE_OUTBOUND procedures in the DBMS_XSTREAM_ADM package. You can use the ADD_*_RULES procedures in the same package for more fine-grained control over the rules used by the XStream Out components.

• The user who performs XStream Out tasks must be a common user.