54 DBMS_DBFS_HS
The Oracle Database File System Hierarchical Store is implemented in the DBMS_DBFS_HS
package. This package provides users the ability to use tape or Amazon S3 Web service as a storage tier when doing Information Lifecycle Management for their database tables.
This chapter contains the following topics:
54.1 DBMS_DBFS_HS Overview
The DBMS_DBFS_HS
package is a service provider underneath the DBMS_DBFS_CONTENT
package that enables use of tape or Amazon S3 Web service as storage for data.
The data on tape or Amazon S3 Web service is part of the Oracle Database and can be accessed through all standard interfaces, but only through the database. The package allows users to use tape or Amazon S3 Web service as a storage tier when doing Information Lifecycle Management of their content.
The package initially stores all content files in level-1 cache. As the level-1 cache fills up, content files are moved to level-2 cache and then to an external storage device using bulk writes.
54.3 DBMS_DBFS_HS Constants
The DBMS_DBFS_HS
package uses the constants shown in the tables in this topic.
Table 54-1 DBMS_DBFS_HS Constants - Used by the CREATESTORE Procedure
Constant | Type | Value | Description |
---|---|---|---|
|
|
|
Use tape as a storage tier |
|
|
|
Use Amazon S3 Web service as a storage tier |
Table 54-2 DBMS_DBFS_HS Constants - Used by the SETSTOREPROPERTY Procedure and the GETSTOREPROPERTY Function
Constant | Type | Value | Description |
---|---|---|---|
|
|
|
Specifies the AWS bucket to be used as a storage tier by the Hierarchical Store. Restrictions on bucket name are: 1) Bucket names can only contain lowercase letters, numbers, periods ( 2) Bucket names must start with a number or letter. 3) Bucket names cannot be in an IP address style ( 4) Bucket names must be between 3 and 63 characters long. 5) Bucket names should not end with a dash. 6) Dashes cannot appear next to periods. For example, |
|
|
|
Specifies the cumulative cache size used for the Hierarchical Store. This property is set by the CREATESTORE Procedure and can be modified by the RECONFIGCACHE Procedure. It cannot be modified by the SETSTOREPROPERTY Procedure, though its value can be queried by the GETSTOREPROPERTY Function. |
|
|
|
Use to enable compression of files stored in the DBFS hierarchical store. It specifies the compression level to be used for compressing the files |
|
|
|
If this property is set to ' |
|
|
|
Specifies the DNS name of the HTTP proxy, if any, that is needed to access the Amazon S3 storage service |
|
|
|
Specifies the license ID associated with the library |
|
|
|
Specifies fraction of the This property cannot be modified by the SETSTOREPROPERTY Procedurethough its value can be queried by the GETSTOREPROPERTY Function. Its value is set by CREATESTORE Procedure and can be modified by the RECONFIGCACHE Procedure. |
|
|
|
Specifies the media pool number to use for storing the content |
|
|
|
Indicates no compression |
|
|
|
Use to set the compression level to |
|
|
|
Use to set the compression level to |
|
|
|
Use to set the compression level to |
|
|
|
Specifies Multiple content files are bundled together into one archive file and then the archive file is transferred to tape or Amazon S3. This is because creating one file on tape or Amazon S3 for every content file in the store is a prohibitively expensive operation. This property cannot be modified by the SETSTOREPROPERTY Procedurethough its value can be queried by the GETSTOREPROPERTY Function. Its value is set by CREATESTORE Procedure and can be modified by the RECONFIGCACHE Procedure. |
|
|
|
Specifies the size used by the SBT protocol to transfer data from tape or S3.This chunk is allocated in memory per transaction for retrieval of content files from an archive store, so the value of this property should be conservative. The default size of 1MB is typically good for most users. |
|
|
|
Specifies the |
|
|
|
Specifies the path of the shared library used by RMAN to communicate with Amazon S3. It is named |
|
|
|
Indicates whether buffer-based |
|
|
|
The value of this property should be of the form:
Defines the Oracle Wallet which contains the credentials of the Amazon S3 account associated with the store under consideration.
The format of
In UNIX or Linux it is, for example:
When the package is executed in the Oracle database server, the wallet is accessed from the database server.
|
|
|
|
Specifies the size used by the SBT protocol to transfer data to tape or S3. This chunk is allocated in memory per transaction for The default size of 1MB is typically good for most users. |
Table 54-3 DBMS_DBFS_HS Constants - Used by the REGISTERSTORECOMMAND Function
Constant | Type | Value | Description |
---|---|---|---|
|
|
|
Specified operation must be performed before writing a |
|
|
|
Specified operation must be performed before a retrieval operation such as reading a |
Table 54-4 DBMS_DBFS_HS Constants - Failure/Success/Error
Constant | Type | Value | Description |
---|---|---|---|
|
|
|
Procedure or function did not execute successfully |
|
|
|
Procedure or function completed successfully |
|
|
|
Procedure or function returned an error |
54.4 DBMS_DBFS_HS Operational Notes
When the DBMS_DBFS_HS
package is executed in the Oracle database server, the wallet is accessed from the database server.
54.5 Summary of DBMS_DBFS_HS Subprograms
This table lists and describes the subprograms in the DBMS_DBFS_HS
package.
Table 54-5 DBMS_DBFS_HS Package Subprograms
Subprogram | Description |
---|---|
Removes files created on the external storage device that hold no currently used data |
|
Creates an AWS bucket, associated with a store of type |
|
Creates a new hierarchical store |
|
Removes a command that had been previously associated with a store through the RECONFIGCACHE Procedure |
|
Deletes a previously created hierarchical store |
|
Flushes (writes out) dirty contents from the level-1 cache. |
|
Retrieves the values of a property of a store |
|
Reconfigures the parameters of the database cache used by the store |
|
Registers commands for a store with the Hierarchical Store to be sent to the Media Manager for the external storage device associated with the store |
|
Sends a command to be executed on the external storage device's Media Manager |
|
Stores properties of a store in the database |
|
Pushes locally staged data to the remote storage |
54.5.1 CLEANUPUNUSEDBACKUPFILES Procedure
This procedure removes files created on the external storage device that hold no currently used data in them.
Syntax
DBMS_DBFS_HS.CLEANUPUNUSEDBACKUPFILES ( store_name IN VARCHAR2);
Parameters
Table 54-6 CLEANUPUNUSEDBACKUPFILES Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
Usage Notes
-
The action of removing files from external storage device can not be rolled back.
-
This method can be executed periodically to clear space on the external storage device. Asynchronously deleting content from the external storage device is useful because it has minimal impact on the OLTP performance. The periodic scheduling can be accomplished using the DBMS_SCHEDULER package.
54.5.2 CREATEBUCKET Procedure
This procedure creates an AWS bucket, associated with a store of type STORETYPE_AMAZONS3
into which the Hierarchical Store can then move data.
Syntax
DBMS_DBFS_HS.CREATEBUCKET ( store_name IN VARCHAR2);
Parameters
Table 54-7 CREATEBUCKET Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
Usage Notes
-
The
PROPNAME_BUCKET
property of the store should be set before this subprogram is called. -
Once this procedure has successfully created a bucket in Amazon S3, the bucket can only be deleted using out-of-band methods, such as logging-in to S3 and deleting data (directories, files, and other items) for the bucket.
54.5.3 CREATESTORE Procedure
This procedure creates a new hierarchical store store_name
of type STORE_TYPE
(STORETYPE_TAPE
or STORETYPE_AMAZONS3
) in schema schema_name
(defaulting to the current schema) under the ownership of the invoking session user.
Syntax
DBMS_DBFS_HS.CREATESTORE ( store_name IN VARCHAR2, store_type IN VARCHAR2, tbl_name IN VARCHAR2, tbs_name IN VARCHAR2, cache_size IN NUMBER, lob_cache_quota IN NUMBER DEFAULT NULL, optimal_tarball_size IN NUMBER DEFAULT NULL, schema_name IN VARCHAR2 DEFAULT NULL);
Parameters
Table 54-8 CREATESTORE Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
|
|
|
Table for store entries |
|
Tablespace for the store |
|
Amount of space used by the store to cache content in given tablespace |
|
Fraction of the cache_size which is allocated for level 1 cache. The default value of this parameter is |
|
Maximum possible size of the archive file. Multiple content files are bundled together into one archive file, and then the archive file is transferred to tape or Amazon S3. This is because creating one file on tape or Amazon S3 for every content file in the store is a prohibitively expensive operation. The value of is set by default to 10GB for tape and to 100MB for Amazon S3. |
|
Schema for the store |
Usage Notes
CREATESTORE()
sets certain properties of the store to default values. The user can use the methods SETSTOREPROPERTY()
and RECONFIGCACHE()
to appropriately change the property values and to set other properties of the store.
-
Store names must be unique for an owner. The same store names can be used for different stores owned by different owners.
-
Once a table space has been specified to store the store's content in a database, it cannot be changed later.
-
This subprogram will execute like a DDL statement, performing an automatic
COMMIT
before and after execution. -
Stores using
DBMS_DBFS_HS
must not use singleton mount. This means that the singleton parameter should beFALSE
and thestore_mount
parameter should have a non-NULL
value in a call to theDBMS_DBFS_CONTENT
.MOUNTSTORE
procedure.
54.5.4 DEREGSTORECOMMAND Function
This procedure removes a command that had been previously associated with a store through the REGISTERSTORECOMMAND Procedure.
Syntax
DBMS_DBFS_HS.DEREGSTORECOMMAND ( store_name IN VARCHAR2, message IN VARCHAR2);
Parameters
Table 54-9 DEREGSTORECOMMAND Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
|
Message to be deregistered |
Usage Notes
If this subprogram successfully executes, its actions cannot be rolled back by the user. If the user wants to restore the previous state, the user must call the REGISTERSTORECOMMAND Procedure.
Related Topics
54.5.5 DROPSTORE Procedure
This procedure deletes a previously created hierarchical store specified by name and owned by the invoking session user.
Syntax
DBMS_DBFS_HS.DROPSTORE ( store_name IN VARCHAR2, opt_flags IN INTEGER DEFAULT 0);
Parameters
Table 54-10 DROPSTORE Procedure Parameters
Parameter | Description |
---|---|
|
Name of store owned by the invoking session user |
|
User can specify optional flags. If |
Usage Notes
-
The procedure executes like a DDL in that it auto-commits before and after its execution.
-
If
CLEANUPBACKUPFILES
is disabled during the procedure, the user must resort to out-of-band techniques to cleanup unused backup files. No further invocations ofCLEANUPBACKFILES
for a dropped store are possible through hierarchical store. -
This subprogram will un-register the store from
DBMS_DBFS_CONTENT
package. All files in the given store are deleted from the store (Tape or Amazon S3 Web service). The database table holding the store's entries in the database, is also dropped by this subprogram.
54.5.6 FLUSHCACHE Procedure
This procedure flushes out dirty contents from level-1 cache, which can be locked, to level-2 cache, thereby freeing-up space in level 1 cache.
Syntax
DBMS_DBFS_HS.FLUSHCACHE ( store_name IN VARCHAR2);
Parameters
Table 54-11 FLUSHCACHE Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
54.5.7 GETSTOREPROPERTY Function
This function retrieves the values of a property.
Syntax
DBMS_DBFS_HS.GETSTOREPROPERTY ( store_name IN VARCHAR2, property_name IN VARCHAR2, noexcp IN BOOLEAN DEFAULT FALSE) RETURN VARCHAR2;
Parameters
Table 54-12 GETSTOREPROPERTY Function Parameters
Parameter | Description |
---|---|
|
Name of store |
|
Name of property |
|
If set to |
Return Values
The values of a property.
Usage Notes
The specified store must already have been created.
54.5.8 RECONFIGCACHE Procedure
This procedure reconfigures the parameters of the database cache being used by the store.
Syntax
DBMS_DBFS_HS.RECONFIGCACHE ( store_name IN VARCHAR2, cache_size IN NUMBER DEFAULT NULL, lobcache_quota IN NUMBER DEFAULT NULL, optimal_tarball_size IN NUMBER DEFAULT NULL);
Parameters
Table 54-13 RECONFIGCACHE Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
|
Cumulative cache size used for the Hierarchical Store |
|
Fraction of the cache size that are assigned to level 1 cache |
|
Maximum possible size of an archive file. Since creating one file for every content file in the store is a prohibitively expensive operation, multiple content files are bundled together into one archive file for transfer to tape or Amazon S3. |
Usage Notes
-
The specified store must already have been created before reconfiguration.
-
The Hierarchical Store uses a level 1 cache and a level 2 cache. The level 1 cache subsumes most of the working set and the level 2 cache is used to perform bulk writes to the backend device.
-
If any of the last 3 parameters is
NULL
, its value specified during store creation is used. If the parameter wasNULL
when the call to the CREATESTORE Procedure was issued, theDBMS_DBFS_HS
package assigns a default value.The
DBMS_DBFS_HS
package optimistically tries to allocate more than 1 tarball's worth of size for level 2 cache to facilitate concurrency, though a minimum of 1 tarball size is necessary for level 2 cache.The values for cumulative cache size and LOB cache quota decide allocation of space for the two caches. If values are not provided, a user might see an
INSUFFICIENT_CACHE
exception. In that case, it is better to revise the cache parameters in order to have a working store. -
If this subprogram successfully executes, its actions cannot be rolled back by the user. In that case, the user should call
RECONFIGCACHE
again with new or modified parameters.
54.5.9 REGISTERSTORECOMMAND Procedure
This procedure registers commands for a store with the Hierarchical Store. These commands are sent to the Media Manager for the external storage device associated with the store.
Syntax
DBMS_DBFS_HS.REGISTERSTORECOMMAND ( store_name IN VARCHAR2, message IN VARCHAR2, flags IN NUMBER);
Parameters
Table 54-14 REGISTERSTORECOMMAND Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
|
Message to be sent to the Media Manager of the external store |
|
Valid values:
|
Usage Notes
-
These commands are sent before the next read or write of content. When the Hierarchical Store wants to push (or get) data to (or from) the storage device, it begins a session (to communicate with the device). After beginning the session, it sends all registered commands for the to the relevant device before writing (or getting) any data.
-
If this method successfully executes, its actions cannot be rolled back by the user. To restore the previous state the user must call the DEREGSTORECOMMAND Function.
54.5.10 SENDCOMMAND Procedure
This procedure sends a command to be executed on the external storage device's Media Manager.
Syntax
DBMS_DBFS_HS.SENDCOMMAND ( store_name IN VARCHAR2, message IN VARCHAR2);
Parameters
Table 54-15 SENDCOMMAND Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
|
Message string to be executed |
54.5.11 SETSTOREPROPERTY Procedure
This procedure stores properties of a store in the database as name-value pairs.
Syntax
DBMS_DBFS_HS.SETSTOREPROPERTY ( store_name IN VARCHAR2, property_name IN VARCHAR2, property_value IN VARCHAR2);
Parameters
Table 54-16 SETSTOREPROPERTY Procedure Parameters
Parameter | Description |
---|---|
|
Name of store |
|
For a store using Tape device, there are three properties whose values must be set by the user, and four properties that have default values. Stores of type |
|
Stores using a Tape Device The values for the following properties must be set by the user:
The following properties, which have default values assigned to them when a store is created, benefit from tuning:
|
|
Stores of type It is mandatory that the following properties have assigned values, and default values are provided:
|
|
The following properties are optional:
|
Usage Notes
-
The specified store must already have been created.
-
If this subprogram successfully executes, its actions cannot be rolled back by the user.
-
The same property can be set multiple times to the same or different values using this subprogram
-
Regarding
PROPNAME_ENABLECLEANUPONDELETE
behavior, a job is created for each store by theDBMS_DBFS_HS
to remove the unused files from the external storage. By default, the job is enabled forSTORETYPE_AMAZONS3
and is disabled forSTORETYPE_TAPE
. If theENABLECLEANUPONDELETE
property is set toTRUE
, the job is enabled; if the property is set toFALSE
, the job is disabled. If enabled, the job runs at an interval of one hour by default. TheDBMS_SCHEDULER
package can be used to modify the schedule. The name of the job can be obtained by queryingUSER_DBFS_HS_FIXED_PROPERTIES
forprop_name
=
'DELJOB_NAME
'.
Examples
Format
The format of wallet_path
in Windows is, for example:
file:c:\WINNT\Profiles\<username>\WALLETS
The format of wallet_path
in UNIX or Linux is, for example:
file:/home/username/wallets
54.5.12 STOREPUSH Procedure
This procedure pushes locally staged data to the remote storage.
Syntax
DBMS_DBFS_HS.STOREPUSH ( store_name IN VARCHAR2, path IN VARCHAR2 DEFAULT NULL);
Parameters
Table 54-17 STOREPUSH Procedure Parameters
Parameter | Description |
---|---|
|
Name of store whose content the client writes from local cache to the external store |
|
A non-mount qualified (without mount point) path within the store. By default, its value is |
Usage Notes
-
The Hierarchical Store caches the content files locally in database tables. When enough content is amassed in the cache to make it efficient to write to the external storage device (or the cache is completely filled), the Hierarchical Store creates a tarball out of the local content and writes these tarballs as files on the external device. The size of the tarball created by the Hierarchical Store is controlled by the store property
PROPNAME_OPTTARBALLSIZE
. -
When the amount of free space in the cache is such that the caching of a content file will push the space used above
cache_size
, the Hierarchical Store will internally callSTOREPUSH
. TheSTOREPUSH
Procedure
creates tarball(s) out of the existing dirty or modified content files in the cache and writes them out to the external device. ASTOREPUSH
call is not guaranteed to write all the dirty content from local cache to the external storage, since some files may be locked by other sessions. -
STOREPUSH
has a built-in ability feature allowing it to automatically resume operation. If aSTOREPUSH
call is interrupted (say by a network outage) after it has transferred some tarballs to the external device, it can be restarted after the outage and will then resume transferring data from the point it was interrupted. In other words, work done before the outage is not lost.STOREPUSH
can safely be restarted and the effect is such as if the outage never occurred. -
If this method successfully executes, its actions cannot be rolled back by the user.
-
By default, when
path
isNULL
, all files in the store are candidates forSTOREPUSH
. Ifpath
has a valid input value, all files which are under the namespace of given path are written from the local cache to the external store. If a given path is an existing file, it is pushed out again to the remote store.