mirror of
https://github.com/yasuhirokimura/db18
synced 2026-06-08 18:29:23 +00:00
Yasuhiro KIMURA
366 lines
15 KiB
HTML
366 lines
15 KiB
HTML
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||
<head>
|
||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
||
<title>External File support</title>
|
||
<link rel="stylesheet" href="gettingStarted.css" type="text/css" />
|
||
<meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
|
||
<link rel="start" href="index.html" title="Berkeley DB Programmer's Reference Guide" />
|
||
<link rel="up" href="am_misc.html" title="Chapter 4. Access Method Wrapup" />
|
||
<link rel="prev" href="am_misc_diskspace.html" title="Disk space requirements" />
|
||
<link rel="next" href="am_misc_tune.html" title="Access method tuning" />
|
||
</head>
|
||
<body>
|
||
<div xmlns="" class="navheader">
|
||
<div class="libver">
|
||
<p>Library Version 18.1.40</p>
|
||
</div>
|
||
<table width="100%" summary="Navigation header">
|
||
<tr>
|
||
<th colspan="3" align="center">External File support</th>
|
||
</tr>
|
||
<tr>
|
||
<td width="20%" align="left"><a accesskey="p" href="am_misc_diskspace.html">Prev</a> </td>
|
||
<th width="60%" align="center">Chapter 4. Access Method Wrapup </th>
|
||
<td width="20%" align="right"> <a accesskey="n" href="am_misc_tune.html">Next</a></td>
|
||
</tr>
|
||
</table>
|
||
<hr />
|
||
</div>
|
||
<div class="sect1" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h2 class="title" style="clear: both"><a id="blobs"></a>External File support</h2>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="toc">
|
||
<dl>
|
||
<dt>
|
||
<span class="sect2">
|
||
<a href="blobs.html#blob_threshold">The external file threshold</a>
|
||
</span>
|
||
</dt>
|
||
<dt>
|
||
<span class="sect2">
|
||
<a href="blobs.html#blob_create">Creating external files</a>
|
||
</span>
|
||
</dt>
|
||
<dt>
|
||
<span class="sect2">
|
||
<a href="blobs.html#blob_access">External file access</a>
|
||
</span>
|
||
</dt>
|
||
<dt>
|
||
<span class="sect2">
|
||
<a href="blobs.html#blob_storage">External file storage</a>
|
||
</span>
|
||
</dt>
|
||
<dt>
|
||
<span class="sect2">
|
||
<a href="blobs.html#blob_replication">External Files and Replication</a>
|
||
</span>
|
||
</dt>
|
||
</dl>
|
||
</div>
|
||
<p>
|
||
External file support is designed for
|
||
efficient storage of large objects. An object is considered to
|
||
be large if it is more than a third of the size of a page.
|
||
Without external file support, large objects must be broken up into
|
||
smaller pieces, and then reassembled and/or disassembled every
|
||
time the record is read or updated. Berkeley DB external file support
|
||
avoids this assembly/disassembly process by storing the large
|
||
object in a special directory set aside for the purpose. The
|
||
data itself is not kept in the database, nor is it placed into
|
||
the in-memory cache.
|
||
</p>
|
||
<p>
|
||
external files can only be stored using the data portion of a
|
||
key/data pair. They are supported only for Btree, Hash, and
|
||
Heap databases, and only so long as the database is not
|
||
configured for duplicate records, or
|
||
duplicate sorted records. In addition, the DBT that you use to
|
||
access the external file data cannot be configured as a partial DBT if
|
||
you want to access the data using the external file's streaming
|
||
interface (introduced below).
|
||
</p>
|
||
<p>
|
||
Note that if the environment is transactionally-protected,
|
||
then all access to the external file is also transactionally protected.
|
||
</p>
|
||
<div class="sect2" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h3 class="title"><a id="blob_threshold"></a>The external file threshold</h3>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
The external file threshold is a positive integer, in bytes,
|
||
which indicates how large an object must be before it is
|
||
considered an external file. By default, the external file
|
||
threshold for any given database is <code class="literal">0</code>, which
|
||
means that no object will ever be considered an external file. This
|
||
means that the external file feature is not used by default for
|
||
Berkeley DB databases.
|
||
</p>
|
||
<p>
|
||
In order to use the external file feature, you must set the
|
||
external file threshold to a non-zero, positive integer value. You
|
||
do this for a given database using the <a href="../api_reference/C/set_blob_threshold.html" class="olink">DB->set_blob_threshold()</a>
|
||
method. Note that this value must be set before you create
|
||
the database. At any point after database creation time,
|
||
this method is ignored.
|
||
</p>
|
||
<p>
|
||
In addition, if you are using an environment, you can
|
||
change the default threshold for databases created in that
|
||
environment to something other than <code class="literal">0</code>
|
||
by using the <a href="../api_reference/C/envset_blob_threshold.html" class="olink">DB_ENV->set_blob_threshold()</a> method.
|
||
</p>
|
||
<p> You can retrieve the external file threshold set for a database
|
||
using the <a href="../api_reference/C/get_blob_threshold.html" class="olink">DB->get_blob_threshold()</a>. You can retrieve the
|
||
default external file threshold set for your environment using the
|
||
<a href="../api_reference/C/envget_blob_threshold.html" class="olink">DB_ENV->get_blob_threshold()</a>.
|
||
</p>
|
||
</div>
|
||
<div class="sect2" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h3 class="title"><a id="blob_create"></a>Creating external files</h3>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
There are two ways to create an external file. Before you can use
|
||
either mechanism, you must set the external file threshold to a
|
||
non-zero positive integer value (see the previous section
|
||
for details). Once the external file threshold has been set, you
|
||
create an external file using one of the two following mechanisms:
|
||
</p>
|
||
<div class="itemizedlist">
|
||
<ul type="disc">
|
||
<li>
|
||
<p>
|
||
Configure the DBT used to access the external file data
|
||
(that is, the DBT used for the data portion of the
|
||
record) with the <a href="../api_reference/C/dbt.html#dbt_DB_DBT_BLOB" class="olink">DB_DBT_BLOB</a> flag. This causes
|
||
the data to be stored as an external file regardless of its
|
||
size, so long as the database otherwise supports
|
||
external files.
|
||
</p>
|
||
</li>
|
||
<li>
|
||
<p>
|
||
Alternatively, creating a data item with a size
|
||
greater than the external file threshold will cause that
|
||
data item to be automatically stored as an external file.
|
||
</p>
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
<div class="sect2" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h3 class="title"><a id="blob_access"></a>External file access</h3>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
external files may be accessed in the same way as other DBT
|
||
data, so long as the data itself will fit into memory.
|
||
More likely, you will find it necessary to use the external file
|
||
streaming API to read and write external file data. You open an
|
||
external file stream using the <a href="../api_reference/C/dbstream.html" class="olink">DBC->db_stream()</a> method, close it with the
|
||
<a href="../api_reference/C/dbstream_close.html" class="olink">DB_STREAM->close()</a> method, write to it using the the
|
||
<a href="../api_reference/C/dbstream_write.html" class="olink">DB_STREAM->write()</a> method, and read it using the
|
||
<a href="../api_reference/C/dbstream_read.html" class="olink">DB_STREAM->read()</a> method.
|
||
</p>
|
||
<p>
|
||
The following example code fragment can be found in
|
||
your DB distribution at
|
||
<code class="filename">.../db/examples/c/ex_external_file.c</code>.
|
||
</p>
|
||
<pre class="programlisting">...
|
||
/* Some necessary variable declarations */
|
||
DBC *dbc; /* Cursor handle */
|
||
DB_ENV *dbenv; /* Environment handle */
|
||
DB *dbp; /* Database handle */
|
||
DB_STREAM *dbs; /* Stream handle */
|
||
DB_TXN *txn; /* Transaction handle */
|
||
DBT data, key; /* DBT handles */
|
||
int ret;
|
||
db_off_t size;
|
||
|
||
...
|
||
|
||
/* Environment creation skipped for brevity's sake */
|
||
|
||
...
|
||
|
||
/* Enable external files and set the size threshold. */
|
||
if ((ret = dbenv->set_ext_file_threshold(dbenv, 1000, 0)) != 0) {
|
||
dbenv->err(dbenv, ret, "set_ext_file_threshold");
|
||
goto err;
|
||
}
|
||
|
||
...
|
||
|
||
/* Database and DBT creation skipped for brevity's sake */
|
||
|
||
...
|
||
|
||
/*
|
||
Access the external file using the DB_STREAM API.
|
||
*/
|
||
if ((ret = dbenv->txn_begin(dbenv, NULL, &txn, 0)) != 0){
|
||
dbenv->err(dbenv, ret, "txn");
|
||
goto err;
|
||
}
|
||
|
||
if ((ret = dbp->cursor(dbp, txn, &dbc, 0)) != 0) {
|
||
dbenv->err(dbenv, ret, "cursor");
|
||
goto err;
|
||
}
|
||
|
||
/*
|
||
* Set the cursor to an external file. Use DB_DBT_PARTIAL with
|
||
* dlen == 0 to avoid getting any external file data.
|
||
*/
|
||
data.flags = DB_DBT_USERMEM | DB_DBT_PARTIAL;
|
||
data.dlen = 0;
|
||
if ((ret = dbc->get(dbc, &key, &data, DB_FIRST)) != 0) {
|
||
dbenv->err(dbenv, ret, "Not an external file");
|
||
goto err;
|
||
}
|
||
data.flags = DB_DBT_USERMEM;
|
||
|
||
/* Create a stream on the external file the cursor points to. */
|
||
if ((ret = dbc->db_stream(dbc, &dbs, DB_STREAM_WRITE)) != 0) {
|
||
dbenv->err(dbenv, 0, "Creating stream.");
|
||
goto err;
|
||
}
|
||
|
||
/* Get the size of the external file. */
|
||
if ((ret = dbs->size(dbs, &size, 0)) != 0) {
|
||
dbenv->err(dbenv, 0, "Stream size.");
|
||
goto err;
|
||
}
|
||
/* Read from the external file. */
|
||
if ((ret = dbs->read(dbs, &data, 0, (u_int32_t)size, 0)) != 0) {
|
||
dbenv->err(dbenv, 0, "Stream read.");
|
||
goto err;
|
||
}
|
||
/* Write data to the external file, increasing its size. */
|
||
if ((ret = dbs->write(dbs, &data, size/2, 0)) != 0) {
|
||
dbenv->err(dbenv, 0, "Stream write.");
|
||
goto err;
|
||
}
|
||
/* Close the stream. */
|
||
if ((ret = dbs->close(dbs, 0)) != 0) {
|
||
dbenv->err(dbenv, 0, "Stream close.");
|
||
goto err;
|
||
}
|
||
dbs = NULL;
|
||
dbc->close(dbc);
|
||
dbc = NULL;
|
||
txn->commit(txn, 0);
|
||
txn = NULL;
|
||
free(data.data);
|
||
data.data = NULL;
|
||
|
||
...
|
||
|
||
/* Handle clean up skipped. */ </pre>
|
||
</div>
|
||
<div class="sect2" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h3 class="title"><a id="blob_storage"></a>External file storage</h3>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
external files are not stored in the normal database files on
|
||
disk in the same way as is other data managed by DB.
|
||
Instead, they are stored as binary files in a special
|
||
directory set aside for the purpose.
|
||
</p>
|
||
<p>
|
||
If you are not using environments, this special external file
|
||
directory is created relative to the current working
|
||
directory from which your application is running. You can
|
||
modify this default location using the <a href="../api_reference/C/set_blob_dir.html" class="olink">DB->set_blob_dir()</a>
|
||
method, and retrieve the current external file directory using
|
||
<a href="../api_reference/C/get_blob_dir.html" class="olink">DB->get_blob_dir()</a>.
|
||
</p>
|
||
<p>
|
||
If you are using an environment, then by default the
|
||
external file directory is created within the environment's home
|
||
directory. You can change this default location using
|
||
<a href="../api_reference/C/envset_blob_dir.html" class="olink">DB_ENV->set_blob_dir()</a> and retrieve the current default
|
||
location using <a href="../api_reference/C/envget_blob_dir.html" class="olink">DB_ENV->get_blob_dir()</a>. (Note that
|
||
<a href="../api_reference/C/envget_blob_dir.html" class="olink">DB_ENV->get_blob_dir()</a> can successfully retrieve the external file
|
||
directory only if <a href="../api_reference/C/envset_blob_dir.html" class="olink">DB_ENV->set_blob_dir()</a> was previously
|
||
called.)
|
||
</p>
|
||
<p>
|
||
Note that because external files are stored outside of the
|
||
Berkeley DB database files, they are not confined by the
|
||
four gigabyte limit used for Berkeley DB key and data
|
||
items. The external file size limit is system dependent. It can be
|
||
the maximum value in bytes of a signed 32 bit integer (if
|
||
the Berkeley DB-defined type <code class="literal">db_off_t</code>
|
||
is four bytes in size), or a signed 64 bit integer (if
|
||
<code class="literal">db_off_t</code> is eight bytes in size).
|
||
</p>
|
||
</div>
|
||
<div class="sect2" lang="en" xml:lang="en">
|
||
<div class="titlepage">
|
||
<div>
|
||
<div>
|
||
<h3 class="title"><a id="blob_replication"></a>External Files and Replication</h3>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<p>
|
||
Replication supports external files without any special
|
||
requirements. However, enabling external files in a replicated
|
||
environment can result in long synchronization times between the
|
||
client and master sites. To avoid this, execute a transaction
|
||
checkpoint after updating or deleting one or more external file
|
||
records.
|
||
</p>
|
||
</div>
|
||
</div>
|
||
<div class="navfooter">
|
||
<hr />
|
||
<table width="100%" summary="Navigation footer">
|
||
<tr>
|
||
<td width="40%" align="left"><a accesskey="p" href="am_misc_diskspace.html">Prev</a> </td>
|
||
<td width="20%" align="center">
|
||
<a accesskey="u" href="am_misc.html">Up</a>
|
||
</td>
|
||
<td width="40%" align="right"> <a accesskey="n" href="am_misc_tune.html">Next</a></td>
|
||
</tr>
|
||
<tr>
|
||
<td width="40%" align="left" valign="top">Disk space
|
||
requirements </td>
|
||
<td width="20%" align="center">
|
||
<a accesskey="h" href="index.html">Home</a>
|
||
</td>
|
||
<td width="40%" align="right" valign="top"> Access method tuning</td>
|
||
</tr>
|
||
</table>
|
||
</div>
|
||
</body>
|
||
</html>
|