You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
192 lines
8.3 KiB
192 lines
8.3 KiB
/*
|
|
* Copyright 2015-2021 Howard Chu, Symas Corp.
|
|
* All rights reserved.
|
|
*
|
|
* Redistribution and use in source and binary forms, with or without
|
|
* modification, are permitted only as authorized by the OpenLDAP
|
|
* Public License.
|
|
*
|
|
* A copy of this license is available in the file LICENSE in the
|
|
* top-level directory of the distribution or, alternatively, at
|
|
* <http://www.OpenLDAP.org/license.html>.
|
|
*/
|
|
/** @page starting Getting Started
|
|
|
|
LMDB is compact, fast, powerful, and robust and implements a simplified
|
|
variant of the BerkeleyDB (BDB) API. (BDB is also very powerful, and verbosely
|
|
documented in its own right.) After reading this page, the main
|
|
\ref mdb documentation should make sense. Thanks to Bert Hubert
|
|
for creating the
|
|
<a href="https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md">
|
|
initial version</a> of this writeup.
|
|
|
|
Everything starts with an environment, created by #mdb_env_create().
|
|
Once created, this environment must also be opened with #mdb_env_open().
|
|
|
|
#mdb_env_open() gets passed a name which is interpreted as a directory
|
|
path. Note that this directory must exist already, it is not created
|
|
for you. Within that directory, a lock file and a storage file will be
|
|
generated. If you don't want to use a directory, you can pass the
|
|
#MDB_NOSUBDIR option, in which case the path you provided is used
|
|
directly as the data file, and another file with a "-lock" suffix
|
|
added will be used for the lock file.
|
|
|
|
Once the environment is open, a transaction can be created within it
|
|
using #mdb_txn_begin(). Transactions may be read-write or read-only,
|
|
and read-write transactions may be nested. A transaction must only
|
|
be used by one thread at a time. Transactions are always required,
|
|
even for read-only access. The transaction provides a consistent
|
|
view of the data.
|
|
|
|
Once a transaction has been created, a database can be opened within it
|
|
using #mdb_dbi_open(). If only one database will ever be used in the
|
|
environment, a NULL can be passed as the database name. For named
|
|
databases, the #MDB_CREATE flag must be used to create the database
|
|
if it doesn't already exist. Also, #mdb_env_set_maxdbs() must be
|
|
called after #mdb_env_create() and before #mdb_env_open() to set the
|
|
maximum number of named databases you want to support.
|
|
|
|
Note: a single transaction can open multiple databases. Generally
|
|
databases should only be opened once, by the first transaction in
|
|
the process. After the first transaction completes, the database
|
|
handles can freely be used by all subsequent transactions.
|
|
|
|
Within a transaction, #mdb_get() and #mdb_put() can store single
|
|
key/value pairs if that is all you need to do (but see \ref Cursors
|
|
below if you want to do more).
|
|
|
|
A key/value pair is expressed as two #MDB_val structures. This struct
|
|
has two fields, \c mv_size and \c mv_data. The data is a \c void pointer to
|
|
an array of \c mv_size bytes.
|
|
|
|
Because LMDB is very efficient (and usually zero-copy), the data returned
|
|
in an #MDB_val structure may be memory-mapped straight from disk. In
|
|
other words <b>look but do not touch</b> (or free() for that matter).
|
|
Once a transaction is closed, the values can no longer be used, so
|
|
make a copy if you need to keep them after that.
|
|
|
|
@section Cursors Cursors
|
|
|
|
To do more powerful things, we must use a cursor.
|
|
|
|
Within the transaction, a cursor can be created with #mdb_cursor_open().
|
|
With this cursor we can store/retrieve/delete (multiple) values using
|
|
#mdb_cursor_get(), #mdb_cursor_put(), and #mdb_cursor_del().
|
|
|
|
#mdb_cursor_get() positions itself depending on the cursor operation
|
|
requested, and for some operations, on the supplied key. For example,
|
|
to list all key/value pairs in a database, use operation #MDB_FIRST for
|
|
the first call to #mdb_cursor_get(), and #MDB_NEXT on subsequent calls,
|
|
until the end is hit.
|
|
|
|
To retrieve all keys starting from a specified key value, use #MDB_SET.
|
|
For more cursor operations, see the \ref mdb docs.
|
|
|
|
When using #mdb_cursor_put(), either the function will position the
|
|
cursor for you based on the \b key, or you can use operation
|
|
#MDB_CURRENT to use the current position of the cursor. Note that
|
|
\b key must then match the current position's key.
|
|
|
|
@subsection summary Summarizing the Opening
|
|
|
|
So we have a cursor in a transaction which opened a database in an
|
|
environment which is opened from a filesystem after it was
|
|
separately created.
|
|
|
|
Or, we create an environment, open it from a filesystem, create a
|
|
transaction within it, open a database within that transaction,
|
|
and create a cursor within all of the above.
|
|
|
|
Got it?
|
|
|
|
@section thrproc Threads and Processes
|
|
|
|
LMDB uses POSIX locks on files, and these locks have issues if one
|
|
process opens a file multiple times. Because of this, do not
|
|
#mdb_env_open() a file multiple times from a single process. Instead,
|
|
share the LMDB environment that has opened the file across all threads.
|
|
Otherwise, if a single process opens the same environment multiple times,
|
|
closing it once will remove all the locks held on it, and the other
|
|
instances will be vulnerable to corruption from other processes.
|
|
|
|
Also note that a transaction is tied to one thread by default using
|
|
Thread Local Storage. If you want to pass read-only transactions across
|
|
threads, you can use the #MDB_NOTLS option on the environment.
|
|
|
|
@section txns Transactions, Rollbacks, etc.
|
|
|
|
To actually get anything done, a transaction must be committed using
|
|
#mdb_txn_commit(). Alternatively, all of a transaction's operations
|
|
can be discarded using #mdb_txn_abort(). In a read-only transaction,
|
|
any cursors will \b not automatically be freed. In a read-write
|
|
transaction, all cursors will be freed and must not be used again.
|
|
|
|
For read-only transactions, obviously there is nothing to commit to
|
|
storage. The transaction still must eventually be aborted to close
|
|
any database handle(s) opened in it, or committed to keep the
|
|
database handles around for reuse in new transactions.
|
|
|
|
In addition, as long as a transaction is open, a consistent view of
|
|
the database is kept alive, which requires storage. A read-only
|
|
transaction that no longer requires this consistent view should
|
|
be terminated (committed or aborted) when the view is no longer
|
|
needed (but see below for an optimization).
|
|
|
|
There can be multiple simultaneously active read-only transactions
|
|
but only one that can write. Once a single read-write transaction
|
|
is opened, all further attempts to begin one will block until the
|
|
first one is committed or aborted. This has no effect on read-only
|
|
transactions, however, and they may continue to be opened at any time.
|
|
|
|
@section dupkeys Duplicate Keys
|
|
|
|
#mdb_get() and #mdb_put() respectively have no and only some support
|
|
for multiple key/value pairs with identical keys. If there are multiple
|
|
values for a key, #mdb_get() will only return the first value.
|
|
|
|
When multiple values for one key are required, pass the #MDB_DUPSORT
|
|
flag to #mdb_dbi_open(). In an #MDB_DUPSORT database, by default
|
|
#mdb_put() will not replace the value for a key if the key existed
|
|
already. Instead it will add the new value to the key. In addition,
|
|
#mdb_del() will pay attention to the value field too, allowing for
|
|
specific values of a key to be deleted.
|
|
|
|
Finally, additional cursor operations become available for
|
|
traversing through and retrieving duplicate values.
|
|
|
|
@section optim Some Optimization
|
|
|
|
If you frequently begin and abort read-only transactions, as an
|
|
optimization, it is possible to only reset and renew a transaction.
|
|
|
|
#mdb_txn_reset() releases any old copies of data kept around for
|
|
a read-only transaction. To reuse this reset transaction, call
|
|
#mdb_txn_renew() on it. Any cursors in this transaction must also
|
|
be renewed using #mdb_cursor_renew().
|
|
|
|
Note that #mdb_txn_reset() is similar to #mdb_txn_abort() and will
|
|
close any databases you opened within the transaction.
|
|
|
|
To permanently free a transaction, reset or not, use #mdb_txn_abort().
|
|
|
|
@section cleanup Cleaning Up
|
|
|
|
For read-only transactions, any cursors created within it must
|
|
be closed using #mdb_cursor_close().
|
|
|
|
It is very rarely necessary to close a database handle, and in
|
|
general they should just be left open.
|
|
|
|
@section onward The Full API
|
|
|
|
The full \ref mdb documentation lists further details, like how to:
|
|
|
|
\li size a database (the default limits are intentionally small)
|
|
\li drop and clean a database
|
|
\li detect and report errors
|
|
\li optimize (bulk) loading speed
|
|
\li (temporarily) reduce robustness to gain even more speed
|
|
\li gather statistics about the database
|
|
\li define custom sort orders
|
|
|
|
*/
|
|
|