NextGraph
/
rust-rocksdb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:rust-rocksdb/rust-rocksdb

Browse Source
master
Maxim Molchanov 6 years ago
parent 278f7762f2 184282609c
commit 4585d2b13a
36 changed files with 6097 additions and 50 deletions
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Show Stats Download Patch File Download Diff File
  1. 3
      .gitignore
  2. 9
      .gitmodules
  3. 8
      .travis.yml
  4. 37
      CHANGELOG.txt
  5. 12
      Cargo.toml
  6. 43
      MAINTAINERSHIP.md
  7. 25
      README.md
  8. 76
      code-of-conduct.md
  9. 2
      librocksdb-sys/.gitignore
  10. 12
      librocksdb-sys/Cargo.toml
  11. 174
      librocksdb-sys/build.rs
  12. 42
      librocksdb-sys/bzip2/LICENSE
  13. 1094
      librocksdb-sys/bzip2/blocksort.c
  14. 1572
      librocksdb-sys/bzip2/bzlib.c
  15. 282
      librocksdb-sys/bzip2/bzlib.h
  16. 509
      librocksdb-sys/bzip2/bzlib_private.h
  17. 672
      librocksdb-sys/bzip2/compress.c
  18. 104
      librocksdb-sys/bzip2/crctable.c
  19. 646
      librocksdb-sys/bzip2/decompress.c
  20. 205
      librocksdb-sys/bzip2/huffman.c
  21. 84
      librocksdb-sys/bzip2/randtable.c
  22. 1
      librocksdb-sys/lz4
  23. 2
      librocksdb-sys/rocksdb
  24. 2
      librocksdb-sys/rocksdb_lib_sources.txt
  25. 2
      librocksdb-sys/snappy
  26. 7
      librocksdb-sys/src/lib.rs
  27. 8
      librocksdb-sys/tests/ffi.rs
  28. 1
      librocksdb-sys/zlib
  29. 1
      librocksdb-sys/zstd
  30. 80
      src/checkpoint.rs
  31. 55
      src/db.rs
  32. 148
      src/db_options.rs
  33. 25
      src/lib.rs
  34. 6
      src/merge_operator.rs
  35. 113
      tests/test_checkpoint.rs
  36. 61
      tests/test_iterator.rs

3
.gitignore vendored
Unescape View File

@ -1,5 +1,4 @@
*.swo *.swo
*.swp
target target
Cargo.lock Cargo.lock
*.orig *.orig
@ -8,3 +7,5 @@ _rust_rocksdb*
*rlib *rlib
tags tags
path path
.DS_Store
.idea

9
.gitmodules vendored
Unescape View File

@ -4,3 +4,12 @@
[submodule "rocksdb_sys/rocksdb"] [submodule "rocksdb_sys/rocksdb"]
path = librocksdb-sys/rocksdb path = librocksdb-sys/rocksdb
url = https://github.com/facebook/rocksdb.git url = https://github.com/facebook/rocksdb.git
[submodule "librocksdb-sys/lz4"]
path = librocksdb-sys/lz4
url = https://github.com/lz4/lz4.git
[submodule "librocksdb-sys/zstd"]
path = librocksdb-sys/zstd
url = https://github.com/facebook/zstd
[submodule "librocksdb-sys/zlib"]
path = librocksdb-sys/zlib
url = https://github.com/madler/zlib.git

8
.travis.yml
Unescape View File

@ -2,6 +2,11 @@ language: rust
dist: trusty dist: trusty
sudo: true sudo: true
os:
- linux
- osx
- windows
rust: rust:
- stable - stable
- beta - beta
@ -22,3 +27,6 @@ script:
- cargo test - cargo test
cache: cargo cache: cargo
before_cache:
- rm -rfv target/

37
CHANGELOG.txt
Unescape View File

@ -1,17 +1,48 @@
Changelog Changelog
========= =========
Unreleased (Future Date)
~~~~~~~~~~~~~~~~~~~~
* add LZ4, ZSTD, ZLIB, and BZIP2 compression support (iSynaptic)
0.10.1 (2018-07-17)
~~~~~~~~~~~~~~~~~~~~
* bump bindgen to 0.37 (ekmartin)
* bump rocksdb to 5.14.2 (ekmartin)
* add disable_cache to block-based options (ekmartin)
* add set_wal_dir (ekmartin)
* add set_memtable_prefix_bloom_ratio (ekmartin)
* add MemtableFactory support (ekmartin)
* add full_iterator (ekmartin)
* allow index type specification on block options (ekmartin)
* fix windows build (iSynaptic)
0.10.0 (2018-03-17)
~~~~~~~~~~~~~~~~~~~~
* Bump rocksdb to 5.11.3 (spacejam)
**New features**
* Link with system rocksdb and snappy libs through envvars (ozkriff)
**Breaking Changes**
* Fix reverse iteration from a given key (ongardie)
0.9.1 (2018-02-10) 0.9.1 (2018-02-10)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
**New features** **New features**
* SliceTransform support (spacejam) * SliceTransform support (spacejam)
0.9.0 (2018-02-10) 0.9.0 (2018-02-10)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
**New features** **New features**
* Allow creating iterators over prefixes (glittershark) * Allow creating iterators over prefixes (glittershark)
**Breaking Changes** **Breaking Changes**
* Open cfs with options (garyttierney, rrichardson) * Open cfs with options (garyttierney, rrichardson)
* Non-Associative merge ops (rrichardson) * Non-Associative merge ops (rrichardson)
@ -45,16 +76,19 @@ Changelog
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
**Breaking Changes** **Breaking Changes**
* Bumped rocksdb to 5.4.6 (derekdreery) * Bumped rocksdb to 5.4.6 (derekdreery)
* Remove `use_direct_writes` now that `use_direct_io_for_flush_and_compaction` exists (derekdreery) * Remove `use_direct_writes` now that `use_direct_io_for_flush_and_compaction` exists (derekdreery)
**New features** **New features**
* ReadOptions is now public (rschmukler) * ReadOptions is now public (rschmukler)
* Implement Clone and AsRef<str> for Error (daboross) * Implement Clone and AsRef<str> for Error (daboross)
* Support for `seek_for_prev` (kaedroho) * Support for `seek_for_prev` (kaedroho)
* Support for DirectIO (kaedroho) * Support for DirectIO (kaedroho)
**Internal cleanups** **Internal cleanups**
* Fixed race condition in tests (debris) * Fixed race condition in tests (debris)
* Move tests to the default `tests` directory (vmx) * Move tests to the default `tests` directory (vmx)
@ -62,15 +96,18 @@ Changelog
~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~
**New features** **New features**
* Support for raw iterator access (kaedroho) * Support for raw iterator access (kaedroho)
0.6 (2016-12-18) 0.6 (2016-12-18)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
**Breaking changes** **Breaking changes**
* Comparator function now returns an Ordering (alexreg) * Comparator function now returns an Ordering (alexreg)
**New features** **New features**
* Compaction filter (tmccombs) * Compaction filter (tmccombs)
* Support for backups (alexreg) * Support for backups (alexreg)

12
Cargo.toml
Unescape View File

@ -1,7 +1,7 @@
[package] [package]
name = "rocksdb" name = "rocksdb"
description = "Rust wrapper for Facebook's RocksDB embeddable database" description = "Rust wrapper for Facebook's RocksDB embeddable database"
version = "0.9.1" version = "0.10.1"
authors = ["Tyler Neely <t@jujit.su>", "David Greenberg <dsg123456789@gmail.com>"] authors = ["Tyler Neely <t@jujit.su>", "David Greenberg <dsg123456789@gmail.com>"]
license = "Apache-2.0" license = "Apache-2.0"
keywords = ["database", "embedded", "LSM-tree", "persistence"] keywords = ["database", "embedded", "LSM-tree", "persistence"]
@ -14,9 +14,15 @@ exclude = [
] ]
[features] [features]
default = [] default = ["snappy", "lz4", "zstd", "zlib", "bzip2"]
valgrind = [] valgrind = []
snappy = ["librocksdb-sys/snappy"]
lz4 = ["librocksdb-sys/lz4"]
zstd = ["librocksdb-sys/zstd"]
zlib = ["librocksdb-sys/zlib"]
bzip2 = ["librocksdb-sys/bzip2"]
[dependencies] [dependencies]
libc = "0.2" libc = "0.2"
librocksdb-sys = { path = "librocksdb-sys", version = "5.10.2" } librocksdb-sys = { path = "librocksdb-sys", version = "5.14.2" }

43
MAINTAINERSHIP.md
Unescape View File

@ -0,0 +1,43 @@
Maintainers agree to operate under this set of guidelines:
#### Authority
Maintainers are trusted to close issues, merge pull requests, and publish crates to cargo.
#### Categories of Work
0. Minor
* updating the changelog
* requires no approval
1. Normal
* librocksdb-sys updates
* API tracking code in the rocksdb crate that does not change control flow
* breaking changes due to removed functionality in rocksdb
* require 1 approval from another maintainer. if no maintainer is able to be reached for 2 weeks, then progress may be made anyway
* patch (and post-1.0, minor) releases to crates.io that contain only the above work
2. Major
* breaking API changes that are not direct consequences of underlying rocksdb changes
* refactoring, which should generally only be done for clearly functional reasons like to aid in the completion of a specific task
* require consensus among all maintainers unless 2 weeks have gone by without full participation
* if 2 weeks have gone by after seeking feedback, and at least one other maintainer has participated, and all participating maintainers are in agreement, then progress may be made anyway
* if action is absolutely urgent, an organization owner may act as a tiebreaker if specifically requested to do so and they agree that making a controversial decision is worth the risk. This should hopefully never occur.
If any maintainer thinks an issue is major, it is major.
#### Changelog Maintenance
* If you are the one who merges a PR that includes an externally-visible change, please describe the change in the changelog and merge it in.
#### Releasing, Publishing
* Releases adhere to [semver](https://semver.org/)
* To cut a release, an issue should be opened for it and reach the required approval based on the above `Categories of Work` section above
* When progress is possible, the issue may be closed and the proposer may publish to crates.io. This is controlled by those in the [crate publishers organization-level team](https://github.com/orgs/rust-rocksdb/teams/crate-publishers).
* Releases should have an associated tag pushed to this repo. I recommend doing this after the publish to crates.io succeeds to prevent any mishaps around pushing a tag for something that can't actually be published.
* The changelog serves as a sort of logical staging area for releases
* If a breaking API change happens, and the changelog has not advanced to a new major version, we roll the changelog to a new major version and open an issue to release the previous patch (and post-1.0, minor) version.
* Before rolling to a new major version, it would be nice to release a non-breaking point release to let current users silently take advantage of any improvements
#### Becoming a Maintainer
* If you have a history of participation in this repo, agree to these rules, and wish to take on maintainership responsibilities, you may open an issue. If an owner agrees, they will add you to the maintainer group and the crate publishers team.

25
README.md
Unescape View File

@ -1,11 +1,30 @@
rust-rocksdb rust-rocksdb
============ ============
[![Build Status](https://travis-ci.org/spacejam/rust-rocksdb.svg?branch=master)](https://travis-ci.org/spacejam/rust-rocksdb) [![Build Status](https://travis-ci.org/rust-rocksdb/rust-rocksdb.svg?branch=master)](https://travis-ci.org/rust-rocksdb/rust-rocksdb)
[![crates.io](http://meritbadge.herokuapp.com/rocksdb)](https://crates.io/crates/rocksdb) [![crates.io](http://meritbadge.herokuapp.com/rocksdb)](https://crates.io/crates/rocksdb)
[![documentation](https://docs.rs/rocksdb/badge.svg)](https://docs.rs/rocksdb) [![documentation](https://docs.rs/rocksdb/badge.svg)](https://docs.rs/rocksdb)
Feedback and pull requests welcome! If a particular feature of RocksDB is important to you, please let me know by opening an issue, and I'll prioritize it. Feedback and pull requests welcome! If a particular feature of RocksDB is
important to you, please let me know by opening an issue, and I'll
prioritize it.
This binding is statically linked with a specific version of RocksDB. If you want to build it yourself, make sure you've also cloned the RocksDB and Snappy submodules: This binding is statically linked with a specific version of RocksDB. If you
want to build it yourself, make sure you've also cloned the RocksDB and
compression submodules:
git submodule update --init --recursive git submodule update --init --recursive
## Compression Support
By default, support for the [Snappy](https://github.com/google/snappy),
[LZ4](https://github.com/lz4/lz4), [Zstd](https://github.com/facebook/zstd),
[Zlib](https://zlib.net), and [Bzip2](http://www.bzip.org) compression
is enabled through crate features. If support for all of these compression
algorithms is not needed, default features can be disabled and specific
compression algorithms can be enabled. For example, to enable only LZ4
compression support, make these changes to your Cargo.toml:
```
[dependencies.rocksdb]
default-features = false
features = ["lz4"]
```

76
code-of-conduct.md
Unescape View File

@ -0,0 +1,76 @@
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at t@jujit.su. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq

2
librocksdb-sys/.gitignore vendored
Unescape View File

@ -1,2 +0,0 @@
target
Cargo.lock

12
librocksdb-sys/Cargo.toml
Unescape View File

@ -1,6 +1,6 @@
[package] [package]
name = "librocksdb-sys" name = "librocksdb-sys"
version = "5.10.2" version = "5.14.2"
authors = ["Karl Hobley <karlhobley10@gmail.com>", "Arkadiy Paronyan <arkadiy@ethcore.io>"] authors = ["Karl Hobley <karlhobley10@gmail.com>", "Arkadiy Paronyan <arkadiy@ethcore.io>"]
license = "MIT/Apache-2.0/BSD-3-Clause" license = "MIT/Apache-2.0/BSD-3-Clause"
description = "Native bindings to librocksdb" description = "Native bindings to librocksdb"
@ -14,14 +14,22 @@ links = "rocksdb"
[features] [features]
default = [ "static" ] default = [ "static" ]
static = [] static = []
snappy = []
lz4 = []
zstd = []
zlib = []
bzip2 = []
[dependencies] [dependencies]
libc = "0.2" libc = "0.2"
[dev-dependencies] [dev-dependencies]
const-cstr = "0.2" const-cstr = "0.2"
uuid = { version = "0.7", features = ["v4"] }
[build-dependencies] [build-dependencies]
cc = { version = "^1.0", features = ["parallel"] } cc = { version = "^1.0", features = ["parallel"] }
make-cmd = "0.1" make-cmd = "0.1"
bindgen = "0.29" bindgen = "0.37"
glob = "0.2.11"

174
librocksdb-sys/build.rs
Unescape View File

@ -1,5 +1,6 @@
extern crate cc; extern crate cc;
extern crate bindgen; extern crate bindgen;
extern crate glob;
use std::env; use std::env;
use std::fs; use std::fs;
@ -29,13 +30,10 @@ fn fail_on_empty_directory(name: &str) {
} }
} }
fn build_rocksdb() { fn bindgen_rocksdb() {
println!("cargo:rerun-if-changed=build.rs");
println!("cargo:rerun-if-changed=rocksdb/");
let bindings = bindgen::Builder::default() let bindings = bindgen::Builder::default()
.header("rocksdb/include/rocksdb/c.h") .header("rocksdb/include/rocksdb/c.h")
.hide_type("max_align_t") // https://github.com/rust-lang-nursery/rust-bindgen/issues/550 .blacklist_type("max_align_t") // https://github.com/rust-lang-nursery/rust-bindgen/issues/550
.ctypes_prefix("libc") .ctypes_prefix("libc")
.generate() .generate()
.expect("unable to generate rocksdb bindings"); .expect("unable to generate rocksdb bindings");
@ -44,16 +42,42 @@ fn build_rocksdb() {
bindings bindings
.write_to_file(out_path.join("bindings.rs")) .write_to_file(out_path.join("bindings.rs"))
.expect("unable to write rocksdb bindings"); .expect("unable to write rocksdb bindings");
}
fn build_rocksdb() {
let mut config = cc::Build::new(); let mut config = cc::Build::new();
config.include("rocksdb/include/"); config.include("rocksdb/include/");
config.include("rocksdb/"); config.include("rocksdb/");
config.include("rocksdb/third-party/gtest-1.7.0/fused-src/"); config.include("rocksdb/third-party/gtest-1.7.0/fused-src/");
if cfg!(feature = "snappy") {
config.define("SNAPPY", Some("1"));
config.include("snappy/"); config.include("snappy/");
config.include("."); }
if cfg!(feature = "lz4") {
config.define("LZ4", Some("1"));
config.include("lz4/lib/");
}
if cfg!(feature = "zstd") {
config.define("ZSTD", Some("1"));
config.include("zstd/lib/");
config.include("zstd/lib/dictBuilder/");
}
if cfg!(feature = "zlib") {
config.define("ZLIB", Some("1"));
config.include("zlib/");
}
if cfg!(feature = "bzip2") {
config.define("BZIP2", Some("1"));
config.include("bzip2/");
}
config.include(".");
config.define("NDEBUG", Some("1")); config.define("NDEBUG", Some("1"));
config.define("SNAPPY", Some("1"));
let mut lib_sources = include_str!("rocksdb_lib_sources.txt") let mut lib_sources = include_str!("rocksdb_lib_sources.txt")
.split(" ") .split(" ")
@ -84,8 +108,9 @@ fn build_rocksdb() {
config.define("ROCKSDB_LIB_IO_POSIX", Some("1")); config.define("ROCKSDB_LIB_IO_POSIX", Some("1"));
} }
if cfg!(windows) { if cfg!(target_os = "windows") {
link("rpcrt4", false); link("rpcrt4", false);
link("shlwapi", false);
config.define("OS_WIN", Some("1")); config.define("OS_WIN", Some("1"));
// Remove POSIX-specific sources // Remove POSIX-specific sources
@ -106,6 +131,7 @@ fn build_rocksdb() {
lib_sources.push("port/win/env_default.cc"); lib_sources.push("port/win/env_default.cc");
lib_sources.push("port/win/win_logger.cc"); lib_sources.push("port/win/win_logger.cc");
lib_sources.push("port/win/io_win.cc"); lib_sources.push("port/win/io_win.cc");
lib_sources.push("port/win/win_thread.cc");
} }
if cfg!(target_env = "msvc") { if cfg!(target_env = "msvc") {
@ -148,9 +174,141 @@ fn build_snappy() {
config.compile("libsnappy.a"); config.compile("libsnappy.a");
} }
fn build_lz4() {
let mut compiler = cc::Build::new();
compiler
.file("lz4/lib/lz4.c")
.file("lz4/lib/lz4frame.c")
.file("lz4/lib/lz4hc.c")
.file("lz4/lib/xxhash.c");
compiler.opt_level(3);
match env::var("TARGET").unwrap().as_str()
{
"i686-pc-windows-gnu" => {
compiler.flag("-fno-tree-vectorize");
},
_ => {}
}
compiler.compile("liblz4.a");
}
fn build_zstd() {
let mut compiler = cc::Build::new();
compiler.include("zstd/lib/");
compiler.include("zstd/lib/common");
compiler.include("zstd/lib/legacy");
let globs = &[
"zstd/lib/common/*.c",
"zstd/lib/compress/*.c",
"zstd/lib/decompress/*.c",
"zstd/lib/dictBuilder/*.c",
"zstd/lib/legacy/*.c",
];
for pattern in globs {
for path in glob::glob(pattern).unwrap() {
let path = path.unwrap();
compiler.file(path);
}
}
compiler.opt_level(3);
compiler.define("ZSTD_LIB_DEPRECATED", Some("0"));
compiler.compile("libzstd.a");
}
fn build_zlib() {
let mut compiler = cc::Build::new();
let globs = &[
"zlib/*.c"
];
for pattern in globs {
for path in glob::glob(pattern).unwrap() {
let path = path.unwrap();
compiler.file(path);
}
}
compiler.opt_level(3);
compiler.compile("libz.a");
}
fn build_bzip2() {
let mut compiler = cc::Build::new();
compiler
.file("bzip2/blocksort.c")
.file("bzip2/bzlib.c")
.file("bzip2/compress.c")
.file("bzip2/crctable.c")
.file("bzip2/decompress.c")
.file("bzip2/huffman.c")
.file("bzip2/randtable.c");
compiler
.define("_FILE_OFFSET_BITS", Some("64"))
.define("BZ_NO_STDIO", None);
compiler.opt_level(3);
compiler.compile("libbz2.a");
}
fn try_to_find_and_link_lib(lib_name: &str) -> bool {
if let Ok(lib_dir) = env::var(&format!("{}_LIB_DIR", lib_name)) {
println!("cargo:rustc-link-search=native={}", lib_dir);
let mode = match env::var_os(&format!("{}_STATIC", lib_name)) {
Some(_) => "static",
None => "dylib",
};
println!("cargo:rustc-link-lib={}={}", mode, lib_name.to_lowercase());
return true;
}
false
}
fn main() { fn main() {
println!("cargo:rerun-if-changed=build.rs");
println!("cargo:rerun-if-changed=rocksdb/");
println!("cargo:rerun-if-changed=snappy/");
println!("cargo:rerun-if-changed=lz4/");
println!("cargo:rerun-if-changed=zstd/");
println!("cargo:rerun-if-changed=zlib/");
println!("cargo:rerun-if-changed=bzip2/");
fail_on_empty_directory("rocksdb"); fail_on_empty_directory("rocksdb");
fail_on_empty_directory("snappy"); fail_on_empty_directory("snappy");
fail_on_empty_directory("lz4");
fail_on_empty_directory("zstd");
fail_on_empty_directory("zlib");
fail_on_empty_directory("bzip2");
bindgen_rocksdb();
if !try_to_find_and_link_lib("ROCKSDB") {
build_rocksdb(); build_rocksdb();
}
if cfg!(feature = "snappy") && !try_to_find_and_link_lib("SNAPPY") {
build_snappy(); build_snappy();
} }
if cfg!(feature = "lz4") && !try_to_find_and_link_lib("LZ4") {
build_lz4();
}
if cfg!(feature = "zstd") && !try_to_find_and_link_lib("ZSTD") {
build_zstd();
}
if cfg!(feature = "zlib") && !try_to_find_and_link_lib("ZLIB") {
build_zlib();
}
if cfg!(feature = "bzip2") && !try_to_find_and_link_lib("BZIP2") {
build_bzip2();
}
}

42
librocksdb-sys/bzip2/LICENSE
Unescape View File

@ -0,0 +1,42 @@
--------------------------------------------------------------------------
This program, "bzip2", the associated library "libbzip2", and all
documentation, are copyright (C) 1996-2010 Julian R Seward. All
rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. The origin of this software must not be misrepresented; you must
not claim that you wrote the original software. If you use this
software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
3. Altered source versions must be plainly marked as such, and must
not be misrepresented as being the original software.
4. The name of the author may not be used to endorse or promote
products derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Julian Seward, jseward@bzip.org
bzip2/libbzip2 version 1.0.6 of 6 September 2010
--------------------------------------------------------------------------

1094
librocksdb-sys/bzip2/blocksort.c
View File

File diff suppressed because it is too large Load Diff

1572
librocksdb-sys/bzip2/bzlib.c
View File

File diff suppressed because it is too large Load Diff

282
librocksdb-sys/bzip2/bzlib.h
Unescape View File

@ -0,0 +1,282 @@
/*-------------------------------------------------------------*/
/*--- Public header file for the library. ---*/
/*--- bzlib.h ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
#ifndef _BZLIB_H
#define _BZLIB_H
#ifdef __cplusplus
extern "C" {
#endif
#define BZ_RUN 0
#define BZ_FLUSH 1
#define BZ_FINISH 2
#define BZ_OK 0
#define BZ_RUN_OK 1
#define BZ_FLUSH_OK 2
#define BZ_FINISH_OK 3
#define BZ_STREAM_END 4
#define BZ_SEQUENCE_ERROR (-1)
#define BZ_PARAM_ERROR (-2)
#define BZ_MEM_ERROR (-3)
#define BZ_DATA_ERROR (-4)
#define BZ_DATA_ERROR_MAGIC (-5)
#define BZ_IO_ERROR (-6)
#define BZ_UNEXPECTED_EOF (-7)
#define BZ_OUTBUFF_FULL (-8)
#define BZ_CONFIG_ERROR (-9)
typedef
struct {
char *next_in;
unsigned int avail_in;
unsigned int total_in_lo32;
unsigned int total_in_hi32;
char *next_out;
unsigned int avail_out;
unsigned int total_out_lo32;
unsigned int total_out_hi32;
void *state;
void *(*bzalloc)(void *,int,int);
void (*bzfree)(void *,void *);
void *opaque;
}
bz_stream;
#ifndef BZ_IMPORT
#define BZ_EXPORT
#endif
#ifndef BZ_NO_STDIO
/* Need a definitition for FILE */
#include <stdio.h>
#endif
#ifdef _WIN32
# include <windows.h>
# ifdef small
/* windows.h define small to char */
# undef small
# endif
# ifdef BZ_EXPORT
# define BZ_API(func) WINAPI func
# define BZ_EXTERN extern
# else
/* import windows dll dynamically */
# define BZ_API(func) (WINAPI * func)
# define BZ_EXTERN
# endif
#else
# define BZ_API(func) func
# define BZ_EXTERN extern
#endif
/*-- Core (low-level) library functions --*/
BZ_EXTERN int BZ_API(BZ2_bzCompressInit) (
bz_stream* strm,
int blockSize100k,
int verbosity,
int workFactor
);
BZ_EXTERN int BZ_API(BZ2_bzCompress) (
bz_stream* strm,
int action
);
BZ_EXTERN int BZ_API(BZ2_bzCompressEnd) (
bz_stream* strm
);
BZ_EXTERN int BZ_API(BZ2_bzDecompressInit) (
bz_stream *strm,
int verbosity,
int small
);
BZ_EXTERN int BZ_API(BZ2_bzDecompress) (
bz_stream* strm
);
BZ_EXTERN int BZ_API(BZ2_bzDecompressEnd) (
bz_stream *strm
);
/*-- High(er) level library functions --*/
#ifndef BZ_NO_STDIO
#define BZ_MAX_UNUSED 5000
typedef void BZFILE;
BZ_EXTERN BZFILE* BZ_API(BZ2_bzReadOpen) (
int* bzerror,
FILE* f,
int verbosity,
int small,
void* unused,
int nUnused
);
BZ_EXTERN void BZ_API(BZ2_bzReadClose) (
int* bzerror,
BZFILE* b
);
BZ_EXTERN void BZ_API(BZ2_bzReadGetUnused) (
int* bzerror,
BZFILE* b,
void** unused,
int* nUnused
);
BZ_EXTERN int BZ_API(BZ2_bzRead) (
int* bzerror,
BZFILE* b,
void* buf,
int len
);
BZ_EXTERN BZFILE* BZ_API(BZ2_bzWriteOpen) (
int* bzerror,
FILE* f,
int blockSize100k,
int verbosity,
int workFactor
);
BZ_EXTERN void BZ_API(BZ2_bzWrite) (
int* bzerror,
BZFILE* b,
void* buf,
int len
);
BZ_EXTERN void BZ_API(BZ2_bzWriteClose) (
int* bzerror,
BZFILE* b,
int abandon,
unsigned int* nbytes_in,
unsigned int* nbytes_out
);
BZ_EXTERN void BZ_API(BZ2_bzWriteClose64) (
int* bzerror,
BZFILE* b,
int abandon,
unsigned int* nbytes_in_lo32,
unsigned int* nbytes_in_hi32,
unsigned int* nbytes_out_lo32,
unsigned int* nbytes_out_hi32
);
#endif
/*-- Utility functions --*/
BZ_EXTERN int BZ_API(BZ2_bzBuffToBuffCompress) (
char* dest,
unsigned int* destLen,
char* source,
unsigned int sourceLen,
int blockSize100k,
int verbosity,
int workFactor
);
BZ_EXTERN int BZ_API(BZ2_bzBuffToBuffDecompress) (
char* dest,
unsigned int* destLen,
char* source,
unsigned int sourceLen,
int small,
int verbosity
);
/*--
Code contributed by Yoshioka Tsuneo (tsuneo@rr.iij4u.or.jp)
to support better zlib compatibility.
This code is not _officially_ part of libbzip2 (yet);
I haven't tested it, documented it, or considered the
threading-safeness of it.
If this code breaks, please contact both Yoshioka and me.
--*/
BZ_EXTERN const char * BZ_API(BZ2_bzlibVersion) (
void
);
#ifndef BZ_NO_STDIO
BZ_EXTERN BZFILE * BZ_API(BZ2_bzopen) (
const char *path,
const char *mode
);
BZ_EXTERN BZFILE * BZ_API(BZ2_bzdopen) (
int fd,
const char *mode
);
BZ_EXTERN int BZ_API(BZ2_bzread) (
BZFILE* b,
void* buf,
int len
);
BZ_EXTERN int BZ_API(BZ2_bzwrite) (
BZFILE* b,
void* buf,
int len
);
BZ_EXTERN int BZ_API(BZ2_bzflush) (
BZFILE* b
);
BZ_EXTERN void BZ_API(BZ2_bzclose) (
BZFILE* b
);
BZ_EXTERN const char * BZ_API(BZ2_bzerror) (
BZFILE *b,
int *errnum
);
#endif
#ifdef __cplusplus
}
#endif
#endif
/*-------------------------------------------------------------*/
/*--- end bzlib.h ---*/
/*-------------------------------------------------------------*/

509
librocksdb-sys/bzip2/bzlib_private.h
Unescape View File

@ -0,0 +1,509 @@
/*-------------------------------------------------------------*/
/*--- Private header file for the library. ---*/
/*--- bzlib_private.h ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
#ifndef _BZLIB_PRIVATE_H
#define _BZLIB_PRIVATE_H
#include <stdlib.h>
#ifndef BZ_NO_STDIO
#include <stdio.h>
#include <ctype.h>
#include <string.h>
#endif
#include "bzlib.h"
/*-- General stuff. --*/
#define BZ_VERSION "1.0.6, 6-Sept-2010"
typedef char Char;
typedef unsigned char Bool;
typedef unsigned char UChar;
typedef int Int32;
typedef unsigned int UInt32;
typedef short Int16;
typedef unsigned short UInt16;
#define True ((Bool)1)
#define False ((Bool)0)
#ifndef __GNUC__
#define __inline__ /* */
#endif
#ifndef BZ_NO_STDIO
extern void BZ2_bz__AssertH__fail ( int errcode );
#define AssertH(cond,errcode) \
{ if (!(cond)) BZ2_bz__AssertH__fail ( errcode ); }
#if BZ_DEBUG
#define AssertD(cond,msg) \
{ if (!(cond)) { \
fprintf ( stderr, \
"\n\nlibbzip2(debug build): internal error\n\t%s\n", msg );\
exit(1); \
}}
#else
#define AssertD(cond,msg) /* */
#endif
#define VPrintf0(zf) \
fprintf(stderr,zf)
#define VPrintf1(zf,za1) \
fprintf(stderr,zf,za1)
#define VPrintf2(zf,za1,za2) \
fprintf(stderr,zf,za1,za2)
#define VPrintf3(zf,za1,za2,za3) \
fprintf(stderr,zf,za1,za2,za3)
#define VPrintf4(zf,za1,za2,za3,za4) \
fprintf(stderr,zf,za1,za2,za3,za4)
#define VPrintf5(zf,za1,za2,za3,za4,za5) \
fprintf(stderr,zf,za1,za2,za3,za4,za5)
#else
extern void bz_internal_error ( int errcode );
#define AssertH(cond,errcode) \
{ if (!(cond)) bz_internal_error ( errcode ); }
#define AssertD(cond,msg) do { } while (0)
#define VPrintf0(zf) do { } while (0)
#define VPrintf1(zf,za1) do { } while (0)
#define VPrintf2(zf,za1,za2) do { } while (0)
#define VPrintf3(zf,za1,za2,za3) do { } while (0)
#define VPrintf4(zf,za1,za2,za3,za4) do { } while (0)
#define VPrintf5(zf,za1,za2,za3,za4,za5) do { } while (0)
#endif
#define BZALLOC(nnn) (strm->bzalloc)(strm->opaque,(nnn),1)
#define BZFREE(ppp) (strm->bzfree)(strm->opaque,(ppp))
/*-- Header bytes. --*/
#define BZ_HDR_B 0x42 /* 'B' */
#define BZ_HDR_Z 0x5a /* 'Z' */
#define BZ_HDR_h 0x68 /* 'h' */
#define BZ_HDR_0 0x30 /* '0' */
/*-- Constants for the back end. --*/
#define BZ_MAX_ALPHA_SIZE 258
#define BZ_MAX_CODE_LEN 23
#define BZ_RUNA 0
#define BZ_RUNB 1
#define BZ_N_GROUPS 6
#define BZ_G_SIZE 50
#define BZ_N_ITERS 4
#define BZ_MAX_SELECTORS (2 + (900000 / BZ_G_SIZE))
/*-- Stuff for randomising repetitive blocks. --*/
extern Int32 BZ2_rNums[512];
#define BZ_RAND_DECLS \
Int32 rNToGo; \
Int32 rTPos \
#define BZ_RAND_INIT_MASK \
s->rNToGo = 0; \
s->rTPos = 0 \
#define BZ_RAND_MASK ((s->rNToGo == 1) ? 1 : 0)
#define BZ_RAND_UPD_MASK \
if (s->rNToGo == 0) { \
s->rNToGo = BZ2_rNums[s->rTPos]; \
s->rTPos++; \
if (s->rTPos == 512) s->rTPos = 0; \
} \
s->rNToGo--;
/*-- Stuff for doing CRCs. --*/
extern UInt32 BZ2_crc32Table[256];
#define BZ_INITIALISE_CRC(crcVar) \
{ \
crcVar = 0xffffffffL; \
}
#define BZ_FINALISE_CRC(crcVar) \
{ \
crcVar = ~(crcVar); \
}
#define BZ_UPDATE_CRC(crcVar,cha) \
{ \
crcVar = (crcVar << 8) ^ \
BZ2_crc32Table[(crcVar >> 24) ^ \
((UChar)cha)]; \
}
/*-- States and modes for compression. --*/
#define BZ_M_IDLE 1
#define BZ_M_RUNNING 2
#define BZ_M_FLUSHING 3
#define BZ_M_FINISHING 4
#define BZ_S_OUTPUT 1
#define BZ_S_INPUT 2
#define BZ_N_RADIX 2
#define BZ_N_QSORT 12
#define BZ_N_SHELL 18
#define BZ_N_OVERSHOOT (BZ_N_RADIX + BZ_N_QSORT + BZ_N_SHELL + 2)
/*-- Structure holding all the compression-side stuff. --*/
typedef
struct {
/* pointer back to the struct bz_stream */
bz_stream* strm;
/* mode this stream is in, and whether inputting */
/* or outputting data */
Int32 mode;
Int32 state;
/* remembers avail_in when flush/finish requested */
UInt32 avail_in_expect;
/* for doing the block sorting */
UInt32* arr1;
UInt32* arr2;
UInt32* ftab;
Int32 origPtr;
/* aliases for arr1 and arr2 */
UInt32* ptr;
UChar* block;
UInt16* mtfv;
UChar* zbits;
/* for deciding when to use the fallback sorting algorithm */
Int32 workFactor;
/* run-length-encoding of the input */
UInt32 state_in_ch;
Int32 state_in_len;
BZ_RAND_DECLS;
/* input and output limits and current posns */
Int32 nblock;
Int32 nblockMAX;
Int32 numZ;
Int32 state_out_pos;
/* map of bytes used in block */
Int32 nInUse;
Bool inUse[256];
UChar unseqToSeq[256];
/* the buffer for bit stream creation */
UInt32 bsBuff;
Int32 bsLive;
/* block and combined CRCs */
UInt32 blockCRC;
UInt32 combinedCRC;
/* misc administratium */
Int32 verbosity;
Int32 blockNo;
Int32 blockSize100k;
/* stuff for coding the MTF values */
Int32 nMTF;
Int32 mtfFreq [BZ_MAX_ALPHA_SIZE];
UChar selector [BZ_MAX_SELECTORS];
UChar selectorMtf[BZ_MAX_SELECTORS];
UChar len [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 code [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 rfreq [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
/* second dimension: only 3 needed; 4 makes index calculations faster */
UInt32 len_pack[BZ_MAX_ALPHA_SIZE][4];
}
EState;
/*-- externs for compression. --*/
extern void
BZ2_blockSort ( EState* );
extern void
BZ2_compressBlock ( EState*, Bool );
extern void
BZ2_bsInitWrite ( EState* );
extern void
BZ2_hbAssignCodes ( Int32*, UChar*, Int32, Int32, Int32 );
extern void
BZ2_hbMakeCodeLengths ( UChar*, Int32*, Int32, Int32 );
/*-- states for decompression. --*/
#define BZ_X_IDLE 1
#define BZ_X_OUTPUT 2
#define BZ_X_MAGIC_1 10
#define BZ_X_MAGIC_2 11
#define BZ_X_MAGIC_3 12
#define BZ_X_MAGIC_4 13
#define BZ_X_BLKHDR_1 14
#define BZ_X_BLKHDR_2 15
#define BZ_X_BLKHDR_3 16
#define BZ_X_BLKHDR_4 17
#define BZ_X_BLKHDR_5 18
#define BZ_X_BLKHDR_6 19
#define BZ_X_BCRC_1 20
#define BZ_X_BCRC_2 21
#define BZ_X_BCRC_3 22
#define BZ_X_BCRC_4 23
#define BZ_X_RANDBIT 24
#define BZ_X_ORIGPTR_1 25
#define BZ_X_ORIGPTR_2 26
#define BZ_X_ORIGPTR_3 27
#define BZ_X_MAPPING_1 28
#define BZ_X_MAPPING_2 29
#define BZ_X_SELECTOR_1 30
#define BZ_X_SELECTOR_2 31
#define BZ_X_SELECTOR_3 32
#define BZ_X_CODING_1 33
#define BZ_X_CODING_2 34
#define BZ_X_CODING_3 35
#define BZ_X_MTF_1 36
#define BZ_X_MTF_2 37
#define BZ_X_MTF_3 38
#define BZ_X_MTF_4 39
#define BZ_X_MTF_5 40
#define BZ_X_MTF_6 41
#define BZ_X_ENDHDR_2 42
#define BZ_X_ENDHDR_3 43
#define BZ_X_ENDHDR_4 44
#define BZ_X_ENDHDR_5 45
#define BZ_X_ENDHDR_6 46
#define BZ_X_CCRC_1 47
#define BZ_X_CCRC_2 48
#define BZ_X_CCRC_3 49
#define BZ_X_CCRC_4 50
/*-- Constants for the fast MTF decoder. --*/
#define MTFA_SIZE 4096
#define MTFL_SIZE 16
/*-- Structure holding all the decompression-side stuff. --*/
typedef
struct {
/* pointer back to the struct bz_stream */
bz_stream* strm;
/* state indicator for this stream */
Int32 state;
/* for doing the final run-length decoding */
UChar state_out_ch;
Int32 state_out_len;
Bool blockRandomised;
BZ_RAND_DECLS;
/* the buffer for bit stream reading */
UInt32 bsBuff;
Int32 bsLive;
/* misc administratium */
Int32 blockSize100k;
Bool smallDecompress;
Int32 currBlockNo;
Int32 verbosity;
/* for undoing the Burrows-Wheeler transform */
Int32 origPtr;
UInt32 tPos;
Int32 k0;
Int32 unzftab[256];
Int32 nblock_used;
Int32 cftab[257];
Int32 cftabCopy[257];
/* for undoing the Burrows-Wheeler transform (FAST) */
UInt32 *tt;
/* for undoing the Burrows-Wheeler transform (SMALL) */
UInt16 *ll16;
UChar *ll4;
/* stored and calculated CRCs */
UInt32 storedBlockCRC;
UInt32 storedCombinedCRC;
UInt32 calculatedBlockCRC;
UInt32 calculatedCombinedCRC;
/* map of bytes used in block */
Int32 nInUse;
Bool inUse[256];
Bool inUse16[16];
UChar seqToUnseq[256];
/* for decoding the MTF values */
UChar mtfa [MTFA_SIZE];
Int32 mtfbase[256 / MTFL_SIZE];
UChar selector [BZ_MAX_SELECTORS];
UChar selectorMtf[BZ_MAX_SELECTORS];
UChar len [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 limit [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 base [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 perm [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 minLens[BZ_N_GROUPS];
/* save area for scalars in the main decompress code */
Int32 save_i;
Int32 save_j;
Int32 save_t;
Int32 save_alphaSize;
Int32 save_nGroups;
Int32 save_nSelectors;
Int32 save_EOB;
Int32 save_groupNo;
Int32 save_groupPos;
Int32 save_nextSym;
Int32 save_nblockMAX;
Int32 save_nblock;
Int32 save_es;
Int32 save_N;
Int32 save_curr;
Int32 save_zt;
Int32 save_zn;
Int32 save_zvec;
Int32 save_zj;
Int32 save_gSel;
Int32 save_gMinlen;
Int32* save_gLimit;
Int32* save_gBase;
Int32* save_gPerm;
}
DState;
/*-- Macros for decompression. --*/
#define BZ_GET_FAST(cccc) \
/* c_tPos is unsigned, hence test < 0 is pointless. */ \
if (s->tPos >= (UInt32)100000 * (UInt32)s->blockSize100k) return True; \
s->tPos = s->tt[s->tPos]; \
cccc = (UChar)(s->tPos & 0xff); \
s->tPos >>= 8;
#define BZ_GET_FAST_C(cccc) \
/* c_tPos is unsigned, hence test < 0 is pointless. */ \
if (c_tPos >= (UInt32)100000 * (UInt32)ro_blockSize100k) return True; \
c_tPos = c_tt[c_tPos]; \
cccc = (UChar)(c_tPos & 0xff); \
c_tPos >>= 8;
#define SET_LL4(i,n) \
{ if (((i) & 0x1) == 0) \
s->ll4[(i) >> 1] = (s->ll4[(i) >> 1] & 0xf0) | (n); else \
s->ll4[(i) >> 1] = (s->ll4[(i) >> 1] & 0x0f) | ((n) << 4); \
}
#define GET_LL4(i) \
((((UInt32)(s->ll4[(i) >> 1])) >> (((i) << 2) & 0x4)) & 0xF)
#define SET_LL(i,n) \
{ s->ll16[i] = (UInt16)(n & 0x0000ffff); \
SET_LL4(i, n >> 16); \
}
#define GET_LL(i) \
(((UInt32)s->ll16[i]) | (GET_LL4(i) << 16))
#define BZ_GET_SMALL(cccc) \
/* c_tPos is unsigned, hence test < 0 is pointless. */ \
if (s->tPos >= (UInt32)100000 * (UInt32)s->blockSize100k) return True; \
cccc = BZ2_indexIntoF ( s->tPos, s->cftab ); \
s->tPos = GET_LL(s->tPos);
/*-- externs for decompression. --*/
extern Int32
BZ2_indexIntoF ( Int32, Int32* );
extern Int32
BZ2_decompress ( DState* );
extern void
BZ2_hbCreateDecodeTables ( Int32*, Int32*, Int32*, UChar*,
Int32, Int32, Int32 );
#endif
/*-- BZ_NO_STDIO seems to make NULL disappear on some platforms. --*/
#ifdef BZ_NO_STDIO
#ifndef NULL
#define NULL 0
#endif
#endif
/*-------------------------------------------------------------*/
/*--- end bzlib_private.h ---*/
/*-------------------------------------------------------------*/

672
librocksdb-sys/bzip2/compress.c
Unescape View File

@ -0,0 +1,672 @@
/*-------------------------------------------------------------*/
/*--- Compression machinery (not incl block sorting) ---*/
/*--- compress.c ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
/* CHANGES
0.9.0 -- original version.
0.9.0a/b -- no changes in this file.
0.9.0c -- changed setting of nGroups in sendMTFValues()
so as to do a bit better on small files
*/
#include "bzlib_private.h"
/*---------------------------------------------------*/
/*--- Bit stream I/O ---*/
/*---------------------------------------------------*/
/*---------------------------------------------------*/
void BZ2_bsInitWrite ( EState* s )
{
s->bsLive = 0;
s->bsBuff = 0;
}
/*---------------------------------------------------*/
static
void bsFinishWrite ( EState* s )
{
while (s->bsLive > 0) {
s->zbits[s->numZ] = (UChar)(s->bsBuff >> 24);
s->numZ++;
s->bsBuff <<= 8;
s->bsLive -= 8;
}
}
/*---------------------------------------------------*/
#define bsNEEDW(nz) \
{ \
while (s->bsLive >= 8) { \
s->zbits[s->numZ] \
= (UChar)(s->bsBuff >> 24); \
s->numZ++; \
s->bsBuff <<= 8; \
s->bsLive -= 8; \
} \
}
/*---------------------------------------------------*/
static
__inline__
void bsW ( EState* s, Int32 n, UInt32 v )
{
bsNEEDW ( n );
s->bsBuff |= (v << (32 - s->bsLive - n));
s->bsLive += n;
}
/*---------------------------------------------------*/
static
void bsPutUInt32 ( EState* s, UInt32 u )
{
bsW ( s, 8, (u >> 24) & 0xffL );
bsW ( s, 8, (u >> 16) & 0xffL );
bsW ( s, 8, (u >> 8) & 0xffL );
bsW ( s, 8, u & 0xffL );
}
/*---------------------------------------------------*/
static
void bsPutUChar ( EState* s, UChar c )
{
bsW( s, 8, (UInt32)c );
}
/*---------------------------------------------------*/
/*--- The back end proper ---*/
/*---------------------------------------------------*/
/*---------------------------------------------------*/
static
void makeMaps_e ( EState* s )
{
Int32 i;
s->nInUse = 0;
for (i = 0; i < 256; i++)
if (s->inUse[i]) {
s->unseqToSeq[i] = s->nInUse;
s->nInUse++;
}
}
/*---------------------------------------------------*/
static
void generateMTFValues ( EState* s )
{
UChar yy[256];
Int32 i, j;
Int32 zPend;
Int32 wr;
Int32 EOB;
/*
After sorting (eg, here),
s->arr1 [ 0 .. s->nblock-1 ] holds sorted order,
and
((UChar*)s->arr2) [ 0 .. s->nblock-1 ]
holds the original block data.
The first thing to do is generate the MTF values,
and put them in
((UInt16*)s->arr1) [ 0 .. s->nblock-1 ].
Because there are strictly fewer or equal MTF values
than block values, ptr values in this area are overwritten
with MTF values only when they are no longer needed.
The final compressed bitstream is generated into the
area starting at
(UChar*) (&((UChar*)s->arr2)[s->nblock])
These storage aliases are set up in bzCompressInit(),
except for the last one, which is arranged in
compressBlock().
*/
UInt32* ptr = s->ptr;
UChar* block = s->block;
UInt16* mtfv = s->mtfv;
makeMaps_e ( s );
EOB = s->nInUse+1;
for (i = 0; i <= EOB; i++) s->mtfFreq[i] = 0;
wr = 0;
zPend = 0;
for (i = 0; i < s->nInUse; i++) yy[i] = (UChar) i;
for (i = 0; i < s->nblock; i++) {
UChar ll_i;
AssertD ( wr <= i, "generateMTFValues(1)" );
j = ptr[i]-1; if (j < 0) j += s->nblock;
ll_i = s->unseqToSeq[block[j]];
AssertD ( ll_i < s->nInUse, "generateMTFValues(2a)" );
if (yy[0] == ll_i) {
zPend++;
} else {
if (zPend > 0) {
zPend--;
while (True) {
if (zPend & 1) {
mtfv[wr] = BZ_RUNB; wr++;
s->mtfFreq[BZ_RUNB]++;
} else {
mtfv[wr] = BZ_RUNA; wr++;
s->mtfFreq[BZ_RUNA]++;
}
if (zPend < 2) break;
zPend = (zPend - 2) / 2;
};
zPend = 0;
}
{
register UChar rtmp;
register UChar* ryy_j;
register UChar rll_i;
rtmp = yy[1];
yy[1] = yy[0];
ryy_j = &(yy[1]);
rll_i = ll_i;
while ( rll_i != rtmp ) {
register UChar rtmp2;
ryy_j++;
rtmp2 = rtmp;
rtmp = *ryy_j;
*ryy_j = rtmp2;
};
yy[0] = rtmp;
j = ryy_j - &(yy[0]);
mtfv[wr] = j+1; wr++; s->mtfFreq[j+1]++;
}
}
}
if (zPend > 0) {
zPend--;
while (True) {
if (zPend & 1) {
mtfv[wr] = BZ_RUNB; wr++;
s->mtfFreq[BZ_RUNB]++;
} else {
mtfv[wr] = BZ_RUNA; wr++;
s->mtfFreq[BZ_RUNA]++;
}
if (zPend < 2) break;
zPend = (zPend - 2) / 2;
};
zPend = 0;
}
mtfv[wr] = EOB; wr++; s->mtfFreq[EOB]++;
s->nMTF = wr;
}
/*---------------------------------------------------*/
#define BZ_LESSER_ICOST 0
#define BZ_GREATER_ICOST 15
static
void sendMTFValues ( EState* s )
{
Int32 v, t, i, j, gs, ge, totc, bt, bc, iter;
Int32 nSelectors, alphaSize, minLen, maxLen, selCtr;
Int32 nGroups, nBytes;
/*--
UChar len [BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
is a global since the decoder also needs it.
Int32 code[BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
Int32 rfreq[BZ_N_GROUPS][BZ_MAX_ALPHA_SIZE];
are also globals only used in this proc.
Made global to keep stack frame size small.
--*/
UInt16 cost[BZ_N_GROUPS];
Int32 fave[BZ_N_GROUPS];
UInt16* mtfv = s->mtfv;
if (s->verbosity >= 3)
VPrintf3( " %d in block, %d after MTF & 1-2 coding, "
"%d+2 syms in use\n",
s->nblock, s->nMTF, s->nInUse );
alphaSize = s->nInUse+2;
for (t = 0; t < BZ_N_GROUPS; t++)
for (v = 0; v < alphaSize; v++)
s->len[t][v] = BZ_GREATER_ICOST;
/*--- Decide how many coding tables to use ---*/
AssertH ( s->nMTF > 0, 3001 );
if (s->nMTF < 200) nGroups = 2; else
if (s->nMTF < 600) nGroups = 3; else
if (s->nMTF < 1200) nGroups = 4; else
if (s->nMTF < 2400) nGroups = 5; else
nGroups = 6;
/*--- Generate an initial set of coding tables ---*/
{
Int32 nPart, remF, tFreq, aFreq;
nPart = nGroups;
remF = s->nMTF;
gs = 0;
while (nPart > 0) {
tFreq = remF / nPart;
ge = gs-1;
aFreq = 0;
while (aFreq < tFreq && ge < alphaSize-1) {
ge++;
aFreq += s->mtfFreq[ge];
}
if (ge > gs
&& nPart != nGroups && nPart != 1
&& ((nGroups-nPart) % 2 == 1)) {
aFreq -= s->mtfFreq[ge];
ge--;
}
if (s->verbosity >= 3)
VPrintf5( " initial group %d, [%d .. %d], "
"has %d syms (%4.1f%%)\n",
nPart, gs, ge, aFreq,
(100.0 * (float)aFreq) / (float)(s->nMTF) );
for (v = 0; v < alphaSize; v++)
if (v >= gs && v <= ge)
s->len[nPart-1][v] = BZ_LESSER_ICOST; else
s->len[nPart-1][v] = BZ_GREATER_ICOST;
nPart--;
gs = ge+1;
remF -= aFreq;
}
}
/*---
Iterate up to BZ_N_ITERS times to improve the tables.
---*/
for (iter = 0; iter < BZ_N_ITERS; iter++) {
for (t = 0; t < nGroups; t++) fave[t] = 0;
for (t = 0; t < nGroups; t++)
for (v = 0; v < alphaSize; v++)
s->rfreq[t][v] = 0;
/*---
Set up an auxiliary length table which is used to fast-track
the common case (nGroups == 6).
---*/
if (nGroups == 6) {
for (v = 0; v < alphaSize; v++) {
s->len_pack[v][0] = (s->len[1][v] << 16) | s->len[0][v];
s->len_pack[v][1] = (s->len[3][v] << 16) | s->len[2][v];
s->len_pack[v][2] = (s->len[5][v] << 16) | s->len[4][v];
}
}
nSelectors = 0;
totc = 0;
gs = 0;
while (True) {
/*--- Set group start & end marks. --*/
if (gs >= s->nMTF) break;
ge = gs + BZ_G_SIZE - 1;
if (ge >= s->nMTF) ge = s->nMTF-1;
/*--
Calculate the cost of this group as coded
by each of the coding tables.
--*/
for (t = 0; t < nGroups; t++) cost[t] = 0;
if (nGroups == 6 && 50 == ge-gs+1) {
/*--- fast track the common case ---*/
register UInt32 cost01, cost23, cost45;
register UInt16 icv;
cost01 = cost23 = cost45 = 0;
# define BZ_ITER(nn) \
icv = mtfv[gs+(nn)]; \
cost01 += s->len_pack[icv][0]; \
cost23 += s->len_pack[icv][1]; \
cost45 += s->len_pack[icv][2]; \
BZ_ITER(0); BZ_ITER(1); BZ_ITER(2); BZ_ITER(3); BZ_ITER(4);
BZ_ITER(5); BZ_ITER(6); BZ_ITER(7); BZ_ITER(8); BZ_ITER(9);
BZ_ITER(10); BZ_ITER(11); BZ_ITER(12); BZ_ITER(13); BZ_ITER(14);
BZ_ITER(15); BZ_ITER(16); BZ_ITER(17); BZ_ITER(18); BZ_ITER(19);
BZ_ITER(20); BZ_ITER(21); BZ_ITER(22); BZ_ITER(23); BZ_ITER(24);
BZ_ITER(25); BZ_ITER(26); BZ_ITER(27); BZ_ITER(28); BZ_ITER(29);
BZ_ITER(30); BZ_ITER(31); BZ_ITER(32); BZ_ITER(33); BZ_ITER(34);
BZ_ITER(35); BZ_ITER(36); BZ_ITER(37); BZ_ITER(38); BZ_ITER(39);
BZ_ITER(40); BZ_ITER(41); BZ_ITER(42); BZ_ITER(43); BZ_ITER(44);
BZ_ITER(45); BZ_ITER(46); BZ_ITER(47); BZ_ITER(48); BZ_ITER(49);
# undef BZ_ITER
cost[0] = cost01 & 0xffff; cost[1] = cost01 >> 16;
cost[2] = cost23 & 0xffff; cost[3] = cost23 >> 16;
cost[4] = cost45 & 0xffff; cost[5] = cost45 >> 16;
} else {
/*--- slow version which correctly handles all situations ---*/
for (i = gs; i <= ge; i++) {
UInt16 icv = mtfv[i];
for (t = 0; t < nGroups; t++) cost[t] += s->len[t][icv];
}
}
/*--
Find the coding table which is best for this group,
and record its identity in the selector table.
--*/
bc = 999999999; bt = -1;
for (t = 0; t < nGroups; t++)
if (cost[t] < bc) { bc = cost[t]; bt = t; };
totc += bc;
fave[bt]++;
s->selector[nSelectors] = bt;
nSelectors++;
/*--
Increment the symbol frequencies for the selected table.
--*/
if (nGroups == 6 && 50 == ge-gs+1) {
/*--- fast track the common case ---*/
# define BZ_ITUR(nn) s->rfreq[bt][ mtfv[gs+(nn)] ]++
BZ_ITUR(0); BZ_ITUR(1); BZ_ITUR(2); BZ_ITUR(3); BZ_ITUR(4);
BZ_ITUR(5); BZ_ITUR(6); BZ_ITUR(7); BZ_ITUR(8); BZ_ITUR(9);
BZ_ITUR(10); BZ_ITUR(11); BZ_ITUR(12); BZ_ITUR(13); BZ_ITUR(14);
BZ_ITUR(15); BZ_ITUR(16); BZ_ITUR(17); BZ_ITUR(18); BZ_ITUR(19);
BZ_ITUR(20); BZ_ITUR(21); BZ_ITUR(22); BZ_ITUR(23); BZ_ITUR(24);
BZ_ITUR(25); BZ_ITUR(26); BZ_ITUR(27); BZ_ITUR(28); BZ_ITUR(29);
BZ_ITUR(30); BZ_ITUR(31); BZ_ITUR(32); BZ_ITUR(33); BZ_ITUR(34);
BZ_ITUR(35); BZ_ITUR(36); BZ_ITUR(37); BZ_ITUR(38); BZ_ITUR(39);
BZ_ITUR(40); BZ_ITUR(41); BZ_ITUR(42); BZ_ITUR(43); BZ_ITUR(44);
BZ_ITUR(45); BZ_ITUR(46); BZ_ITUR(47); BZ_ITUR(48); BZ_ITUR(49);
# undef BZ_ITUR
} else {
/*--- slow version which correctly handles all situations ---*/
for (i = gs; i <= ge; i++)
s->rfreq[bt][ mtfv[i] ]++;
}
gs = ge+1;
}
if (s->verbosity >= 3) {
VPrintf2 ( " pass %d: size is %d, grp uses are ",
iter+1, totc/8 );
for (t = 0; t < nGroups; t++)
VPrintf1 ( "%d ", fave[t] );
VPrintf0 ( "\n" );
}
/*--
Recompute the tables based on the accumulated frequencies.
--*/
/* maxLen was changed from 20 to 17 in bzip2-1.0.3. See
comment in huffman.c for details. */
for (t = 0; t < nGroups; t++)
BZ2_hbMakeCodeLengths ( &(s->len[t][0]), &(s->rfreq[t][0]),
alphaSize, 17 /*20*/ );
}
AssertH( nGroups < 8, 3002 );
AssertH( nSelectors < 32768 &&
nSelectors <= (2 + (900000 / BZ_G_SIZE)),
3003 );
/*--- Compute MTF values for the selectors. ---*/
{
UChar pos[BZ_N_GROUPS], ll_i, tmp2, tmp;
for (i = 0; i < nGroups; i++) pos[i] = i;
for (i = 0; i < nSelectors; i++) {
ll_i = s->selector[i];
j = 0;
tmp = pos[j];
while ( ll_i != tmp ) {
j++;
tmp2 = tmp;
tmp = pos[j];
pos[j] = tmp2;
};
pos[0] = tmp;
s->selectorMtf[i] = j;
}
};
/*--- Assign actual codes for the tables. --*/
for (t = 0; t < nGroups; t++) {
minLen = 32;
maxLen = 0;
for (i = 0; i < alphaSize; i++) {
if (s->len[t][i] > maxLen) maxLen = s->len[t][i];
if (s->len[t][i] < minLen) minLen = s->len[t][i];
}
AssertH ( !(maxLen > 17 /*20*/ ), 3004 );
AssertH ( !(minLen < 1), 3005 );
BZ2_hbAssignCodes ( &(s->code[t][0]), &(s->len[t][0]),
minLen, maxLen, alphaSize );
}
/*--- Transmit the mapping table. ---*/
{
Bool inUse16[16];
for (i = 0; i < 16; i++) {
inUse16[i] = False;
for (j = 0; j < 16; j++)
if (s->inUse[i * 16 + j]) inUse16[i] = True;
}
nBytes = s->numZ;
for (i = 0; i < 16; i++)
if (inUse16[i]) bsW(s,1,1); else bsW(s,1,0);
for (i = 0; i < 16; i++)
if (inUse16[i])
for (j = 0; j < 16; j++) {
if (s->inUse[i * 16 + j]) bsW(s,1,1); else bsW(s,1,0);
}
if (s->verbosity >= 3)
VPrintf1( " bytes: mapping %d, ", s->numZ-nBytes );
}
/*--- Now the selectors. ---*/
nBytes = s->numZ;
bsW ( s, 3, nGroups );
bsW ( s, 15, nSelectors );
for (i = 0; i < nSelectors; i++) {
for (j = 0; j < s->selectorMtf[i]; j++) bsW(s,1,1);
bsW(s,1,0);
}
if (s->verbosity >= 3)
VPrintf1( "selectors %d, ", s->numZ-nBytes );
/*--- Now the coding tables. ---*/
nBytes = s->numZ;
for (t = 0; t < nGroups; t++) {
Int32 curr = s->len[t][0];
bsW ( s, 5, curr );
for (i = 0; i < alphaSize; i++) {
while (curr < s->len[t][i]) { bsW(s,2,2); curr++; /* 10 */ };
while (curr > s->len[t][i]) { bsW(s,2,3); curr--; /* 11 */ };
bsW ( s, 1, 0 );
}
}
if (s->verbosity >= 3)
VPrintf1 ( "code lengths %d, ", s->numZ-nBytes );
/*--- And finally, the block data proper ---*/
nBytes = s->numZ;
selCtr = 0;
gs = 0;
while (True) {
if (gs >= s->nMTF) break;
ge = gs + BZ_G_SIZE - 1;
if (ge >= s->nMTF) ge = s->nMTF-1;
AssertH ( s->selector[selCtr] < nGroups, 3006 );
if (nGroups == 6 && 50 == ge-gs+1) {
/*--- fast track the common case ---*/
UInt16 mtfv_i;
UChar* s_len_sel_selCtr
= &(s->len[s->selector[selCtr]][0]);
Int32* s_code_sel_selCtr
= &(s->code[s->selector[selCtr]][0]);
# define BZ_ITAH(nn) \
mtfv_i = mtfv[gs+(nn)]; \
bsW ( s, \
s_len_sel_selCtr[mtfv_i], \
s_code_sel_selCtr[mtfv_i] )
BZ_ITAH(0); BZ_ITAH(1); BZ_ITAH(2); BZ_ITAH(3); BZ_ITAH(4);
BZ_ITAH(5); BZ_ITAH(6); BZ_ITAH(7); BZ_ITAH(8); BZ_ITAH(9);
BZ_ITAH(10); BZ_ITAH(11); BZ_ITAH(12); BZ_ITAH(13); BZ_ITAH(14);
BZ_ITAH(15); BZ_ITAH(16); BZ_ITAH(17); BZ_ITAH(18); BZ_ITAH(19);
BZ_ITAH(20); BZ_ITAH(21); BZ_ITAH(22); BZ_ITAH(23); BZ_ITAH(24);
BZ_ITAH(25); BZ_ITAH(26); BZ_ITAH(27); BZ_ITAH(28); BZ_ITAH(29);
BZ_ITAH(30); BZ_ITAH(31); BZ_ITAH(32); BZ_ITAH(33); BZ_ITAH(34);
BZ_ITAH(35); BZ_ITAH(36); BZ_ITAH(37); BZ_ITAH(38); BZ_ITAH(39);
BZ_ITAH(40); BZ_ITAH(41); BZ_ITAH(42); BZ_ITAH(43); BZ_ITAH(44);
BZ_ITAH(45); BZ_ITAH(46); BZ_ITAH(47); BZ_ITAH(48); BZ_ITAH(49);
# undef BZ_ITAH
} else {
/*--- slow version which correctly handles all situations ---*/
for (i = gs; i <= ge; i++) {
bsW ( s,
s->len [s->selector[selCtr]] [mtfv[i]],
s->code [s->selector[selCtr]] [mtfv[i]] );
}
}
gs = ge+1;
selCtr++;
}
AssertH( selCtr == nSelectors, 3007 );
if (s->verbosity >= 3)
VPrintf1( "codes %d\n", s->numZ-nBytes );
}
/*---------------------------------------------------*/
void BZ2_compressBlock ( EState* s, Bool is_last_block )
{
if (s->nblock > 0) {
BZ_FINALISE_CRC ( s->blockCRC );
s->combinedCRC = (s->combinedCRC << 1) | (s->combinedCRC >> 31);
s->combinedCRC ^= s->blockCRC;
if (s->blockNo > 1) s->numZ = 0;
if (s->verbosity >= 2)
VPrintf4( " block %d: crc = 0x%08x, "
"combined CRC = 0x%08x, size = %d\n",
s->blockNo, s->blockCRC, s->combinedCRC, s->nblock );
BZ2_blockSort ( s );
}
s->zbits = (UChar*) (&((UChar*)s->arr2)[s->nblock]);
/*-- If this is the first block, create the stream header. --*/
if (s->blockNo == 1) {
BZ2_bsInitWrite ( s );
bsPutUChar ( s, BZ_HDR_B );
bsPutUChar ( s, BZ_HDR_Z );
bsPutUChar ( s, BZ_HDR_h );
bsPutUChar ( s, (UChar)(BZ_HDR_0 + s->blockSize100k) );
}
if (s->nblock > 0) {
bsPutUChar ( s, 0x31 ); bsPutUChar ( s, 0x41 );
bsPutUChar ( s, 0x59 ); bsPutUChar ( s, 0x26 );
bsPutUChar ( s, 0x53 ); bsPutUChar ( s, 0x59 );
/*-- Now the block's CRC, so it is in a known place. --*/
bsPutUInt32 ( s, s->blockCRC );
/*--
Now a single bit indicating (non-)randomisation.
As of version 0.9.5, we use a better sorting algorithm
which makes randomisation unnecessary. So always set
the randomised bit to 'no'. Of course, the decoder
still needs to be able to handle randomised blocks
so as to maintain backwards compatibility with
older versions of bzip2.
--*/
bsW(s,1,0);
bsW ( s, 24, s->origPtr );
generateMTFValues ( s );
sendMTFValues ( s );
}
/*-- If this is the last block, add the stream trailer. --*/
if (is_last_block) {
bsPutUChar ( s, 0x17 ); bsPutUChar ( s, 0x72 );
bsPutUChar ( s, 0x45 ); bsPutUChar ( s, 0x38 );
bsPutUChar ( s, 0x50 ); bsPutUChar ( s, 0x90 );
bsPutUInt32 ( s, s->combinedCRC );
if (s->verbosity >= 2)
VPrintf1( " final combined CRC = 0x%08x\n ", s->combinedCRC );
bsFinishWrite ( s );
}
}
/*-------------------------------------------------------------*/
/*--- end compress.c ---*/
/*-------------------------------------------------------------*/

104
librocksdb-sys/bzip2/crctable.c
Unescape View File

@ -0,0 +1,104 @@
/*-------------------------------------------------------------*/
/*--- Table for doing CRCs ---*/
/*--- crctable.c ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
#include "bzlib_private.h"
/*--
I think this is an implementation of the AUTODIN-II,
Ethernet & FDDI 32-bit CRC standard. Vaguely derived
from code by Rob Warnock, in Section 51 of the
comp.compression FAQ.
--*/
UInt32 BZ2_crc32Table[256] = {
/*-- Ugly, innit? --*/
0x00000000L, 0x04c11db7L, 0x09823b6eL, 0x0d4326d9L,
0x130476dcL, 0x17c56b6bL, 0x1a864db2L, 0x1e475005L,
0x2608edb8L, 0x22c9f00fL, 0x2f8ad6d6L, 0x2b4bcb61L,
0x350c9b64L, 0x31cd86d3L, 0x3c8ea00aL, 0x384fbdbdL,
0x4c11db70L, 0x48d0c6c7L, 0x4593e01eL, 0x4152fda9L,
0x5f15adacL, 0x5bd4b01bL, 0x569796c2L, 0x52568b75L,
0x6a1936c8L, 0x6ed82b7fL, 0x639b0da6L, 0x675a1011L,
0x791d4014L, 0x7ddc5da3L, 0x709f7b7aL, 0x745e66cdL,
0x9823b6e0L, 0x9ce2ab57L, 0x91a18d8eL, 0x95609039L,
0x8b27c03cL, 0x8fe6dd8bL, 0x82a5fb52L, 0x8664e6e5L,
0xbe2b5b58L, 0xbaea46efL, 0xb7a96036L, 0xb3687d81L,
0xad2f2d84L, 0xa9ee3033L, 0xa4ad16eaL, 0xa06c0b5dL,
0xd4326d90L, 0xd0f37027L, 0xddb056feL, 0xd9714b49L,
0xc7361b4cL, 0xc3f706fbL, 0xceb42022L, 0xca753d95L,
0xf23a8028L, 0xf6fb9d9fL, 0xfbb8bb46L, 0xff79a6f1L,
0xe13ef6f4L, 0xe5ffeb43L, 0xe8bccd9aL, 0xec7dd02dL,
0x34867077L, 0x30476dc0L, 0x3d044b19L, 0x39c556aeL,
0x278206abL, 0x23431b1cL, 0x2e003dc5L, 0x2ac12072L,
0x128e9dcfL, 0x164f8078L, 0x1b0ca6a1L, 0x1fcdbb16L,
0x018aeb13L, 0x054bf6a4L, 0x0808d07dL, 0x0cc9cdcaL,
0x7897ab07L, 0x7c56b6b0L, 0x71159069L, 0x75d48ddeL,
0x6b93dddbL, 0x6f52c06cL, 0x6211e6b5L, 0x66d0fb02L,
0x5e9f46bfL, 0x5a5e5b08L, 0x571d7dd1L, 0x53dc6066L,
0x4d9b3063L, 0x495a2dd4L, 0x44190b0dL, 0x40d816baL,
0xaca5c697L, 0xa864db20L, 0xa527fdf9L, 0xa1e6e04eL,
0xbfa1b04bL, 0xbb60adfcL, 0xb6238b25L, 0xb2e29692L,
0x8aad2b2fL, 0x8e6c3698L, 0x832f1041L, 0x87ee0df6L,
0x99a95df3L, 0x9d684044L, 0x902b669dL, 0x94ea7b2aL,
0xe0b41de7L, 0xe4750050L, 0xe9362689L, 0xedf73b3eL,
0xf3b06b3bL, 0xf771768cL, 0xfa325055L, 0xfef34de2L,
0xc6bcf05fL, 0xc27dede8L, 0xcf3ecb31L, 0xcbffd686L,
0xd5b88683L, 0xd1799b34L, 0xdc3abdedL, 0xd8fba05aL,
0x690ce0eeL, 0x6dcdfd59L, 0x608edb80L, 0x644fc637L,
0x7a089632L, 0x7ec98b85L, 0x738aad5cL, 0x774bb0ebL,
0x4f040d56L, 0x4bc510e1L, 0x46863638L, 0x42472b8fL,
0x5c007b8aL, 0x58c1663dL, 0x558240e4L, 0x51435d53L,
0x251d3b9eL, 0x21dc2629L, 0x2c9f00f0L, 0x285e1d47L,
0x36194d42L, 0x32d850f5L, 0x3f9b762cL, 0x3b5a6b9bL,
0x0315d626L, 0x07d4cb91L, 0x0a97ed48L, 0x0e56f0ffL,
0x1011a0faL, 0x14d0bd4dL, 0x19939b94L, 0x1d528623L,
0xf12f560eL, 0xf5ee4bb9L, 0xf8ad6d60L, 0xfc6c70d7L,
0xe22b20d2L, 0xe6ea3d65L, 0xeba91bbcL, 0xef68060bL,
0xd727bbb6L, 0xd3e6a601L, 0xdea580d8L, 0xda649d6fL,
0xc423cd6aL, 0xc0e2d0ddL, 0xcda1f604L, 0xc960ebb3L,
0xbd3e8d7eL, 0xb9ff90c9L, 0xb4bcb610L, 0xb07daba7L,
0xae3afba2L, 0xaafbe615L, 0xa7b8c0ccL, 0xa379dd7bL,
0x9b3660c6L, 0x9ff77d71L, 0x92b45ba8L, 0x9675461fL,
0x8832161aL, 0x8cf30badL, 0x81b02d74L, 0x857130c3L,
0x5d8a9099L, 0x594b8d2eL, 0x5408abf7L, 0x50c9b640L,
0x4e8ee645L, 0x4a4ffbf2L, 0x470cdd2bL, 0x43cdc09cL,
0x7b827d21L, 0x7f436096L, 0x7200464fL, 0x76c15bf8L,
0x68860bfdL, 0x6c47164aL, 0x61043093L, 0x65c52d24L,
0x119b4be9L, 0x155a565eL, 0x18197087L, 0x1cd86d30L,
0x029f3d35L, 0x065e2082L, 0x0b1d065bL, 0x0fdc1becL,
0x3793a651L, 0x3352bbe6L, 0x3e119d3fL, 0x3ad08088L,
0x2497d08dL, 0x2056cd3aL, 0x2d15ebe3L, 0x29d4f654L,
0xc5a92679L, 0xc1683bceL, 0xcc2b1d17L, 0xc8ea00a0L,
0xd6ad50a5L, 0xd26c4d12L, 0xdf2f6bcbL, 0xdbee767cL,
0xe3a1cbc1L, 0xe760d676L, 0xea23f0afL, 0xeee2ed18L,
0xf0a5bd1dL, 0xf464a0aaL, 0xf9278673L, 0xfde69bc4L,
0x89b8fd09L, 0x8d79e0beL, 0x803ac667L, 0x84fbdbd0L,
0x9abc8bd5L, 0x9e7d9662L, 0x933eb0bbL, 0x97ffad0cL,
0xafb010b1L, 0xab710d06L, 0xa6322bdfL, 0xa2f33668L,
0xbcb4666dL, 0xb8757bdaL, 0xb5365d03L, 0xb1f740b4L
};
/*-------------------------------------------------------------*/
/*--- end crctable.c ---*/
/*-------------------------------------------------------------*/

646
librocksdb-sys/bzip2/decompress.c
Unescape View File

@ -0,0 +1,646 @@
/*-------------------------------------------------------------*/
/*--- Decompression machinery ---*/
/*--- decompress.c ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
#include "bzlib_private.h"
/*---------------------------------------------------*/
static
void makeMaps_d ( DState* s )
{
Int32 i;
s->nInUse = 0;
for (i = 0; i < 256; i++)
if (s->inUse[i]) {
s->seqToUnseq[s->nInUse] = i;
s->nInUse++;
}
}
/*---------------------------------------------------*/
#define RETURN(rrr) \
{ retVal = rrr; goto save_state_and_return; };
#define GET_BITS(lll,vvv,nnn) \
case lll: s->state = lll; \
while (True) { \
if (s->bsLive >= nnn) { \
UInt32 v; \
v = (s->bsBuff >> \
(s->bsLive-nnn)) & ((1 << nnn)-1); \
s->bsLive -= nnn; \
vvv = v; \
break; \
} \
if (s->strm->avail_in == 0) RETURN(BZ_OK); \
s->bsBuff \
= (s->bsBuff << 8) | \
((UInt32) \
(*((UChar*)(s->strm->next_in)))); \
s->bsLive += 8; \
s->strm->next_in++; \
s->strm->avail_in--; \
s->strm->total_in_lo32++; \
if (s->strm->total_in_lo32 == 0) \
s->strm->total_in_hi32++; \
}
#define GET_UCHAR(lll,uuu) \
GET_BITS(lll,uuu,8)
#define GET_BIT(lll,uuu) \
GET_BITS(lll,uuu,1)
/*---------------------------------------------------*/
#define GET_MTF_VAL(label1,label2,lval) \
{ \
if (groupPos == 0) { \
groupNo++; \
if (groupNo >= nSelectors) \
RETURN(BZ_DATA_ERROR); \
groupPos = BZ_G_SIZE; \
gSel = s->selector[groupNo]; \
gMinlen = s->minLens[gSel]; \
gLimit = &(s->limit[gSel][0]); \
gPerm = &(s->perm[gSel][0]); \
gBase = &(s->base[gSel][0]); \
} \
groupPos--; \
zn = gMinlen; \
GET_BITS(label1, zvec, zn); \
while (1) { \
if (zn > 20 /* the longest code */) \
RETURN(BZ_DATA_ERROR); \
if (zvec <= gLimit[zn]) break; \
zn++; \
GET_BIT(label2, zj); \
zvec = (zvec << 1) | zj; \
}; \
if (zvec - gBase[zn] < 0 \
|| zvec - gBase[zn] >= BZ_MAX_ALPHA_SIZE) \
RETURN(BZ_DATA_ERROR); \
lval = gPerm[zvec - gBase[zn]]; \
}
/*---------------------------------------------------*/
Int32 BZ2_decompress ( DState* s )
{
UChar uc;
Int32 retVal;
Int32 minLen, maxLen;
bz_stream* strm = s->strm;
/* stuff that needs to be saved/restored */
Int32 i;
Int32 j;
Int32 t;
Int32 alphaSize;
Int32 nGroups;
Int32 nSelectors;
Int32 EOB;
Int32 groupNo;
Int32 groupPos;
Int32 nextSym;
Int32 nblockMAX;
Int32 nblock;
Int32 es;
Int32 N;
Int32 curr;
Int32 zt;
Int32 zn;
Int32 zvec;
Int32 zj;
Int32 gSel;
Int32 gMinlen;
Int32* gLimit;
Int32* gBase;
Int32* gPerm;
if (s->state == BZ_X_MAGIC_1) {
/*initialise the save area*/
s->save_i = 0;
s->save_j = 0;
s->save_t = 0;
s->save_alphaSize = 0;
s->save_nGroups = 0;
s->save_nSelectors = 0;
s->save_EOB = 0;
s->save_groupNo = 0;
s->save_groupPos = 0;
s->save_nextSym = 0;
s->save_nblockMAX = 0;
s->save_nblock = 0;
s->save_es = 0;
s->save_N = 0;
s->save_curr = 0;
s->save_zt = 0;
s->save_zn = 0;
s->save_zvec = 0;
s->save_zj = 0;
s->save_gSel = 0;
s->save_gMinlen = 0;
s->save_gLimit = NULL;
s->save_gBase = NULL;
s->save_gPerm = NULL;
}
/*restore from the save area*/
i = s->save_i;
j = s->save_j;
t = s->save_t;
alphaSize = s->save_alphaSize;
nGroups = s->save_nGroups;
nSelectors = s->save_nSelectors;
EOB = s->save_EOB;
groupNo = s->save_groupNo;
groupPos = s->save_groupPos;
nextSym = s->save_nextSym;
nblockMAX = s->save_nblockMAX;
nblock = s->save_nblock;
es = s->save_es;
N = s->save_N;
curr = s->save_curr;
zt = s->save_zt;
zn = s->save_zn;
zvec = s->save_zvec;
zj = s->save_zj;
gSel = s->save_gSel;
gMinlen = s->save_gMinlen;
gLimit = s->save_gLimit;
gBase = s->save_gBase;
gPerm = s->save_gPerm;
retVal = BZ_OK;
switch (s->state) {
GET_UCHAR(BZ_X_MAGIC_1, uc);
if (uc != BZ_HDR_B) RETURN(BZ_DATA_ERROR_MAGIC);
GET_UCHAR(BZ_X_MAGIC_2, uc);
if (uc != BZ_HDR_Z) RETURN(BZ_DATA_ERROR_MAGIC);
GET_UCHAR(BZ_X_MAGIC_3, uc)
if (uc != BZ_HDR_h) RETURN(BZ_DATA_ERROR_MAGIC);
GET_BITS(BZ_X_MAGIC_4, s->blockSize100k, 8)
if (s->blockSize100k < (BZ_HDR_0 + 1) ||
s->blockSize100k > (BZ_HDR_0 + 9)) RETURN(BZ_DATA_ERROR_MAGIC);
s->blockSize100k -= BZ_HDR_0;
if (s->smallDecompress) {
s->ll16 = BZALLOC( s->blockSize100k * 100000 * sizeof(UInt16) );
s->ll4 = BZALLOC(
((1 + s->blockSize100k * 100000) >> 1) * sizeof(UChar)
);
if (s->ll16 == NULL || s->ll4 == NULL) RETURN(BZ_MEM_ERROR);
} else {
s->tt = BZALLOC( s->blockSize100k * 100000 * sizeof(Int32) );
if (s->tt == NULL) RETURN(BZ_MEM_ERROR);
}
GET_UCHAR(BZ_X_BLKHDR_1, uc);
if (uc == 0x17) goto endhdr_2;
if (uc != 0x31) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_BLKHDR_2, uc);
if (uc != 0x41) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_BLKHDR_3, uc);
if (uc != 0x59) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_BLKHDR_4, uc);
if (uc != 0x26) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_BLKHDR_5, uc);
if (uc != 0x53) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_BLKHDR_6, uc);
if (uc != 0x59) RETURN(BZ_DATA_ERROR);
s->currBlockNo++;
if (s->verbosity >= 2)
VPrintf1 ( "\n [%d: huff+mtf ", s->currBlockNo );
s->storedBlockCRC = 0;
GET_UCHAR(BZ_X_BCRC_1, uc);
s->storedBlockCRC = (s->storedBlockCRC << 8) | ((UInt32)uc);
GET_UCHAR(BZ_X_BCRC_2, uc);
s->storedBlockCRC = (s->storedBlockCRC << 8) | ((UInt32)uc);
GET_UCHAR(BZ_X_BCRC_3, uc);
s->storedBlockCRC = (s->storedBlockCRC << 8) | ((UInt32)uc);
GET_UCHAR(BZ_X_BCRC_4, uc);
s->storedBlockCRC = (s->storedBlockCRC << 8) | ((UInt32)uc);
GET_BITS(BZ_X_RANDBIT, s->blockRandomised, 1);
s->origPtr = 0;
GET_UCHAR(BZ_X_ORIGPTR_1, uc);
s->origPtr = (s->origPtr << 8) | ((Int32)uc);
GET_UCHAR(BZ_X_ORIGPTR_2, uc);
s->origPtr = (s->origPtr << 8) | ((Int32)uc);
GET_UCHAR(BZ_X_ORIGPTR_3, uc);
s->origPtr = (s->origPtr << 8) | ((Int32)uc);
if (s->origPtr < 0)
RETURN(BZ_DATA_ERROR);
if (s->origPtr > 10 + 100000*s->blockSize100k)
RETURN(BZ_DATA_ERROR);
/*--- Receive the mapping table ---*/
for (i = 0; i < 16; i++) {
GET_BIT(BZ_X_MAPPING_1, uc);
if (uc == 1)
s->inUse16[i] = True; else
s->inUse16[i] = False;
}
for (i = 0; i < 256; i++) s->inUse[i] = False;
for (i = 0; i < 16; i++)
if (s->inUse16[i])
for (j = 0; j < 16; j++) {
GET_BIT(BZ_X_MAPPING_2, uc);
if (uc == 1) s->inUse[i * 16 + j] = True;
}
makeMaps_d ( s );
if (s->nInUse == 0) RETURN(BZ_DATA_ERROR);
alphaSize = s->nInUse+2;
/*--- Now the selectors ---*/
GET_BITS(BZ_X_SELECTOR_1, nGroups, 3);
if (nGroups < 2 || nGroups > 6) RETURN(BZ_DATA_ERROR);
GET_BITS(BZ_X_SELECTOR_2, nSelectors, 15);
if (nSelectors < 1) RETURN(BZ_DATA_ERROR);
for (i = 0; i < nSelectors; i++) {
j = 0;
while (True) {
GET_BIT(BZ_X_SELECTOR_3, uc);
if (uc == 0) break;
j++;
if (j >= nGroups) RETURN(BZ_DATA_ERROR);
}
s->selectorMtf[i] = j;
}
/*--- Undo the MTF values for the selectors. ---*/
{
UChar pos[BZ_N_GROUPS], tmp, v;
for (v = 0; v < nGroups; v++) pos[v] = v;
for (i = 0; i < nSelectors; i++) {
v = s->selectorMtf[i];
tmp = pos[v];
while (v > 0) { pos[v] = pos[v-1]; v--; }
pos[0] = tmp;
s->selector[i] = tmp;
}
}
/*--- Now the coding tables ---*/
for (t = 0; t < nGroups; t++) {
GET_BITS(BZ_X_CODING_1, curr, 5);
for (i = 0; i < alphaSize; i++) {
while (True) {
if (curr < 1 || curr > 20) RETURN(BZ_DATA_ERROR);
GET_BIT(BZ_X_CODING_2, uc);
if (uc == 0) break;
GET_BIT(BZ_X_CODING_3, uc);
if (uc == 0) curr++; else curr--;
}
s->len[t][i] = curr;
}
}
/*--- Create the Huffman decoding tables ---*/
for (t = 0; t < nGroups; t++) {
minLen = 32;
maxLen = 0;
for (i = 0; i < alphaSize; i++) {
if (s->len[t][i] > maxLen) maxLen = s->len[t][i];
if (s->len[t][i] < minLen) minLen = s->len[t][i];
}
BZ2_hbCreateDecodeTables (
&(s->limit[t][0]),
&(s->base[t][0]),
&(s->perm[t][0]),
&(s->len[t][0]),
minLen, maxLen, alphaSize
);
s->minLens[t] = minLen;
}
/*--- Now the MTF values ---*/
EOB = s->nInUse+1;
nblockMAX = 100000 * s->blockSize100k;
groupNo = -1;
groupPos = 0;
for (i = 0; i <= 255; i++) s->unzftab[i] = 0;
/*-- MTF init --*/
{
Int32 ii, jj, kk;
kk = MTFA_SIZE-1;
for (ii = 256 / MTFL_SIZE - 1; ii >= 0; ii--) {
for (jj = MTFL_SIZE-1; jj >= 0; jj--) {
s->mtfa[kk] = (UChar)(ii * MTFL_SIZE + jj);
kk--;
}
s->mtfbase[ii] = kk + 1;
}
}
/*-- end MTF init --*/
nblock = 0;
GET_MTF_VAL(BZ_X_MTF_1, BZ_X_MTF_2, nextSym);
while (True) {
if (nextSym == EOB) break;
if (nextSym == BZ_RUNA || nextSym == BZ_RUNB) {
es = -1;
N = 1;
do {
/* Check that N doesn't get too big, so that es doesn't
go negative. The maximum value that can be
RUNA/RUNB encoded is equal to the block size (post
the initial RLE), viz, 900k, so bounding N at 2
million should guard against overflow without
rejecting any legitimate inputs. */
if (N >= 2*1024*1024) RETURN(BZ_DATA_ERROR);
if (nextSym == BZ_RUNA) es = es + (0+1) * N; else
if (nextSym == BZ_RUNB) es = es + (1+1) * N;
N = N * 2;
GET_MTF_VAL(BZ_X_MTF_3, BZ_X_MTF_4, nextSym);
}
while (nextSym == BZ_RUNA || nextSym == BZ_RUNB);
es++;
uc = s->seqToUnseq[ s->mtfa[s->mtfbase[0]] ];
s->unzftab[uc] += es;
if (s->smallDecompress)
while (es > 0) {
if (nblock >= nblockMAX) RETURN(BZ_DATA_ERROR);
s->ll16[nblock] = (UInt16)uc;
nblock++;
es--;
}
else
while (es > 0) {
if (nblock >= nblockMAX) RETURN(BZ_DATA_ERROR);
s->tt[nblock] = (UInt32)uc;
nblock++;
es--;
};
continue;
} else {
if (nblock >= nblockMAX) RETURN(BZ_DATA_ERROR);
/*-- uc = MTF ( nextSym-1 ) --*/
{
Int32 ii, jj, kk, pp, lno, off;
UInt32 nn;
nn = (UInt32)(nextSym - 1);
if (nn < MTFL_SIZE) {
/* avoid general-case expense */
pp = s->mtfbase[0];
uc = s->mtfa[pp+nn];
while (nn > 3) {
Int32 z = pp+nn;
s->mtfa[(z) ] = s->mtfa[(z)-1];
s->mtfa[(z)-1] = s->mtfa[(z)-2];
s->mtfa[(z)-2] = s->mtfa[(z)-3];
s->mtfa[(z)-3] = s->mtfa[(z)-4];
nn -= 4;
}
while (nn > 0) {
s->mtfa[(pp+nn)] = s->mtfa[(pp+nn)-1]; nn--;
};
s->mtfa[pp] = uc;
} else {
/* general case */
lno = nn / MTFL_SIZE;
off = nn % MTFL_SIZE;
pp = s->mtfbase[lno] + off;
uc = s->mtfa[pp];
while (pp > s->mtfbase[lno]) {
s->mtfa[pp] = s->mtfa[pp-1]; pp--;
};
s->mtfbase[lno]++;
while (lno > 0) {
s->mtfbase[lno]--;
s->mtfa[s->mtfbase[lno]]
= s->mtfa[s->mtfbase[lno-1] + MTFL_SIZE - 1];
lno--;
}
s->mtfbase[0]--;
s->mtfa[s->mtfbase[0]] = uc;
if (s->mtfbase[0] == 0) {
kk = MTFA_SIZE-1;
for (ii = 256 / MTFL_SIZE-1; ii >= 0; ii--) {
for (jj = MTFL_SIZE-1; jj >= 0; jj--) {
s->mtfa[kk] = s->mtfa[s->mtfbase[ii] + jj];
kk--;
}
s->mtfbase[ii] = kk + 1;
}
}
}
}
/*-- end uc = MTF ( nextSym-1 ) --*/
s->unzftab[s->seqToUnseq[uc]]++;
if (s->smallDecompress)
s->ll16[nblock] = (UInt16)(s->seqToUnseq[uc]); else
s->tt[nblock] = (UInt32)(s->seqToUnseq[uc]);
nblock++;
GET_MTF_VAL(BZ_X_MTF_5, BZ_X_MTF_6, nextSym);
continue;
}
}
/* Now we know what nblock is, we can do a better sanity
check on s->origPtr.
*/
if (s->origPtr < 0 || s->origPtr >= nblock)
RETURN(BZ_DATA_ERROR);
/*-- Set up cftab to facilitate generation of T^(-1) --*/
/* Check: unzftab entries in range. */
for (i = 0; i <= 255; i++) {
if (s->unzftab[i] < 0 || s->unzftab[i] > nblock)
RETURN(BZ_DATA_ERROR);
}
/* Actually generate cftab. */
s->cftab[0] = 0;
for (i = 1; i <= 256; i++) s->cftab[i] = s->unzftab[i-1];
for (i = 1; i <= 256; i++) s->cftab[i] += s->cftab[i-1];
/* Check: cftab entries in range. */
for (i = 0; i <= 256; i++) {
if (s->cftab[i] < 0 || s->cftab[i] > nblock) {
/* s->cftab[i] can legitimately be == nblock */
RETURN(BZ_DATA_ERROR);
}
}
/* Check: cftab entries non-descending. */
for (i = 1; i <= 256; i++) {
if (s->cftab[i-1] > s->cftab[i]) {
RETURN(BZ_DATA_ERROR);
}
}
s->state_out_len = 0;
s->state_out_ch = 0;
BZ_INITIALISE_CRC ( s->calculatedBlockCRC );
s->state = BZ_X_OUTPUT;
if (s->verbosity >= 2) VPrintf0 ( "rt+rld" );
if (s->smallDecompress) {
/*-- Make a copy of cftab, used in generation of T --*/
for (i = 0; i <= 256; i++) s->cftabCopy[i] = s->cftab[i];
/*-- compute the T vector --*/
for (i = 0; i < nblock; i++) {
uc = (UChar)(s->ll16[i]);
SET_LL(i, s->cftabCopy[uc]);
s->cftabCopy[uc]++;
}
/*-- Compute T^(-1) by pointer reversal on T --*/
i = s->origPtr;
j = GET_LL(i);
do {
Int32 tmp = GET_LL(j);
SET_LL(j, i);
i = j;
j = tmp;
}
while (i != s->origPtr);
s->tPos = s->origPtr;
s->nblock_used = 0;
if (s->blockRandomised) {
BZ_RAND_INIT_MASK;
BZ_GET_SMALL(s->k0); s->nblock_used++;
BZ_RAND_UPD_MASK; s->k0 ^= BZ_RAND_MASK;
} else {
BZ_GET_SMALL(s->k0); s->nblock_used++;
}
} else {
/*-- compute the T^(-1) vector --*/
for (i = 0; i < nblock; i++) {
uc = (UChar)(s->tt[i] & 0xff);
s->tt[s->cftab[uc]] |= (i << 8);
s->cftab[uc]++;
}
s->tPos = s->tt[s->origPtr] >> 8;
s->nblock_used = 0;
if (s->blockRandomised) {
BZ_RAND_INIT_MASK;
BZ_GET_FAST(s->k0); s->nblock_used++;
BZ_RAND_UPD_MASK; s->k0 ^= BZ_RAND_MASK;
} else {
BZ_GET_FAST(s->k0); s->nblock_used++;
}
}
RETURN(BZ_OK);
endhdr_2:
GET_UCHAR(BZ_X_ENDHDR_2, uc);
if (uc != 0x72) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_ENDHDR_3, uc);
if (uc != 0x45) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_ENDHDR_4, uc);
if (uc != 0x38) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_ENDHDR_5, uc);
if (uc != 0x50) RETURN(BZ_DATA_ERROR);
GET_UCHAR(BZ_X_ENDHDR_6, uc);
if (uc != 0x90) RETURN(BZ_DATA_ERROR);
s->storedCombinedCRC = 0;
GET_UCHAR(BZ_X_CCRC_1, uc);
s->storedCombinedCRC = (s->storedCombinedCRC << 8) | ((UInt32)uc);
GET_UCHAR(BZ_X_CCRC_2, uc);
s->storedCombinedCRC = (s->storedCombinedCRC << 8) | ((UInt32)uc);
GET_UCHAR(BZ_X_CCRC_3, uc);
s->storedCombinedCRC = (s->storedCombinedCRC << 8) | ((UInt32)uc);
GET_UCHAR(BZ_X_CCRC_4, uc);
s->storedCombinedCRC = (s->storedCombinedCRC << 8) | ((UInt32)uc);
s->state = BZ_X_IDLE;
RETURN(BZ_STREAM_END);
default: AssertH ( False, 4001 );
}
AssertH ( False, 4002 );
save_state_and_return:
s->save_i = i;
s->save_j = j;
s->save_t = t;
s->save_alphaSize = alphaSize;
s->save_nGroups = nGroups;
s->save_nSelectors = nSelectors;
s->save_EOB = EOB;
s->save_groupNo = groupNo;
s->save_groupPos = groupPos;
s->save_nextSym = nextSym;
s->save_nblockMAX = nblockMAX;
s->save_nblock = nblock;
s->save_es = es;
s->save_N = N;
s->save_curr = curr;
s->save_zt = zt;
s->save_zn = zn;
s->save_zvec = zvec;
s->save_zj = zj;
s->save_gSel = gSel;
s->save_gMinlen = gMinlen;
s->save_gLimit = gLimit;
s->save_gBase = gBase;
s->save_gPerm = gPerm;
return retVal;
}
/*-------------------------------------------------------------*/
/*--- end decompress.c ---*/
/*-------------------------------------------------------------*/

205
librocksdb-sys/bzip2/huffman.c
Unescape View File

@ -0,0 +1,205 @@
/*-------------------------------------------------------------*/
/*--- Huffman coding low-level stuff ---*/
/*--- huffman.c ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
#include "bzlib_private.h"
/*---------------------------------------------------*/
#define WEIGHTOF(zz0) ((zz0) & 0xffffff00)
#define DEPTHOF(zz1) ((zz1) & 0x000000ff)
#define MYMAX(zz2,zz3) ((zz2) > (zz3) ? (zz2) : (zz3))
#define ADDWEIGHTS(zw1,zw2) \
(WEIGHTOF(zw1)+WEIGHTOF(zw2)) | \
(1 + MYMAX(DEPTHOF(zw1),DEPTHOF(zw2)))
#define UPHEAP(z) \
{ \
Int32 zz, tmp; \
zz = z; tmp = heap[zz]; \
while (weight[tmp] < weight[heap[zz >> 1]]) { \
heap[zz] = heap[zz >> 1]; \
zz >>= 1; \
} \
heap[zz] = tmp; \
}
#define DOWNHEAP(z) \
{ \
Int32 zz, yy, tmp; \
zz = z; tmp = heap[zz]; \
while (True) { \
yy = zz << 1; \
if (yy > nHeap) break; \
if (yy < nHeap && \
weight[heap[yy+1]] < weight[heap[yy]]) \
yy++; \
if (weight[tmp] < weight[heap[yy]]) break; \
heap[zz] = heap[yy]; \
zz = yy; \
} \
heap[zz] = tmp; \
}
/*---------------------------------------------------*/
void BZ2_hbMakeCodeLengths ( UChar *len,
Int32 *freq,
Int32 alphaSize,
Int32 maxLen )
{
/*--
Nodes and heap entries run from 1. Entry 0
for both the heap and nodes is a sentinel.
--*/
Int32 nNodes, nHeap, n1, n2, i, j, k;
Bool tooLong;
Int32 heap [ BZ_MAX_ALPHA_SIZE + 2 ];
Int32 weight [ BZ_MAX_ALPHA_SIZE * 2 ];
Int32 parent [ BZ_MAX_ALPHA_SIZE * 2 ];
for (i = 0; i < alphaSize; i++)
weight[i+1] = (freq[i] == 0 ? 1 : freq[i]) << 8;
while (True) {
nNodes = alphaSize;
nHeap = 0;
heap[0] = 0;
weight[0] = 0;
parent[0] = -2;
for (i = 1; i <= alphaSize; i++) {
parent[i] = -1;
nHeap++;
heap[nHeap] = i;
UPHEAP(nHeap);
}
AssertH( nHeap < (BZ_MAX_ALPHA_SIZE+2), 2001 );
while (nHeap > 1) {
n1 = heap[1]; heap[1] = heap[nHeap]; nHeap--; DOWNHEAP(1);
n2 = heap[1]; heap[1] = heap[nHeap]; nHeap--; DOWNHEAP(1);
nNodes++;
parent[n1] = parent[n2] = nNodes;
weight[nNodes] = ADDWEIGHTS(weight[n1], weight[n2]);
parent[nNodes] = -1;
nHeap++;
heap[nHeap] = nNodes;
UPHEAP(nHeap);
}
AssertH( nNodes < (BZ_MAX_ALPHA_SIZE * 2), 2002 );
tooLong = False;
for (i = 1; i <= alphaSize; i++) {
j = 0;
k = i;
while (parent[k] >= 0) { k = parent[k]; j++; }
len[i-1] = j;
if (j > maxLen) tooLong = True;
}
if (! tooLong) break;
/* 17 Oct 04: keep-going condition for the following loop used
to be 'i < alphaSize', which missed the last element,
theoretically leading to the possibility of the compressor
looping. However, this count-scaling step is only needed if
one of the generated Huffman code words is longer than
maxLen, which up to and including version 1.0.2 was 20 bits,
which is extremely unlikely. In version 1.0.3 maxLen was
changed to 17 bits, which has minimal effect on compression
ratio, but does mean this scaling step is used from time to
time, enough to verify that it works.
This means that bzip2-1.0.3 and later will only produce
Huffman codes with a maximum length of 17 bits. However, in
order to preserve backwards compatibility with bitstreams
produced by versions pre-1.0.3, the decompressor must still
handle lengths of up to 20. */
for (i = 1; i <= alphaSize; i++) {
j = weight[i] >> 8;
j = 1 + (j / 2);
weight[i] = j << 8;
}
}
}
/*---------------------------------------------------*/
void BZ2_hbAssignCodes ( Int32 *code,
UChar *length,
Int32 minLen,
Int32 maxLen,
Int32 alphaSize )
{
Int32 n, vec, i;
vec = 0;
for (n = minLen; n <= maxLen; n++) {
for (i = 0; i < alphaSize; i++)
if (length[i] == n) { code[i] = vec; vec++; };
vec <<= 1;
}
}
/*---------------------------------------------------*/
void BZ2_hbCreateDecodeTables ( Int32 *limit,
Int32 *base,
Int32 *perm,
UChar *length,
Int32 minLen,
Int32 maxLen,
Int32 alphaSize )
{
Int32 pp, i, j, vec;
pp = 0;
for (i = minLen; i <= maxLen; i++)
for (j = 0; j < alphaSize; j++)
if (length[j] == i) { perm[pp] = j; pp++; };
for (i = 0; i < BZ_MAX_CODE_LEN; i++) base[i] = 0;
for (i = 0; i < alphaSize; i++) base[length[i]+1]++;
for (i = 1; i < BZ_MAX_CODE_LEN; i++) base[i] += base[i-1];
for (i = 0; i < BZ_MAX_CODE_LEN; i++) limit[i] = 0;
vec = 0;
for (i = minLen; i <= maxLen; i++) {
vec += (base[i+1] - base[i]);
limit[i] = vec-1;
vec <<= 1;
}
for (i = minLen + 1; i <= maxLen; i++)
base[i] = ((limit[i-1] + 1) << 1) - base[i];
}
/*-------------------------------------------------------------*/
/*--- end huffman.c ---*/
/*-------------------------------------------------------------*/

84
librocksdb-sys/bzip2/randtable.c
Unescape View File

@ -0,0 +1,84 @@
/*-------------------------------------------------------------*/
/*--- Table for randomising repetitive blocks ---*/
/*--- randtable.c ---*/
/*-------------------------------------------------------------*/
/* ------------------------------------------------------------------
This file is part of bzip2/libbzip2, a program and library for
lossless, block-sorting data compression.
bzip2/libbzip2 version 1.0.6 of 6 September 2010
Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
Please read the WARNING, DISCLAIMER and PATENTS sections in the
README file.
This program is released under the terms of the license contained
in the file LICENSE.
------------------------------------------------------------------ */
#include "bzlib_private.h"
/*---------------------------------------------*/
Int32 BZ2_rNums[512] = {
619, 720, 127, 481, 931, 816, 813, 233, 566, 247,
985, 724, 205, 454, 863, 491, 741, 242, 949, 214,
733, 859, 335, 708, 621, 574, 73, 654, 730, 472,
419, 436, 278, 496, 867, 210, 399, 680, 480, 51,
878, 465, 811, 169, 869, 675, 611, 697, 867, 561,
862, 687, 507, 283, 482, 129, 807, 591, 733, 623,
150, 238, 59, 379, 684, 877, 625, 169, 643, 105,
170, 607, 520, 932, 727, 476, 693, 425, 174, 647,
73, 122, 335, 530, 442, 853, 695, 249, 445, 515,
909, 545, 703, 919, 874, 474, 882, 500, 594, 612,
641, 801, 220, 162, 819, 984, 589, 513, 495, 799,
161, 604, 958, 533, 221, 400, 386, 867, 600, 782,
382, 596, 414, 171, 516, 375, 682, 485, 911, 276,
98, 553, 163, 354, 666, 933, 424, 341, 533, 870,
227, 730, 475, 186, 263, 647, 537, 686, 600, 224,
469, 68, 770, 919, 190, 373, 294, 822, 808, 206,
184, 943, 795, 384, 383, 461, 404, 758, 839, 887,
715, 67, 618, 276, 204, 918, 873, 777, 604, 560,
951, 160, 578, 722, 79, 804, 96, 409, 713, 940,
652, 934, 970, 447, 318, 353, 859, 672, 112, 785,
645, 863, 803, 350, 139, 93, 354, 99, 820, 908,
609, 772, 154, 274, 580, 184, 79, 626, 630, 742,
653, 282, 762, 623, 680, 81, 927, 626, 789, 125,
411, 521, 938, 300, 821, 78, 343, 175, 128, 250,
170, 774, 972, 275, 999, 639, 495, 78, 352, 126,
857, 956, 358, 619, 580, 124, 737, 594, 701, 612,
669, 112, 134, 694, 363, 992, 809, 743, 168, 974,
944, 375, 748, 52, 600, 747, 642, 182, 862, 81,
344, 805, 988, 739, 511, 655, 814, 334, 249, 515,
897, 955, 664, 981, 649, 113, 974, 459, 893, 228,
433, 837, 553, 268, 926, 240, 102, 654, 459, 51,
686, 754, 806, 760, 493, 403, 415, 394, 687, 700,
946, 670, 656, 610, 738, 392, 760, 799, 887, 653,
978, 321, 576, 617, 626, 502, 894, 679, 243, 440,
680, 879, 194, 572, 640, 724, 926, 56, 204, 700,
707, 151, 457, 449, 797, 195, 791, 558, 945, 679,
297, 59, 87, 824, 713, 663, 412, 693, 342, 606,
134, 108, 571, 364, 631, 212, 174, 643, 304, 329,
343, 97, 430, 751, 497, 314, 983, 374, 822, 928,
140, 206, 73, 263, 980, 736, 876, 478, 430, 305,
170, 514, 364, 692, 829, 82, 855, 953, 676, 246,
369, 970, 294, 750, 807, 827, 150, 790, 288, 923,
804, 378, 215, 828, 592, 281, 565, 555, 710, 82,
896, 831, 547, 261, 524, 462, 293, 465, 502, 56,
661, 821, 976, 991, 658, 869, 905, 758, 745, 193,
768, 550, 608, 933, 378, 286, 215, 979, 792, 961,
61, 688, 793, 644, 986, 403, 106, 366, 905, 644,
372, 567, 466, 434, 645, 210, 389, 550, 919, 135,
780, 773, 635, 389, 707, 100, 626, 958, 165, 504,
920, 176, 193, 713, 857, 265, 203, 50, 668, 108,
645, 990, 626, 197, 510, 357, 358, 850, 858, 364,
936, 638
};
/*-------------------------------------------------------------*/
/*--- end randtable.c ---*/
/*-------------------------------------------------------------*/

1
librocksdb-sys/lz4

@ -0,0 +1 @@
Subproject commit c10863b98e1503af90616ae99725ecd120265dfb

2
librocksdb-sys/rocksdb

@ -1 +1 @@
Subproject commit 7ac544656d1de78c4da09f1e347ab0782308935c Subproject commit 5089e121166c46956a8a21c8ef967f1896c239de

2
librocksdb-sys/rocksdb_lib_sources.txt
View File

File diff suppressed because one or more lines are too long

2
librocksdb-sys/snappy

@ -1 +1 @@
Subproject commit efb39e81b8b66b757ec900db33657c47b4750740 Subproject commit 2d99bd14d471664758e4dfdf81b44f413a7353fd

7
librocksdb-sys/src/lib.rs
Unescape View File

@ -18,4 +18,11 @@
extern crate libc; extern crate libc;
use libc::c_int;
include!(concat!(env!("OUT_DIR"), "/bindings.rs")); include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
#[no_mangle]
pub fn bz_internal_error(errcode: c_int) {
panic!("bz internal error: {}", errcode);
}

8
librocksdb-sys/tests/ffi.rs
Unescape View File

@ -21,6 +21,7 @@
extern crate const_cstr; extern crate const_cstr;
extern crate libc; extern crate libc;
extern crate librocksdb_sys as ffi; extern crate librocksdb_sys as ffi;
extern crate uuid;
use ::ffi::*; use ::ffi::*;
use ::libc::*; use ::libc::*;
@ -33,6 +34,7 @@ use ::std::path::PathBuf;
use ::std::ptr; use ::std::ptr;
use ::std::slice; use ::std::slice;
use ::std::str; use ::std::str;
use ::uuid::Uuid;
macro_rules! err_println { macro_rules! err_println {
($($arg:tt)*) => (writeln!(&mut ::std::io::stderr(), $($arg)*).expect("failed printing to stderr")); ($($arg:tt)*) => (writeln!(&mut ::std::io::stderr(), $($arg)*).expect("failed printing to stderr"));
@ -392,15 +394,17 @@ fn ffi() {
let mut err: *mut c_char = ptr::null_mut(); let mut err: *mut c_char = ptr::null_mut();
let run: c_int = -1; let run: c_int = -1;
let test_uuid = Uuid::new_v4().to_simple();
let dbname = { let dbname = {
let mut dir = GetTempDir(); let mut dir = GetTempDir();
dir.push(format!("rocksdb_c_test-{}", geteuid())); dir.push(format!("rocksdb_c_test-{}", test_uuid));
let path = dir.to_str().unwrap(); let path = dir.to_str().unwrap();
CString::new(path).unwrap() CString::new(path).unwrap()
}; };
let dbbackupname = { let dbbackupname = {
let mut dir = GetTempDir(); let mut dir = GetTempDir();
dir.push(format!("rocksdb_c_test-{}-backup", geteuid())); dir.push(format!("rocksdb_c_test-{}-backup", test_uuid));
let path = dir.to_str().unwrap(); let path = dir.to_str().unwrap();
CString::new(path).unwrap() CString::new(path).unwrap()
}; };

1
librocksdb-sys/zlib

@ -0,0 +1 @@
Subproject commit cacf7f1d4e3d44d871b605da3b647f07d718623f

1
librocksdb-sys/zstd

@ -0,0 +1 @@
Subproject commit f3a8bd553a865c59f1bd6e1f68bf182cf75a8f00

80
src/checkpoint.rs
Unescape View File

@ -0,0 +1,80 @@
// Copyright 2018 Eugene P.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
///! Implementation of bindings to RocksDB Checkpoint[1] API
///
/// [1]: https://github.com/facebook/rocksdb/wiki/Checkpoints
use {DB, Error};
use ffi;
use std::ffi::CString;
use std::path::Path;
/// Undocumented parameter for `ffi::rocksdb_checkpoint_create` function. Zero by default.
const LOG_SIZE_FOR_FLUSH: u64 = 0_u64;
/// Database's checkpoint object.
/// Used to create checkpoints of the specified DB from time to time.
pub struct Checkpoint {
inner: *mut ffi::rocksdb_checkpoint_t,
}
impl Checkpoint {
/// Creates new checkpoint object for specific DB.
///
/// Does not actually produce checkpoints, call `.create_checkpoint()` method to produce
/// a DB checkpoint.
pub fn new(db: &DB) -> Result<Checkpoint, Error> {
let checkpoint: *mut ffi::rocksdb_checkpoint_t;
unsafe { checkpoint = ffi_try!(ffi::rocksdb_checkpoint_object_create(db.inner,)) };
if checkpoint.is_null() {
return Err(Error::new("Could not create checkpoint object.".to_owned()));
}
Ok(Checkpoint {
inner: checkpoint,
})
}
/// Creates new physical DB checkpoint in directory specified by `path`.
pub fn create_checkpoint<P: AsRef<Path>>(&self, path: P) -> Result<(), Error> {
let path = path.as_ref();
let cpath = match CString::new(path.to_string_lossy().as_bytes()) {
Ok(c) => c,
Err(_) => {
return Err(Error::new(
"Failed to convert path to CString when creating DB checkpoint"
.to_owned(),
));
}
};
unsafe {
ffi_try!(ffi::rocksdb_checkpoint_create(self.inner, cpath.as_ptr(), LOG_SIZE_FOR_FLUSH,));
Ok(())
}
}
}
impl Drop for Checkpoint {
fn drop(&mut self) {
unsafe {
ffi::rocksdb_checkpoint_object_destroy(self.inner);
}
}
}

55
src/db.rs
Unescape View File

@ -28,6 +28,7 @@ use std::path::Path;
use std::ptr; use std::ptr;
use std::slice; use std::slice;
use std::str; use std::str;
use std::ffi::CStr;
pub fn new_bloom_filter(bits: c_int) -> *mut ffi::rocksdb_filterpolicy_t { pub fn new_bloom_filter(bits: c_int) -> *mut ffi::rocksdb_filterpolicy_t {
unsafe { ffi::rocksdb_filterpolicy_create_bloom(bits) } unsafe { ffi::rocksdb_filterpolicy_create_bloom(bits) }
@ -44,6 +45,7 @@ pub enum DBCompressionType {
Bz2 = ffi::rocksdb_bz2_compression as isize, Bz2 = ffi::rocksdb_bz2_compression as isize,
Lz4 = ffi::rocksdb_lz4_compression as isize, Lz4 = ffi::rocksdb_lz4_compression as isize,
Lz4hc = ffi::rocksdb_lz4hc_compression as isize, Lz4hc = ffi::rocksdb_lz4hc_compression as isize,
Zstd = ffi::rocksdb_zstd_compression as isize,
} }
#[derive(Debug, Copy, Clone, PartialEq)] #[derive(Debug, Copy, Clone, PartialEq)]
@ -469,10 +471,13 @@ impl DBIterator {
self.raw.seek_to_last(); self.raw.seek_to_last();
self.direction = Direction::Reverse; self.direction = Direction::Reverse;
} }
IteratorMode::From(key, dir) => { IteratorMode::From(key, Direction::Forward) => {
// TODO: Should use seek_for_prev when reversing
self.raw.seek(key); self.raw.seek(key);
self.direction = dir; self.direction = Direction::Forward;
}
IteratorMode::From(key, Direction::Reverse) => {
self.raw.seek_for_prev(key);
self.direction = Direction::Reverse;
} }
}; };
@ -698,8 +703,7 @@ impl DB {
} }
pub fn list_cf<P: AsRef<Path>>(opts: &Options, path: P) -> Result<Vec<String>, Error> { pub fn list_cf<P: AsRef<Path>>(opts: &Options, path: P) -> Result<Vec<String>, Error> {
let path = path.as_ref(); let cpath = match CString::new(path.as_ref().to_string_lossy().as_bytes()) {
let cpath = match CString::new(path.to_string_lossy().as_bytes()) {
Ok(c) => c, Ok(c) => c,
Err(_) => { Err(_) => {
return Err(Error::new( return Err(Error::new(
@ -719,10 +723,11 @@ impl DB {
&mut length, &mut length,
)); ));
let vec = Vec::from_raw_parts(ptr, length, length) let vec = slice::from_raw_parts(ptr, length)
.iter() .iter()
.map(|&ptr| CString::from_raw(ptr).into_string().unwrap()) .map(|ptr| CStr::from_ptr(*ptr).to_string_lossy().into_owned())
.collect(); .collect();
ffi::rocksdb_list_column_families_destroy(ptr, length);
Ok(vec) Ok(vec)
} }
} }
@ -888,6 +893,15 @@ impl DB {
DBIterator::new(self, &opts, mode) DBIterator::new(self, &opts, mode)
} }
/// Opens an interator with `set_total_order_seek` enabled.
/// This must be used to iterate across prefixes when `set_memtable_factory` has been called
/// with a Hash-based implementation.
pub fn full_iterator(&self, mode: IteratorMode) -> DBIterator {
let mut opts = ReadOptions::default();
opts.set_total_order_seek(true);
DBIterator::new(self, &opts, mode)
}
pub fn prefix_iterator<'a>(&self, prefix: &'a [u8]) -> DBIterator { pub fn prefix_iterator<'a>(&self, prefix: &'a [u8]) -> DBIterator {
let mut opts = ReadOptions::default(); let mut opts = ReadOptions::default();
opts.set_prefix_same_as_start(true); opts.set_prefix_same_as_start(true);
@ -903,6 +917,16 @@ impl DB {
DBIterator::new_cf(self, cf_handle, &opts, mode) DBIterator::new_cf(self, cf_handle, &opts, mode)
} }
pub fn full_iterator_cf(
&self,
cf_handle: ColumnFamily,
mode: IteratorMode,
) -> Result<DBIterator, Error> {
let mut opts = ReadOptions::default();
opts.set_total_order_seek(true);
DBIterator::new_cf(self, cf_handle, &opts, mode)
}
pub fn prefix_iterator_cf<'a>( pub fn prefix_iterator_cf<'a>(
&self, &self,
cf_handle: ColumnFamily, cf_handle: ColumnFamily,
@ -1087,6 +1111,15 @@ impl WriteBatch {
unsafe { ffi::rocksdb_writebatch_count(self.inner) as usize } unsafe { ffi::rocksdb_writebatch_count(self.inner) as usize }
} }
/// Return WriteBatch serialized size (in bytes).
pub fn size_in_bytes(&self) -> usize {
unsafe {
let mut batch_size: size_t = 0;
ffi::rocksdb_writebatch_data(self.inner, &mut batch_size);
batch_size as usize
}
}
pub fn is_empty(&self) -> bool { pub fn is_empty(&self) -> bool {
self.len() == 0 self.len() == 0
} }
@ -1384,6 +1417,14 @@ fn writebatch_works() {
assert!(p.is_ok()); assert!(p.is_ok());
assert!(db.get(b"k1").unwrap().is_none()); assert!(db.get(b"k1").unwrap().is_none());
} }
{
// test size_in_bytes
let mut batch = WriteBatch::default();
let before = batch.size_in_bytes();
let _ = batch.put(b"k1", b"v1234567890");
let after = batch.size_in_bytes();
assert!(before + 10 <= after);
}
} }
let opts = Options::default(); let opts = Options::default();
assert!(DB::destroy(&opts, path).is_ok()); assert!(DB::destroy(&opts, path).is_ok());

148
src/db_options.rs
Unescape View File

@ -14,11 +14,12 @@
use std::ffi::{CStr, CString}; use std::ffi::{CStr, CString};
use std::mem; use std::mem;
use std::path::Path;
use libc::{self, c_int, c_uchar, c_uint, c_void, size_t, uint64_t}; use libc::{self, c_int, c_uchar, c_uint, c_void, size_t, uint64_t};
use ffi; use ffi;
use {BlockBasedOptions, DBCompactionStyle, DBCompressionType, DBRecoveryMode, use {BlockBasedOptions, BlockBasedIndexType, DBCompactionStyle, DBCompressionType, DBRecoveryMode, MemtableFactory,
Options, WriteOptions}; Options, WriteOptions};
use compaction_filter::{self, CompactionFilterCallback, CompactionFilterFn, filter_callback}; use compaction_filter::{self, CompactionFilterCallback, CompactionFilterFn, filter_callback};
use comparator::{self, ComparatorCallback, CompareFn}; use comparator::{self, ComparatorCallback, CompareFn};
@ -72,6 +73,12 @@ impl BlockBasedOptions {
} }
} }
pub fn disable_cache(&mut self) {
unsafe {
ffi::rocksdb_block_based_options_set_no_block_cache(self.inner, true as c_uchar);
}
}
pub fn set_bloom_filter(&mut self, bits_per_key: c_int, block_based: bool) { pub fn set_bloom_filter(&mut self, bits_per_key: c_int, block_based: bool) {
unsafe { unsafe {
let bloom = if block_based { let bloom = if block_based {
@ -89,6 +96,24 @@ impl BlockBasedOptions {
ffi::rocksdb_block_based_options_set_cache_index_and_filter_blocks(self.inner, v as u8); ffi::rocksdb_block_based_options_set_cache_index_and_filter_blocks(self.inner, v as u8);
} }
} }
/// Defines the index type to be used for SS-table lookups.
///
/// # Example
///
/// ```
/// use rocksdb::{BlockBasedOptions, BlockBasedIndexType, Options};
///
/// let mut opts = Options::default();
/// let mut block_opts = BlockBasedOptions::default();
/// block_opts.set_index_type(BlockBasedIndexType::HashSearch);
/// ```
pub fn set_index_type(&mut self, index_type: BlockBasedIndexType) {
let index = index_type as i32;
unsafe {
ffi::rocksdb_block_based_options_set_index_type(self.inner, index);
}
}
} }
impl Default for BlockBasedOptions { impl Default for BlockBasedOptions {
@ -221,6 +246,20 @@ impl Options {
} }
} }
/// If non-zero, we perform bigger reads when doing compaction. If you're
/// running RocksDB on spinning disks, you should set this to at least 2MB.
/// That way RocksDB's compaction is doing sequential instead of random reads.
///
/// When non-zero, we also force new_table_reader_for_compaction_inputs to
/// true.
///
/// Default: `0`
pub fn set_compaction_readahead_size(&mut self, compaction_readahead_size: usize) {
unsafe {
ffi::rocksdb_options_compaction_readahead_size(self.inner, compaction_readahead_size as usize);
}
}
pub fn set_merge_operator(&mut self, name: &str, pub fn set_merge_operator(&mut self, name: &str,
full_merge_fn: MergeFn, full_merge_fn: MergeFn,
partial_merge_fn: Option<MergeFn>) { partial_merge_fn: Option<MergeFn>) {
@ -883,6 +922,47 @@ impl Options {
unsafe { ffi::rocksdb_options_set_disable_auto_compactions(self.inner, disable as c_int) } unsafe { ffi::rocksdb_options_set_disable_auto_compactions(self.inner, disable as c_int) }
} }
/// Defines the underlying memtable implementation.
/// See https://github.com/facebook/rocksdb/wiki/MemTable for more information.
/// Defaults to using a skiplist.
///
/// # Example
///
/// ```
/// use rocksdb::{Options, MemtableFactory};
/// let mut opts = Options::default();
/// let factory = MemtableFactory::HashSkipList {
/// bucket_count: 1_000_000,
/// height: 4,
/// branching_factor: 4,
/// };
///
/// opts.set_allow_concurrent_memtable_write(false);
/// opts.set_memtable_factory(factory);
/// ```
pub fn set_memtable_factory(&mut self, factory: MemtableFactory) {
match factory {
MemtableFactory::Vector => unsafe {
ffi::rocksdb_options_set_memtable_vector_rep(self.inner);
},
MemtableFactory::HashSkipList {
bucket_count,
height,
branching_factor,
} => unsafe {
ffi::rocksdb_options_set_hash_skip_list_rep(
self.inner,
bucket_count,
height,
branching_factor,
);
},
MemtableFactory::HashLinkList { bucket_count } => unsafe {
ffi::rocksdb_options_set_hash_link_list_rep(self.inner, bucket_count);
},
};
}
pub fn set_block_based_table_factory(&mut self, factory: &BlockBasedOptions) { pub fn set_block_based_table_factory(&mut self, factory: &BlockBasedOptions) {
unsafe { unsafe {
ffi::rocksdb_options_set_block_based_table_factory(self.inner, factory.inner); ffi::rocksdb_options_set_block_based_table_factory(self.inner, factory.inner);
@ -980,6 +1060,59 @@ impl Options {
ffi::rocksdb_options_set_num_levels(self.inner, n); ffi::rocksdb_options_set_num_levels(self.inner, n);
} }
} }
/// When a `prefix_extractor` is defined through `opts.set_prefix_extractor` this
/// creates a prefix bloom filter for each memtable with the size of
/// `write_buffer_size * memtable_prefix_bloom_ratio` (capped at 0.25).
///
/// Default: `0`
///
/// # Example
///
/// ```
/// use rocksdb::{Options, SliceTransform};
///
/// let mut opts = Options::default();
/// let transform = SliceTransform::create_fixed_prefix(10);
/// opts.set_prefix_extractor(transform);
/// opts.set_memtable_prefix_bloom_ratio(0.2);
/// ```
pub fn set_memtable_prefix_bloom_ratio(&mut self, ratio: f64) {
unsafe {
ffi::rocksdb_options_set_memtable_prefix_bloom_size_ratio(self.inner, ratio);
}
}
/// Specifies the absolute path of the directory the
/// write-ahead log (WAL) should be written to.
///
/// Default: same directory as the database
///
/// # Example
///
/// ```
/// use rocksdb::Options;
///
/// let mut opts = Options::default();
/// opts.set_wal_dir("/path/to/dir");
/// ```
pub fn set_wal_dir<P: AsRef<Path>>(&mut self, path: P) {
let p = CString::new(path.as_ref().to_string_lossy().as_bytes()).unwrap();
unsafe {
ffi::rocksdb_options_set_wal_dir(self.inner, p.as_ptr());
}
}
/// If true, then DB::Open() will not update the statistics used to optimize
/// compaction decision by loading table properties from many files.
/// Turning off this feature will improve DBOpen time especially in disk environment.
///
/// Default: false
pub fn set_skip_stats_update_on_db_open(&mut self, skip: bool) {
unsafe {
ffi::rocksdb_options_set_skip_stats_update_on_db_open(self.inner, skip as c_uchar);
}
}
} }
impl Default for Options { impl Default for Options {
@ -1024,6 +1157,7 @@ impl Default for WriteOptions {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use MemtableFactory;
use Options; use Options;
#[test] #[test]
@ -1036,4 +1170,16 @@ mod tests {
let opts = Options::default(); let opts = Options::default();
assert!(opts.get_statistics().is_none()); assert!(opts.get_statistics().is_none());
} }
#[test]
fn test_set_memtable_factory() {
let mut opts = Options::default();
opts.set_memtable_factory(MemtableFactory::Vector);
opts.set_memtable_factory(MemtableFactory::HashLinkList { bucket_count: 100 });
opts.set_memtable_factory(MemtableFactory::HashSkipList {
bucket_count: 100,
height: 4,
branching_factor: 4,
});
}
} }

25
src/lib.rs
Unescape View File

@ -53,6 +53,7 @@ extern crate librocksdb_sys as ffi;
mod ffi_util; mod ffi_util;
pub mod backup; pub mod backup;
pub mod checkpoint;
mod comparator; mod comparator;
pub mod merge_operator; pub mod merge_operator;
pub mod compaction_filter; pub mod compaction_filter;
@ -136,6 +137,28 @@ pub struct BlockBasedOptions {
inner: *mut ffi::rocksdb_block_based_table_options_t, inner: *mut ffi::rocksdb_block_based_table_options_t,
} }
/// Used by BlockBasedOptions::set_index_type.
pub enum BlockBasedIndexType {
/// A space efficient index block that is optimized for
/// binary-search-based index.
BinarySearch,
/// The hash index, if enabled, will perform a hash lookup if
/// a prefix extractor has been provided through Options::set_prefix_extractor.
HashSearch,
/// A two-level index implementation. Both levels are binary search indexes.
TwoLevelIndexSearch,
}
/// Defines the underlying memtable implementation.
/// See https://github.com/facebook/rocksdb/wiki/MemTable for more information.
pub enum MemtableFactory {
Vector,
HashSkipList { bucket_count: usize, height: i32, branching_factor: i32 },
HashLinkList { bucket_count: usize }
}
/// Database-wide options around performance and behavior. /// Database-wide options around performance and behavior.
/// ///
/// Please read [the official tuning guide](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide), and most importantly, measure performance under realistic workloads with realistic hardware. /// Please read [the official tuning guide](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide), and most importantly, measure performance under realistic workloads with realistic hardware.
@ -206,3 +229,5 @@ pub struct WriteOptions {
pub struct ColumnFamily { pub struct ColumnFamily {
inner: *mut ffi::rocksdb_column_family_handle_t, inner: *mut ffi::rocksdb_column_family_handle_t,
} }
unsafe impl Send for ColumnFamily {}

6
src/merge_operator.rs
Unescape View File

@ -278,7 +278,7 @@ mod test {
#[repr(packed)] #[repr(packed)]
#[derive(Clone, Debug)] #[derive(Copy, Clone, Debug)]
struct ValueCounts { struct ValueCounts {
num_a: u32, num_a: u32,
num_b: u32, num_b: u32,
@ -423,7 +423,7 @@ mod test {
match db.get(b"k2") { match db.get(b"k2") {
Ok(Some(value)) => { Ok(Some(value)) => {
match from_slice::<ValueCounts>(&*value) { match from_slice::<ValueCounts>(&*value) {
Some(v) => { Some(v) => unsafe {
assert_eq!(v.num_a, 1000); assert_eq!(v.num_a, 1000);
assert_eq!(v.num_b, 500); assert_eq!(v.num_b, 500);
assert_eq!(v.num_c, 2000); assert_eq!(v.num_c, 2000);
@ -438,7 +438,7 @@ mod test {
match db.get(b"k1") { match db.get(b"k1") {
Ok(Some(value)) => { Ok(Some(value)) => {
match from_slice::<ValueCounts>(&*value) { match from_slice::<ValueCounts>(&*value) {
Some(v) => { Some(v) => unsafe {
assert_eq!(v.num_a, 3); assert_eq!(v.num_a, 3);
assert_eq!(v.num_b, 2); assert_eq!(v.num_b, 2);
assert_eq!(v.num_c, 0); assert_eq!(v.num_c, 0);

113
tests/test_checkpoint.rs
Unescape View File

@ -0,0 +1,113 @@
// Copyright 2018 Eugene P.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
extern crate rocksdb;
use rocksdb::{checkpoint::Checkpoint, DB, Options};
use std::fs::remove_dir_all;
#[test]
pub fn test_single_checkpoint() {
const PATH_PREFIX: &str = "_rust_rocksdb_cp_single_";
// Create DB with some data
let db_path = format!("{}db1", PATH_PREFIX);
let _ = remove_dir_all(&db_path);
let mut opts = Options::default();
opts.create_if_missing(true);
let db = DB::open(&opts, &db_path).unwrap();
db.put(b"k1", b"v1").unwrap();
db.put(b"k2", b"v2").unwrap();
db.put(b"k3", b"v3").unwrap();
db.put(b"k4", b"v4").unwrap();
// Create checkpoint
let cp1 = Checkpoint::new(&db).unwrap();
let cp1_path = format!("{}cp1", PATH_PREFIX);
let _ = remove_dir_all(&cp1_path);
cp1.create_checkpoint(&cp1_path).unwrap();
// Verify checkpoint
let cp = DB::open_default(&cp1_path).unwrap();
assert_eq!(*cp.get(b"k1").unwrap().unwrap(), *b"v1");
assert_eq!(*cp.get(b"k2").unwrap().unwrap(), *b"v2");
assert_eq!(*cp.get(b"k3").unwrap().unwrap(), *b"v3");
assert_eq!(*cp.get(b"k4").unwrap().unwrap(), *b"v4");
let _ = remove_dir_all(&db_path);
let _ = remove_dir_all(&cp1_path);
}
#[test]
pub fn test_multi_checkpoints() {
const PATH_PREFIX: &str = "_rust_rocksdb_cp_multi_";
// Create DB with some data
let db_path = format!("{}db1", PATH_PREFIX);
let _ = remove_dir_all(&db_path);
let mut opts = Options::default();
opts.create_if_missing(true);
let db = DB::open(&opts, &db_path).unwrap();
db.put(b"k1", b"v1").unwrap();
db.put(b"k2", b"v2").unwrap();
db.put(b"k3", b"v3").unwrap();
db.put(b"k4", b"v4").unwrap();
// Create first checkpoint
let cp1 = Checkpoint::new(&db).unwrap();
let cp1_path = format!("{}cp1", PATH_PREFIX);
let _ = remove_dir_all(&cp1_path);
cp1.create_checkpoint(&cp1_path).unwrap();
// Verify checkpoint
let cp = DB::open_default(&cp1_path).unwrap();
assert_eq!(*cp.get(b"k1").unwrap().unwrap(), *b"v1");
assert_eq!(*cp.get(b"k2").unwrap().unwrap(), *b"v2");
assert_eq!(*cp.get(b"k3").unwrap().unwrap(), *b"v3");
assert_eq!(*cp.get(b"k4").unwrap().unwrap(), *b"v4");
let _ = remove_dir_all(&cp1_path);
// Change some existing keys
db.put(b"k1", b"modified").unwrap();
db.put(b"k2", b"changed").unwrap();
// Add some new keys
db.put(b"k5", b"v5").unwrap();
db.put(b"k6", b"v6").unwrap();
// Create another checkpoint
let cp2 = Checkpoint::new(&db).unwrap();
let cp2_path = format!("{}cp2", PATH_PREFIX);
let _ = remove_dir_all(&cp2_path);
cp2.create_checkpoint(&cp2_path).unwrap();
// Verify second checkpoint
let cp = DB::open_default(&cp2_path).unwrap();
assert_eq!(*cp.get(b"k1").unwrap().unwrap(), *b"modified");
assert_eq!(*cp.get(b"k2").unwrap().unwrap(), *b"changed");
assert_eq!(*cp.get(b"k5").unwrap().unwrap(), *b"v5");
assert_eq!(*cp.get(b"k6").unwrap().unwrap(), *b"v6");
let _ = remove_dir_all(&db_path);
let _ = remove_dir_all(&cp2_path);
}

61
tests/test_iterator.rs
Unescape View File

@ -14,7 +14,7 @@
// //
extern crate rocksdb; extern crate rocksdb;
use rocksdb::{DB, Direction, IteratorMode, Options}; use rocksdb::{DB, Direction, IteratorMode, MemtableFactory, Options};
fn cba(input: &Box<[u8]>) -> Box<[u8]> { fn cba(input: &Box<[u8]>) -> Box<[u8]> {
input.iter().cloned().collect::<Vec<_>>().into_boxed_slice() input.iter().cloned().collect::<Vec<_>>().into_boxed_slice()
@ -122,6 +122,11 @@ pub fn test_iterator() {
let expected = vec![(cba(&k2), cba(&v2)), (cba(&k1), cba(&v1))]; let expected = vec![(cba(&k2), cba(&v2)), (cba(&k1), cba(&v1))];
assert_eq!(iterator1.collect::<Vec<_>>(), expected); assert_eq!(iterator1.collect::<Vec<_>>(), expected);
} }
{
let iterator1 = db.iterator(IteratorMode::From(b"zz", Direction::Reverse));
let expected = vec![(cba(&k4), cba(&v4)), (cba(&k3), cba(&v3))];
assert_eq!(iterator1.take(2).collect::<Vec<_>>(), expected);
}
{ {
let iterator1 = db.iterator(IteratorMode::From(b"k0", Direction::Forward)); let iterator1 = db.iterator(IteratorMode::From(b"k0", Direction::Forward));
assert!(iterator1.valid()); assert!(iterator1.valid());
@ -132,13 +137,13 @@ pub fn test_iterator() {
let iterator4 = db.iterator(IteratorMode::From(b"k5", Direction::Forward)); let iterator4 = db.iterator(IteratorMode::From(b"k5", Direction::Forward));
assert!(!iterator4.valid()); assert!(!iterator4.valid());
let iterator5 = db.iterator(IteratorMode::From(b"k0", Direction::Reverse)); let iterator5 = db.iterator(IteratorMode::From(b"k0", Direction::Reverse));
assert!(iterator5.valid()); assert!(!iterator5.valid());
let iterator6 = db.iterator(IteratorMode::From(b"k1", Direction::Reverse)); let iterator6 = db.iterator(IteratorMode::From(b"k1", Direction::Reverse));
assert!(iterator6.valid()); assert!(iterator6.valid());
let iterator7 = db.iterator(IteratorMode::From(b"k11", Direction::Reverse)); let iterator7 = db.iterator(IteratorMode::From(b"k11", Direction::Reverse));
assert!(iterator7.valid()); assert!(iterator7.valid());
let iterator8 = db.iterator(IteratorMode::From(b"k5", Direction::Reverse)); let iterator8 = db.iterator(IteratorMode::From(b"k5", Direction::Reverse));
assert!(!iterator8.valid()); assert!(iterator8.valid());
} }
{ {
let mut iterator1 = db.iterator(IteratorMode::From(b"k4", Direction::Forward)); let mut iterator1 = db.iterator(IteratorMode::From(b"k4", Direction::Forward));
@ -188,4 +193,54 @@ pub fn test_prefix_iterator() {
assert_eq!(b_iterator.collect::<Vec<_>>(), expected) assert_eq!(b_iterator.collect::<Vec<_>>(), expected)
} }
} }
let opts = Options::default();
assert!(DB::destroy(&opts, path).is_ok());
}
#[test]
pub fn test_full_iterator() {
let path = "_rust_rocksdb_fulliteratortest";
{
let a1: Box<[u8]> = key(b"aaa1");
let a2: Box<[u8]> = key(b"aaa2");
let b1: Box<[u8]> = key(b"bbb1");
let b2: Box<[u8]> = key(b"bbb2");
let prefix_extractor = rocksdb::SliceTransform::create_fixed_prefix(3);
let factory = MemtableFactory::HashSkipList {
bucket_count: 1_000_000,
height: 4,
branching_factor: 4,
};
let mut opts = Options::default();
opts.create_if_missing(true);
opts.set_prefix_extractor(prefix_extractor);
opts.set_allow_concurrent_memtable_write(false);
opts.set_memtable_factory(factory);
let db = DB::open(&opts, path).unwrap();
assert!(db.put(&*a1, &*a1).is_ok());
assert!(db.put(&*a2, &*a2).is_ok());
assert!(db.put(&*b1, &*b1).is_ok());
assert!(db.put(&*b2, &*b2).is_ok());
// A normal iterator won't work here since we're using a HashSkipList for our memtable
// implementation (which buckets keys based on their prefix):
let bad_iterator = db.iterator(IteratorMode::Start);
assert_eq!(bad_iterator.collect::<Vec<_>>(), vec![]);
let expected = vec![
(cba(&a1), cba(&a1)),
(cba(&a2), cba(&a2)),
(cba(&b1), cba(&b1)),
(cba(&b2), cba(&b2)),
];
let a_iterator = db.full_iterator(IteratorMode::Start);
assert_eq!(a_iterator.collect::<Vec<_>>(), expected)
}
let opts = Options::default();
assert!(DB::destroy(&opts, path).is_ok());
} }

Write Preview
Loading…
Cancel
Save