21e8daced5
fix assertion failure in Prev()
...
Summary:
fix assertion failure in db_stress.
It happens because of prefix seek key is larger than merge iterator key when they have the same user key
Test Plan: ./db_stress --max_background_compactions=1 --max_write_buffer_number=3 --sync=0 --reopen=20 --write_buffer_size=33554432 --delpercent=5 --log2_keys_per_lock=10 --block_size=16384 --allow_concurrent_memtable_write=0 --test_batches_snapshots=0 --max_bytes_for_level_base=67108864 --progress_reports=0 --mmap_read=0 --writepercent=35 --disable_data_sync=0 --readpercent=50 --subcompactions=4 --ops_per_thread=20000000 --memtablerep=skip_list --prefix_size=0 --target_file_size_multiplier=1 --column_families=1 --threads=32 --disable_wal=0 --open_files=500000 --destroy_db_initially=0 --target_file_size_base=16777216 --nooverwritepercent=1 --iterpercent=10 --max_key=100000000 --prefixpercent=0 --use_clock_cache=false --kill_random_test=888887 --cache_size=1048576 --verify_checksum=1
Reviewers: sdong, andrewkr, yiwu, yhchiang
Reviewed By: yhchiang
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D65025
8 years ago
f517d9dd09
Add SeekForPrev() to Iterator
...
Summary:
Add new Iterator API, `SeekForPrev`: find the last key that <= target key
support prefix_extractor
support prefix_same_as_start
support upper_bound
not supported in iterators without Prev()
Also add tests in db_iter_test and db_iterator_test
Pass all tests
Cheers!
Test Plan: make all check -j64
Reviewers: andrewkr, yiwu, IslamAbdelRahman, sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D64149
8 years ago
9ab38c45ad
Remove %z Format Specifier and Fix Windows Build of sim_cache.cc ( #1224 )
...
* Replace %zu format specifier with Windows-compatible macro 'ROCKSDB_PRIszt'
* Added "port/port.h" include to sim_cache.cc for call to snprintf().
* Applied cleaner fix to windows build, reverting part of 7bedd94
8 years ago
730f7e2e21
Fix win build
...
Summary: Fixing error with win build where we compare int64_t with size_t.
Test Plan: make check
Reviewers: andrewkr
Reviewed By: andrewkr
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D57885
8 years ago
967476eaee
Fix valgrind (DBIteratorTest.ReadAhead)
...
Summary: This test is failing under valgrind because we dont delete the Env that we allocated
Test Plan: run the test under valgrind
Reviewers: andrewkr, yhchiang, yiwu, sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D57693
8 years ago
a4ea345b04
Fixing lite build
...
Summary: Fixing lite build broke in unit test. `FilesPerLevel()` depends on `DB::GetProperty()`, which lite build doesn't support.
Test Plan: OPT=-DROCKSDB_LITE make check -j64
Reviewers: sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba, leveldb
Differential Revision: https://reviews.facebook.net/D57651
8 years ago
24a24f013d
Enable configurable readahead for iterators
...
Summary:
Add an option `iterator_readahead_size` to `ReadOptions` to enable
configurable readahead for iterators similar to the corresponding
option for compaction.
Test Plan:
```
make commit_prereq
```
Reviewers: kumar.rangarajan, ott, igor, sdong
Reviewed By: sdong
Subscribers: yiwu, andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D55419
8 years ago
ff4b3fb5b4
Fix Iterator::Prev memory pinning bug
...
Summary: We should not use IterKey::SetKey with copy = false except if we are pinning the iterator thru it's life time, otherwise we may release the temporarily pinned blocks and in this case the IterKey will be pointing to freed memory
Test Plan: added a new test
Reviewers: sdong, andrewkr
Reviewed By: andrewkr
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D57561
8 years ago
6e801b0bd1
Eliminate memcpy in Iterator::Prev() by pinning blocks for keys spanning multiple blocks
...
Summary:
This diff is stacked on top of this diff https://reviews.facebook.net/D56493
The current Iterator::Prev() implementation need to copy every value since the underlying Iterator may move after reading the value.
This can be optimized by making sure that the block containing the value is pinned until the Iterator move. which will improve the throughput by up to 1.5X
master
```
==> 1000000_Keys_100Byte.txt <==
readreverse : 0.449 micros/op 2225887 ops/sec; 246.2 MB/s
readreverse : 0.433 micros/op 2311508 ops/sec; 255.7 MB/s
readreverse : 0.436 micros/op 2294335 ops/sec; 253.8 MB/s
readreverse : 0.471 micros/op 2121295 ops/sec; 234.7 MB/s
readreverse : 0.465 micros/op 2152227 ops/sec; 238.1 MB/s
readreverse : 0.454 micros/op 2203011 ops/sec; 243.7 MB/s
readreverse : 0.451 micros/op 2216095 ops/sec; 245.2 MB/s
readreverse : 0.462 micros/op 2162447 ops/sec; 239.2 MB/s
readreverse : 0.476 micros/op 2099151 ops/sec; 232.2 MB/s
readreverse : 0.472 micros/op 2120710 ops/sec; 234.6 MB/s
avg : 242.34 MB/s
==> 1000000_Keys_1KB.txt <==
readreverse : 1.013 micros/op 986793 ops/sec; 978.7 MB/s
readreverse : 0.942 micros/op 1061136 ops/sec; 1052.5 MB/s
readreverse : 0.951 micros/op 1051901 ops/sec; 1043.3 MB/s
readreverse : 0.932 micros/op 1072894 ops/sec; 1064.1 MB/s
readreverse : 1.024 micros/op 976720 ops/sec; 968.7 MB/s
readreverse : 0.935 micros/op 1069169 ops/sec; 1060.4 MB/s
readreverse : 1.012 micros/op 988132 ops/sec; 980.1 MB/s
readreverse : 0.962 micros/op 1039579 ops/sec; 1031.1 MB/s
readreverse : 0.991 micros/op 1008924 ops/sec; 1000.7 MB/s
readreverse : 1.004 micros/op 996144 ops/sec; 988.0 MB/s
avg : 1016.76 MB/s
==> 1000000_Keys_10KB.txt <==
readreverse : 4.167 micros/op 239952 ops/sec; 2346.9 MB/s
readreverse : 4.070 micros/op 245713 ops/sec; 2403.3 MB/s
readreverse : 4.572 micros/op 218733 ops/sec; 2139.4 MB/s
readreverse : 4.497 micros/op 222388 ops/sec; 2175.2 MB/s
readreverse : 4.203 micros/op 237920 ops/sec; 2327.1 MB/s
readreverse : 4.206 micros/op 237756 ops/sec; 2325.5 MB/s
readreverse : 4.181 micros/op 239149 ops/sec; 2339.1 MB/s
readreverse : 4.157 micros/op 240552 ops/sec; 2352.8 MB/s
readreverse : 4.187 micros/op 238848 ops/sec; 2336.1 MB/s
readreverse : 4.106 micros/op 243575 ops/sec; 2382.4 MB/s
avg : 2312.78 MB/s
==> 100000_Keys_100KB.txt <==
readreverse : 41.281 micros/op 24224 ops/sec; 2366.0 MB/s
readreverse : 39.722 micros/op 25175 ops/sec; 2458.9 MB/s
readreverse : 40.319 micros/op 24802 ops/sec; 2422.5 MB/s
readreverse : 39.762 micros/op 25149 ops/sec; 2456.4 MB/s
readreverse : 40.916 micros/op 24440 ops/sec; 2387.1 MB/s
readreverse : 41.188 micros/op 24278 ops/sec; 2371.4 MB/s
readreverse : 40.061 micros/op 24962 ops/sec; 2438.1 MB/s
readreverse : 40.221 micros/op 24862 ops/sec; 2428.4 MB/s
readreverse : 40.084 micros/op 24947 ops/sec; 2436.7 MB/s
readreverse : 40.655 micros/op 24597 ops/sec; 2402.4 MB/s
avg : 2416.79 MB/s
==> 10000_Keys_1MB.txt <==
readreverse : 298.038 micros/op 3355 ops/sec; 3355.3 MB/s
readreverse : 335.001 micros/op 2985 ops/sec; 2985.1 MB/s
readreverse : 286.956 micros/op 3484 ops/sec; 3484.9 MB/s
readreverse : 329.954 micros/op 3030 ops/sec; 3030.8 MB/s
readreverse : 306.428 micros/op 3263 ops/sec; 3263.5 MB/s
readreverse : 330.749 micros/op 3023 ops/sec; 3023.5 MB/s
readreverse : 328.903 micros/op 3040 ops/sec; 3040.5 MB/s
readreverse : 324.853 micros/op 3078 ops/sec; 3078.4 MB/s
readreverse : 320.488 micros/op 3120 ops/sec; 3120.3 MB/s
readreverse : 320.536 micros/op 3119 ops/sec; 3119.8 MB/s
avg : 3150.21 MB/s
```
After memcpy elimination
```
==> 1000000_Keys_100Byte.txt <==
readreverse : 0.395 micros/op 2529890 ops/sec; 279.9 MB/s
readreverse : 0.368 micros/op 2715922 ops/sec; 300.5 MB/s
readreverse : 0.384 micros/op 2603929 ops/sec; 288.1 MB/s
readreverse : 0.375 micros/op 2663286 ops/sec; 294.6 MB/s
readreverse : 0.357 micros/op 2802180 ops/sec; 310.0 MB/s
readreverse : 0.363 micros/op 2757684 ops/sec; 305.1 MB/s
readreverse : 0.372 micros/op 2689603 ops/sec; 297.5 MB/s
readreverse : 0.379 micros/op 2638599 ops/sec; 291.9 MB/s
readreverse : 0.375 micros/op 2663803 ops/sec; 294.7 MB/s
readreverse : 0.375 micros/op 2665579 ops/sec; 294.9 MB/s
avg: 295.72 MB/s (1.22 X)
==> 1000000_Keys_1KB.txt <==
readreverse : 0.879 micros/op 1138112 ops/sec; 1128.8 MB/s
readreverse : 0.842 micros/op 1187998 ops/sec; 1178.3 MB/s
readreverse : 0.837 micros/op 1194915 ops/sec; 1185.1 MB/s
readreverse : 0.845 micros/op 1182983 ops/sec; 1173.3 MB/s
readreverse : 0.877 micros/op 1140308 ops/sec; 1131.0 MB/s
readreverse : 0.849 micros/op 1177581 ops/sec; 1168.0 MB/s
readreverse : 0.915 micros/op 1093284 ops/sec; 1084.3 MB/s
readreverse : 0.863 micros/op 1159418 ops/sec; 1149.9 MB/s
readreverse : 0.895 micros/op 1117670 ops/sec; 1108.5 MB/s
readreverse : 0.852 micros/op 1174116 ops/sec; 1164.5 MB/s
avg: 1147.17 MB/s (1.12 X)
==> 1000000_Keys_10KB.txt <==
readreverse : 3.870 micros/op 258386 ops/sec; 2527.2 MB/s
readreverse : 3.568 micros/op 280296 ops/sec; 2741.5 MB/s
readreverse : 4.005 micros/op 249694 ops/sec; 2442.2 MB/s
readreverse : 3.550 micros/op 281719 ops/sec; 2755.5 MB/s
readreverse : 3.562 micros/op 280758 ops/sec; 2746.1 MB/s
readreverse : 3.507 micros/op 285125 ops/sec; 2788.8 MB/s
readreverse : 3.463 micros/op 288739 ops/sec; 2824.1 MB/s
readreverse : 3.428 micros/op 291734 ops/sec; 2853.4 MB/s
readreverse : 3.553 micros/op 281491 ops/sec; 2753.2 MB/s
readreverse : 3.535 micros/op 282885 ops/sec; 2766.9 MB/s
avg : 2719.89 MB/s (1.17 X)
==> 100000_Keys_100KB.txt <==
readreverse : 22.815 micros/op 43830 ops/sec; 4281.0 MB/s
readreverse : 29.957 micros/op 33381 ops/sec; 3260.4 MB/s
readreverse : 25.334 micros/op 39473 ops/sec; 3855.4 MB/s
readreverse : 23.037 micros/op 43409 ops/sec; 4239.8 MB/s
readreverse : 27.810 micros/op 35958 ops/sec; 3512.1 MB/s
readreverse : 30.327 micros/op 32973 ops/sec; 3220.6 MB/s
readreverse : 29.704 micros/op 33665 ops/sec; 3288.2 MB/s
readreverse : 29.423 micros/op 33987 ops/sec; 3319.6 MB/s
readreverse : 23.334 micros/op 42856 ops/sec; 4185.9 MB/s
readreverse : 29.969 micros/op 33368 ops/sec; 3259.1 MB/s
avg : 3642.21 MB/s (1.5 X)
==> 10000_Keys_1MB.txt <==
readreverse : 244.748 micros/op 4085 ops/sec; 4085.9 MB/s
readreverse : 230.208 micros/op 4343 ops/sec; 4344.0 MB/s
readreverse : 235.655 micros/op 4243 ops/sec; 4243.6 MB/s
readreverse : 235.730 micros/op 4242 ops/sec; 4242.2 MB/s
readreverse : 237.346 micros/op 4213 ops/sec; 4213.3 MB/s
readreverse : 227.306 micros/op 4399 ops/sec; 4399.4 MB/s
readreverse : 194.957 micros/op 5129 ops/sec; 5129.4 MB/s
readreverse : 238.359 micros/op 4195 ops/sec; 4195.4 MB/s
readreverse : 221.588 micros/op 4512 ops/sec; 4513.0 MB/s
readreverse : 235.911 micros/op 4238 ops/sec; 4239.0 MB/s
avg : 4360.52 MB/s (1.38 X)
```
Test Plan: COMPILE_WITH_ASAN=1 make check -j64
Reviewers: andrewkr, yhchiang, sdong
Reviewed By: sdong
Subscribers: andrewkr, dhruba
Differential Revision: https://reviews.facebook.net/D56511
9 years ago
6d4832a998
Merge pull request #1101 from flyd1005/wip-fix-typo
...
fix typos and remove duplicated words
9 years ago
792762c42c
Split db_test.cc
...
Summary: Split db_test.cc into several files. Moving several helper functions into DBTestBase.
Test Plan: make check
Reviewers: sdong, yhchiang, IslamAbdelRahman
Reviewed By: IslamAbdelRahman
Subscribers: dhruba, andrewkr, kradhakrishnan, yhchiang, leveldb, sdong
Differential Revision: https://reviews.facebook.net/D56715
9 years ago
Aaron Gao