Advisor: README and blog, and also tests for DBBenchRunner, DatabaseOptions (#4201)
Summary: This pull request adds a README file and a blog post for the Advisor tool. It also adds the missing tests for some Optimizer modules. Some comments are added to the classes being tested for improved readability. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4201 Reviewed By: maysamyabandeh Differential Revision: D9125311 Pulled By: poojam23 fbshipit-source-id: aefcf2f06eaa05490cc2834ef5aa6e21f0d1dc55main
Pooja Malik
6 years ago
committed by
Facebook Github Bot
parent
f8f6983f89
commit
892a156267
@ -0,0 +1,58 @@ |
|||||||
|
--- |
||||||
|
title: Rocksdb Tuning Advisor |
||||||
|
layout: post |
||||||
|
author: poojam23 |
||||||
|
category: blog |
||||||
|
--- |
||||||
|
|
||||||
|
The performance of Rocksdb is contingent on its tuning. However, because |
||||||
|
of the complexity of its underlying technology and a large number of |
||||||
|
configurable parameters, a good configuration is sometimes hard to obtain. The aim of |
||||||
|
the python command-line tool, Rocksdb Advisor, is to automate the process of |
||||||
|
suggesting improvements in the configuration based on advice from Rocksdb |
||||||
|
experts. |
||||||
|
|
||||||
|
### Overview |
||||||
|
|
||||||
|
Experts share their wisdom as rules comprising of conditions and suggestions in the INI format (refer |
||||||
|
[rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)). |
||||||
|
Users provide the Rocksdb configuration that they want to improve upon (as the |
||||||
|
familiar Rocksdb OPTIONS file — |
||||||
|
[example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini)) |
||||||
|
and the path of the file which contains Rocksdb logs and statistics. |
||||||
|
The [Advisor](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser_example.py) |
||||||
|
creates appropriate DataSource objects (for Rocksdb |
||||||
|
[logs](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_log_parser.py), |
||||||
|
[options](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_options_parser.py), |
||||||
|
[statistics](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_stats_fetcher.py) etc.) |
||||||
|
and provides them to the [Rules Engine](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser.py). |
||||||
|
The Rules uses rules from experts to parse data-sources and trigger appropriate rules. |
||||||
|
The Advisor's output gives information about which rules were triggered, |
||||||
|
why they were triggered and what each of them suggests. Each suggestion |
||||||
|
provided by a triggered rule advises some action on a Rocksdb |
||||||
|
configuration option, for example, increase CFOptions.write_buffer_size, |
||||||
|
set bloom_bits to 2 etc. |
||||||
|
|
||||||
|
### Usage |
||||||
|
|
||||||
|
An example command to run the tool: |
||||||
|
|
||||||
|
```shell |
||||||
|
cd rocksdb/tools/advisor |
||||||
|
python3 -m advisor.rule_parser_example --rules_spec=advisor/rules.ini --rocksdb_options=test/input_files/OPTIONS-000005 --log_files_path_prefix=test/input_files/LOG-0 --stats_dump_period_sec=20 |
||||||
|
``` |
||||||
|
|
||||||
|
Sample output where a Rocksdb log-based rule has been triggered : |
||||||
|
|
||||||
|
```shell |
||||||
|
Rule: stall-too-many-memtables |
||||||
|
LogCondition: stall-too-many-memtables regex: Stopping writes because we have \d+ immutable memtables \(waiting for flush\), max_write_buffer_number is set to \d+ |
||||||
|
Suggestion: inc-bg-flush option : DBOptions.max_background_flushes action : increase suggested_values : ['2'] |
||||||
|
Suggestion: inc-write-buffer option : CFOptions.max_write_buffer_number action : increase |
||||||
|
scope: col_fam: |
||||||
|
{'default'} |
||||||
|
``` |
||||||
|
|
||||||
|
### Read more |
||||||
|
|
||||||
|
For more information, refer to [advisor](https://github.com/facebook/rocksdb/tree/master/tools/advisor/README.md). |
||||||
@ -0,0 +1,96 @@ |
|||||||
|
# Rocksdb Tuning Advisor |
||||||
|
|
||||||
|
## Motivation |
||||||
|
|
||||||
|
The performance of Rocksdb is contingent on its tuning. However, |
||||||
|
because of the complexity of its underlying technology and a large number of |
||||||
|
configurable parameters, a good configuration is sometimes hard to obtain. The aim of |
||||||
|
the python command-line tool, Rocksdb Advisor, is to automate the process of |
||||||
|
suggesting improvements in the configuration based on advice from Rocksdb |
||||||
|
experts. |
||||||
|
|
||||||
|
## Overview |
||||||
|
|
||||||
|
Experts share their wisdom as rules comprising of conditions and suggestions in the INI format (refer |
||||||
|
[rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)). |
||||||
|
Users provide the Rocksdb configuration that they want to improve upon (as the |
||||||
|
familiar Rocksdb OPTIONS file — |
||||||
|
[example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini)) |
||||||
|
and the path of the file which contains Rocksdb logs and statistics. |
||||||
|
The [Advisor](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser_example.py) |
||||||
|
creates appropriate DataSource objects (for Rocksdb |
||||||
|
[logs](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_log_parser.py), |
||||||
|
[options](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_options_parser.py), |
||||||
|
[statistics](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_stats_fetcher.py) etc.) |
||||||
|
and provides them to the [Rules Engine](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser.py). |
||||||
|
The Rules uses rules from experts to parse data-sources and trigger appropriate rules. |
||||||
|
The Advisor's output gives information about which rules were triggered, |
||||||
|
why they were triggered and what each of them suggests. Each suggestion |
||||||
|
provided by a triggered rule advises some action on a Rocksdb |
||||||
|
configuration option, for example, increase CFOptions.write_buffer_size, |
||||||
|
set bloom_bits to 2 etc. |
||||||
|
|
||||||
|
## Usage |
||||||
|
|
||||||
|
### Prerequisites |
||||||
|
The tool needs the following to run: |
||||||
|
* python3 |
||||||
|
|
||||||
|
### Running the tool |
||||||
|
An example command to run the tool: |
||||||
|
|
||||||
|
```shell |
||||||
|
cd rocksdb/tools/advisor |
||||||
|
python3 -m advisor.rule_parser_example --rules_spec=advisor/rules.ini --rocksdb_options=test/input_files/OPTIONS-000005 --log_files_path_prefix=test/input_files/LOG-0 --stats_dump_period_sec=20 |
||||||
|
``` |
||||||
|
|
||||||
|
### Command-line arguments |
||||||
|
|
||||||
|
Most important amongst all the input that the Advisor needs, are the rules |
||||||
|
spec and starting Rocksdb configuration. The configuration is provided as the |
||||||
|
familiar Rocksdb Options file (refer [example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini)). |
||||||
|
The Rules spec is written in the INI format (more details in |
||||||
|
[rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)). |
||||||
|
|
||||||
|
In brief, a Rule is made of conditions and is triggered when all its |
||||||
|
constituent conditions are triggered. When triggered, a Rule suggests changes |
||||||
|
(increase/decrease/set to a suggested value) to certain Rocksdb options that |
||||||
|
aim to improve Rocksdb performance. Every Condition has a 'source' i.e. |
||||||
|
the data source that would be checked for triggering that condition. |
||||||
|
For example, a log Condition (with 'source=LOG') is triggered if a particular |
||||||
|
'regex' is found in the Rocksdb LOG files. As of now the Rules Engine |
||||||
|
supports 3 types of Conditions (and consequently data-sources): |
||||||
|
LOG, OPTIONS, TIME_SERIES. The TIME_SERIES data can be sourced from the |
||||||
|
Rocksdb [statistics](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/statistics.h) |
||||||
|
or [perf context](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/perf_context.h). |
||||||
|
|
||||||
|
For more information about the remaining command-line arguments, run: |
||||||
|
|
||||||
|
```shell |
||||||
|
cd rocksdb/tools/advisor |
||||||
|
python3 -m advisor.rule_parser_example --help |
||||||
|
``` |
||||||
|
|
||||||
|
### Sample output |
||||||
|
|
||||||
|
Here, a Rocksdb log-based rule has been triggered: |
||||||
|
|
||||||
|
```shell |
||||||
|
Rule: stall-too-many-memtables |
||||||
|
LogCondition: stall-too-many-memtables regex: Stopping writes because we have \d+ immutable memtables \(waiting for flush\), max_write_buffer_number is set to \d+ |
||||||
|
Suggestion: inc-bg-flush option : DBOptions.max_background_flushes action : increase suggested_values : ['2'] |
||||||
|
Suggestion: inc-write-buffer option : CFOptions.max_write_buffer_number action : increase |
||||||
|
scope: col_fam: |
||||||
|
{'default'} |
||||||
|
``` |
||||||
|
|
||||||
|
## Running the tests |
||||||
|
|
||||||
|
Tests for the code have been added to the |
||||||
|
[test/](https://github.com/facebook/rocksdb/tree/master/tools/advisor/test) |
||||||
|
directory. For example, to run the unit tests for db_log_parser.py: |
||||||
|
|
||||||
|
```shell |
||||||
|
cd rocksdb/tools/advisor |
||||||
|
python3 -m unittest -v test.test_db_log_parser |
||||||
|
``` |
||||||
@ -0,0 +1,89 @@ |
|||||||
|
# Copyright (c) 2011-present, Facebook, Inc. All rights reserved. |
||||||
|
# This source code is licensed under both the GPLv2 (found in the |
||||||
|
# COPYING file in the root directory) and Apache 2.0 License |
||||||
|
# (found in the LICENSE.Apache file in the root directory). |
||||||
|
|
||||||
|
from advisor.rule_parser import RulesSpec |
||||||
|
from advisor.db_log_parser import DatabaseLogs, DataSource |
||||||
|
from advisor.db_options_parser import DatabaseOptions |
||||||
|
from advisor.db_stats_fetcher import LogStatsParser, OdsStatsFetcher |
||||||
|
import argparse |
||||||
|
|
||||||
|
|
||||||
|
def main(args): |
||||||
|
# initialise the RulesSpec parser |
||||||
|
rule_spec_parser = RulesSpec(args.rules_spec) |
||||||
|
rule_spec_parser.load_rules_from_spec() |
||||||
|
rule_spec_parser.perform_section_checks() |
||||||
|
# initialize the DatabaseOptions object |
||||||
|
db_options = DatabaseOptions(args.rocksdb_options) |
||||||
|
# Create DatabaseLogs object |
||||||
|
db_logs = DatabaseLogs( |
||||||
|
args.log_files_path_prefix, db_options.get_column_families() |
||||||
|
) |
||||||
|
# Create the Log STATS object |
||||||
|
db_log_stats = LogStatsParser( |
||||||
|
args.log_files_path_prefix, args.stats_dump_period_sec |
||||||
|
) |
||||||
|
data_sources = { |
||||||
|
DataSource.Type.DB_OPTIONS: [db_options], |
||||||
|
DataSource.Type.LOG: [db_logs], |
||||||
|
DataSource.Type.TIME_SERIES: [db_log_stats] |
||||||
|
} |
||||||
|
if args.ods_client: |
||||||
|
data_sources[DataSource.Type.TIME_SERIES].append(OdsStatsFetcher( |
||||||
|
args.ods_client, |
||||||
|
args.ods_entity, |
||||||
|
args.ods_tstart, |
||||||
|
args.ods_tend, |
||||||
|
args.ods_key_prefix |
||||||
|
)) |
||||||
|
triggered_rules = rule_spec_parser.get_triggered_rules( |
||||||
|
data_sources, db_options.get_column_families() |
||||||
|
) |
||||||
|
rule_spec_parser.print_rules(triggered_rules) |
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__': |
||||||
|
parser = argparse.ArgumentParser(description='Use this script to get\ |
||||||
|
suggestions for improving Rocksdb performance.') |
||||||
|
parser.add_argument( |
||||||
|
'--rules_spec', required=True, type=str, |
||||||
|
help='path of the file containing the expert-specified Rules' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--rocksdb_options', required=True, type=str, |
||||||
|
help='path of the starting Rocksdb OPTIONS file' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--log_files_path_prefix', required=True, type=str, |
||||||
|
help='path prefix of the Rocksdb LOG files' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--stats_dump_period_sec', required=True, type=int, |
||||||
|
help='the frequency (in seconds) at which STATISTICS are printed to ' + |
||||||
|
'the Rocksdb LOG file' |
||||||
|
) |
||||||
|
# ODS arguments |
||||||
|
parser.add_argument( |
||||||
|
'--ods_client', type=str, help='the ODS client binary' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--ods_entity', type=str, |
||||||
|
help='the servers for which the ODS stats need to be fetched' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--ods_key_prefix', type=str, |
||||||
|
help='the prefix that needs to be attached to the keys of time ' + |
||||||
|
'series to be fetched from ODS' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--ods_tstart', type=int, |
||||||
|
help='start time of timeseries to be fetched from ODS' |
||||||
|
) |
||||||
|
parser.add_argument( |
||||||
|
'--ods_tend', type=int, |
||||||
|
help='end time of timeseries to be fetched from ODS' |
||||||
|
) |
||||||
|
args = parser.parse_args() |
||||||
|
main(args) |
||||||
@ -0,0 +1,3 @@ |
|||||||
|
rocksdb.number.block.decompressed.count: 1530896335 88.0, 1530896361 788338.0, 1530896387 1539256.0, 1530896414 2255696.0, 1530896440 3009325.0, 1530896466 3767183.0, 1530896492 4529775.0, 1530896518 5297809.0, 1530896545 6033802.0, 1530896570 6794129.0 |
||||||
|
rocksdb.db.get.micros.p50: 1530896335 295.5, 1530896361 16.561841, 1530896387 16.20677, 1530896414 16.31508, 1530896440 16.346602, 1530896466 16.284669, 1530896492 16.16005, 1530896518 16.069096, 1530896545 16.028746, 1530896570 15.9638 |
||||||
|
rocksdb.manifest.file.sync.micros.p99: 1530896335 649.0, 1530896361 835.0, 1530896387 1435.0, 1530896414 9938.0, 1530896440 9938.0, 1530896466 9938.0, 1530896492 9938.0, 1530896518 1882.0, 1530896545 1837.0, 1530896570 1792.0 |
||||||
@ -0,0 +1,147 @@ |
|||||||
|
# Copyright (c) 2011-present, Facebook, Inc. All rights reserved. |
||||||
|
# This source code is licensed under both the GPLv2 (found in the |
||||||
|
# COPYING file in the root directory) and Apache 2.0 License |
||||||
|
# (found in the LICENSE.Apache file in the root directory). |
||||||
|
|
||||||
|
from advisor.db_bench_runner import DBBenchRunner |
||||||
|
from advisor.db_log_parser import NO_COL_FAMILY, DataSource |
||||||
|
from advisor.db_options_parser import DatabaseOptions |
||||||
|
import os |
||||||
|
import unittest |
||||||
|
|
||||||
|
|
||||||
|
class TestDBBenchRunnerMethods(unittest.TestCase): |
||||||
|
def setUp(self): |
||||||
|
self.pos_args = [ |
||||||
|
'./../../db_bench', |
||||||
|
'overwrite', |
||||||
|
'use_existing_db=true', |
||||||
|
'duration=10' |
||||||
|
] |
||||||
|
self.bench_runner = DBBenchRunner(self.pos_args) |
||||||
|
this_path = os.path.abspath(os.path.dirname(__file__)) |
||||||
|
options_path = os.path.join(this_path, 'input_files/OPTIONS-000005') |
||||||
|
self.db_options = DatabaseOptions(options_path) |
||||||
|
|
||||||
|
def test_setup(self): |
||||||
|
self.assertEqual(self.bench_runner.db_bench_binary, self.pos_args[0]) |
||||||
|
self.assertEqual(self.bench_runner.benchmark, self.pos_args[1]) |
||||||
|
self.assertSetEqual( |
||||||
|
set(self.bench_runner.db_bench_args), set(self.pos_args[2:]) |
||||||
|
) |
||||||
|
|
||||||
|
def test_get_info_log_file_name(self): |
||||||
|
log_file_name = DBBenchRunner.get_info_log_file_name( |
||||||
|
None, 'random_path' |
||||||
|
) |
||||||
|
self.assertEqual(log_file_name, 'LOG') |
||||||
|
|
||||||
|
log_file_name = DBBenchRunner.get_info_log_file_name( |
||||||
|
'/dev/shm/', '/tmp/rocksdbtest-155919/dbbench/' |
||||||
|
) |
||||||
|
self.assertEqual(log_file_name, 'tmp_rocksdbtest-155919_dbbench_LOG') |
||||||
|
|
||||||
|
def test_get_opt_args_str(self): |
||||||
|
misc_opt_dict = {'bloom_bits': 2, 'empty_opt': None, 'rate_limiter': 3} |
||||||
|
optional_args_str = DBBenchRunner.get_opt_args_str(misc_opt_dict) |
||||||
|
self.assertEqual(optional_args_str, ' --bloom_bits=2 --rate_limiter=3') |
||||||
|
|
||||||
|
def test_get_log_options(self): |
||||||
|
db_path = '/tmp/rocksdb-155919/dbbench' |
||||||
|
# when db_log_dir is present in the db_options |
||||||
|
update_dict = { |
||||||
|
'DBOptions.db_log_dir': {NO_COL_FAMILY: '/dev/shm'}, |
||||||
|
'DBOptions.stats_dump_period_sec': {NO_COL_FAMILY: '20'} |
||||||
|
} |
||||||
|
self.db_options.update_options(update_dict) |
||||||
|
log_file_prefix, stats_freq = self.bench_runner.get_log_options( |
||||||
|
self.db_options, db_path |
||||||
|
) |
||||||
|
self.assertEqual( |
||||||
|
log_file_prefix, '/dev/shm/tmp_rocksdb-155919_dbbench_LOG' |
||||||
|
) |
||||||
|
self.assertEqual(stats_freq, 20) |
||||||
|
|
||||||
|
update_dict = { |
||||||
|
'DBOptions.db_log_dir': {NO_COL_FAMILY: None}, |
||||||
|
'DBOptions.stats_dump_period_sec': {NO_COL_FAMILY: '30'} |
||||||
|
} |
||||||
|
self.db_options.update_options(update_dict) |
||||||
|
log_file_prefix, stats_freq = self.bench_runner.get_log_options( |
||||||
|
self.db_options, db_path |
||||||
|
) |
||||||
|
self.assertEqual(log_file_prefix, '/tmp/rocksdb-155919/dbbench/LOG') |
||||||
|
self.assertEqual(stats_freq, 30) |
||||||
|
|
||||||
|
def test_build_experiment_command(self): |
||||||
|
# add some misc_options to db_options |
||||||
|
update_dict = { |
||||||
|
'bloom_bits': {NO_COL_FAMILY: 2}, |
||||||
|
'rate_limiter_bytes_per_sec': {NO_COL_FAMILY: 128000000} |
||||||
|
} |
||||||
|
self.db_options.update_options(update_dict) |
||||||
|
db_path = '/dev/shm' |
||||||
|
experiment_command = self.bench_runner._build_experiment_command( |
||||||
|
self.db_options, db_path |
||||||
|
) |
||||||
|
opt_args_str = DBBenchRunner.get_opt_args_str( |
||||||
|
self.db_options.get_misc_options() |
||||||
|
) |
||||||
|
opt_args_str += ( |
||||||
|
' --options_file=' + |
||||||
|
self.db_options.generate_options_config('12345') |
||||||
|
) |
||||||
|
for arg in self.pos_args[2:]: |
||||||
|
opt_args_str += (' --' + arg) |
||||||
|
expected_command = ( |
||||||
|
self.pos_args[0] + ' --benchmarks=' + self.pos_args[1] + |
||||||
|
' --statistics --perf_level=3 --db=' + db_path + opt_args_str |
||||||
|
) |
||||||
|
self.assertEqual(experiment_command, expected_command) |
||||||
|
|
||||||
|
|
||||||
|
class TestDBBenchRunner(unittest.TestCase): |
||||||
|
def setUp(self): |
||||||
|
# Note: the db_bench binary should be present in the rocksdb/ directory |
||||||
|
self.pos_args = [ |
||||||
|
'./../../db_bench', |
||||||
|
'overwrite', |
||||||
|
'use_existing_db=true', |
||||||
|
'duration=20' |
||||||
|
] |
||||||
|
self.bench_runner = DBBenchRunner(self.pos_args) |
||||||
|
this_path = os.path.abspath(os.path.dirname(__file__)) |
||||||
|
options_path = os.path.join(this_path, 'input_files/OPTIONS-000005') |
||||||
|
self.db_options = DatabaseOptions(options_path) |
||||||
|
|
||||||
|
def test_experiment_output(self): |
||||||
|
update_dict = {'bloom_bits': {NO_COL_FAMILY: 2}} |
||||||
|
self.db_options.update_options(update_dict) |
||||||
|
db_path = '/dev/shm' |
||||||
|
data_sources, throughput = self.bench_runner.run_experiment( |
||||||
|
self.db_options, db_path |
||||||
|
) |
||||||
|
self.assertEqual( |
||||||
|
data_sources[DataSource.Type.DB_OPTIONS][0].type, |
||||||
|
DataSource.Type.DB_OPTIONS |
||||||
|
) |
||||||
|
self.assertEqual( |
||||||
|
data_sources[DataSource.Type.LOG][0].type, |
||||||
|
DataSource.Type.LOG |
||||||
|
) |
||||||
|
self.assertEqual(len(data_sources[DataSource.Type.TIME_SERIES]), 2) |
||||||
|
self.assertEqual( |
||||||
|
data_sources[DataSource.Type.TIME_SERIES][0].type, |
||||||
|
DataSource.Type.TIME_SERIES |
||||||
|
) |
||||||
|
self.assertEqual( |
||||||
|
data_sources[DataSource.Type.TIME_SERIES][1].type, |
||||||
|
DataSource.Type.TIME_SERIES |
||||||
|
) |
||||||
|
self.assertEqual( |
||||||
|
data_sources[DataSource.Type.TIME_SERIES][1].stats_freq_sec, 0 |
||||||
|
) |
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__': |
||||||
|
unittest.main() |
||||||
@ -0,0 +1,216 @@ |
|||||||
|
# Copyright (c) 2011-present, Facebook, Inc. All rights reserved. |
||||||
|
# This source code is licensed under both the GPLv2 (found in the |
||||||
|
# COPYING file in the root directory) and Apache 2.0 License |
||||||
|
# (found in the LICENSE.Apache file in the root directory). |
||||||
|
|
||||||
|
from advisor.db_log_parser import NO_COL_FAMILY |
||||||
|
from advisor.db_options_parser import DatabaseOptions |
||||||
|
from advisor.rule_parser import Condition, OptionCondition |
||||||
|
import os |
||||||
|
import unittest |
||||||
|
|
||||||
|
|
||||||
|
class TestDatabaseOptions(unittest.TestCase): |
||||||
|
def setUp(self): |
||||||
|
self.this_path = os.path.abspath(os.path.dirname(__file__)) |
||||||
|
self.og_options = os.path.join( |
||||||
|
self.this_path, 'input_files/OPTIONS-000005' |
||||||
|
) |
||||||
|
misc_options = [ |
||||||
|
'bloom_bits = 4', 'rate_limiter_bytes_per_sec = 1024000' |
||||||
|
] |
||||||
|
# create the options object |
||||||
|
self.db_options = DatabaseOptions(self.og_options, misc_options) |
||||||
|
# perform clean-up before running tests |
||||||
|
self.generated_options = os.path.join( |
||||||
|
self.this_path, '../temp/OPTIONS_testing.tmp' |
||||||
|
) |
||||||
|
if os.path.isfile(self.generated_options): |
||||||
|
os.remove(self.generated_options) |
||||||
|
|
||||||
|
def test_get_options_diff(self): |
||||||
|
old_opt = { |
||||||
|
'DBOptions.stats_dump_freq_sec': {NO_COL_FAMILY: '20'}, |
||||||
|
'CFOptions.write_buffer_size': { |
||||||
|
'default': '1024000', |
||||||
|
'col_fam_A': '128000', |
||||||
|
'col_fam_B': '128000000' |
||||||
|
}, |
||||||
|
'DBOptions.use_fsync': {NO_COL_FAMILY: 'true'}, |
||||||
|
'DBOptions.max_log_file_size': {NO_COL_FAMILY: '128000000'} |
||||||
|
} |
||||||
|
new_opt = { |
||||||
|
'bloom_bits': {NO_COL_FAMILY: '4'}, |
||||||
|
'CFOptions.write_buffer_size': { |
||||||
|
'default': '128000000', |
||||||
|
'col_fam_A': '128000', |
||||||
|
'col_fam_C': '128000000' |
||||||
|
}, |
||||||
|
'DBOptions.use_fsync': {NO_COL_FAMILY: 'true'}, |
||||||
|
'DBOptions.max_log_file_size': {NO_COL_FAMILY: '0'} |
||||||
|
} |
||||||
|
diff = DatabaseOptions.get_options_diff(old_opt, new_opt) |
||||||
|
|
||||||
|
expected_diff = { |
||||||
|
'DBOptions.stats_dump_freq_sec': {NO_COL_FAMILY: ('20', None)}, |
||||||
|
'bloom_bits': {NO_COL_FAMILY: (None, '4')}, |
||||||
|
'CFOptions.write_buffer_size': { |
||||||
|
'default': ('1024000', '128000000'), |
||||||
|
'col_fam_B': ('128000000', None), |
||||||
|
'col_fam_C': (None, '128000000') |
||||||
|
}, |
||||||
|
'DBOptions.max_log_file_size': {NO_COL_FAMILY: ('128000000', '0')} |
||||||
|
} |
||||||
|
self.assertDictEqual(diff, expected_diff) |
||||||
|
|
||||||
|
def test_is_misc_option(self): |
||||||
|
self.assertTrue(DatabaseOptions.is_misc_option('bloom_bits')) |
||||||
|
self.assertFalse( |
||||||
|
DatabaseOptions.is_misc_option('DBOptions.stats_dump_freq_sec') |
||||||
|
) |
||||||
|
|
||||||
|
def test_set_up(self): |
||||||
|
options = self.db_options.get_all_options() |
||||||
|
self.assertEqual(22, len(options.keys())) |
||||||
|
expected_misc_options = { |
||||||
|
'bloom_bits': '4', 'rate_limiter_bytes_per_sec': '1024000' |
||||||
|
} |
||||||
|
self.assertDictEqual( |
||||||
|
expected_misc_options, self.db_options.get_misc_options() |
||||||
|
) |
||||||
|
self.assertListEqual( |
||||||
|
['default', 'col_fam_A'], self.db_options.get_column_families() |
||||||
|
) |
||||||
|
|
||||||
|
def test_get_options(self): |
||||||
|
opt_to_get = [ |
||||||
|
'DBOptions.manual_wal_flush', 'DBOptions.db_write_buffer_size', |
||||||
|
'bloom_bits', 'CFOptions.compaction_filter_factory', |
||||||
|
'CFOptions.num_levels', 'rate_limiter_bytes_per_sec', |
||||||
|
'TableOptions.BlockBasedTable.block_align', 'random_option' |
||||||
|
] |
||||||
|
options = self.db_options.get_options(opt_to_get) |
||||||
|
expected_options = { |
||||||
|
'DBOptions.manual_wal_flush': {NO_COL_FAMILY: 'false'}, |
||||||
|
'DBOptions.db_write_buffer_size': {NO_COL_FAMILY: '0'}, |
||||||
|
'bloom_bits': {NO_COL_FAMILY: '4'}, |
||||||
|
'CFOptions.compaction_filter_factory': { |
||||||
|
'default': 'nullptr', 'col_fam_A': 'nullptr' |
||||||
|
}, |
||||||
|
'CFOptions.num_levels': {'default': '7', 'col_fam_A': '5'}, |
||||||
|
'rate_limiter_bytes_per_sec': {NO_COL_FAMILY: '1024000'}, |
||||||
|
'TableOptions.BlockBasedTable.block_align': { |
||||||
|
'default': 'false', 'col_fam_A': 'true' |
||||||
|
} |
||||||
|
} |
||||||
|
self.assertDictEqual(expected_options, options) |
||||||
|
|
||||||
|
def test_update_options(self): |
||||||
|
# add new, update old, set old |
||||||
|
# before updating |
||||||
|
expected_old_opts = { |
||||||
|
'DBOptions.db_log_dir': {NO_COL_FAMILY: None}, |
||||||
|
'DBOptions.manual_wal_flush': {NO_COL_FAMILY: 'false'}, |
||||||
|
'bloom_bits': {NO_COL_FAMILY: '4'}, |
||||||
|
'CFOptions.num_levels': {'default': '7', 'col_fam_A': '5'}, |
||||||
|
'TableOptions.BlockBasedTable.block_restart_interval': { |
||||||
|
'col_fam_A': '16' |
||||||
|
} |
||||||
|
} |
||||||
|
get_opts = list(expected_old_opts.keys()) |
||||||
|
options = self.db_options.get_options(get_opts) |
||||||
|
self.assertEqual(expected_old_opts, options) |
||||||
|
# after updating options |
||||||
|
update_opts = { |
||||||
|
'DBOptions.db_log_dir': {NO_COL_FAMILY: '/dev/shm'}, |
||||||
|
'DBOptions.manual_wal_flush': {NO_COL_FAMILY: 'true'}, |
||||||
|
'bloom_bits': {NO_COL_FAMILY: '2'}, |
||||||
|
'CFOptions.num_levels': {'col_fam_A': '7'}, |
||||||
|
'TableOptions.BlockBasedTable.block_restart_interval': { |
||||||
|
'default': '32' |
||||||
|
}, |
||||||
|
'random_misc_option': {NO_COL_FAMILY: 'something'} |
||||||
|
} |
||||||
|
self.db_options.update_options(update_opts) |
||||||
|
update_opts['CFOptions.num_levels']['default'] = '7' |
||||||
|
update_opts['TableOptions.BlockBasedTable.block_restart_interval'] = { |
||||||
|
'default': '32', 'col_fam_A': '16' |
||||||
|
} |
||||||
|
get_opts.append('random_misc_option') |
||||||
|
options = self.db_options.get_options(get_opts) |
||||||
|
self.assertDictEqual(update_opts, options) |
||||||
|
expected_misc_options = { |
||||||
|
'bloom_bits': '2', |
||||||
|
'rate_limiter_bytes_per_sec': '1024000', |
||||||
|
'random_misc_option': 'something' |
||||||
|
} |
||||||
|
self.assertDictEqual( |
||||||
|
expected_misc_options, self.db_options.get_misc_options() |
||||||
|
) |
||||||
|
|
||||||
|
def test_generate_options_config(self): |
||||||
|
# make sure file does not exist from before |
||||||
|
self.assertFalse(os.path.isfile(self.generated_options)) |
||||||
|
self.db_options.generate_options_config('testing') |
||||||
|
self.assertTrue(os.path.isfile(self.generated_options)) |
||||||
|
|
||||||
|
def test_check_and_trigger_conditions(self): |
||||||
|
# options only from CFOptions |
||||||
|
# setup the OptionCondition objects to check and trigger |
||||||
|
update_dict = { |
||||||
|
'CFOptions.level0_file_num_compaction_trigger': {'col_fam_A': '4'}, |
||||||
|
'CFOptions.max_bytes_for_level_base': {'col_fam_A': '10'} |
||||||
|
} |
||||||
|
self.db_options.update_options(update_dict) |
||||||
|
cond1 = Condition('opt-cond-1') |
||||||
|
cond1 = OptionCondition.create(cond1) |
||||||
|
cond1.set_parameter( |
||||||
|
'options', [ |
||||||
|
'CFOptions.level0_file_num_compaction_trigger', |
||||||
|
'TableOptions.BlockBasedTable.block_restart_interval', |
||||||
|
'CFOptions.max_bytes_for_level_base' |
||||||
|
] |
||||||
|
) |
||||||
|
cond1.set_parameter( |
||||||
|
'evaluate', |
||||||
|
'int(options[0])*int(options[1])-int(options[2])>=0' |
||||||
|
) |
||||||
|
# only DBOptions |
||||||
|
cond2 = Condition('opt-cond-2') |
||||||
|
cond2 = OptionCondition.create(cond2) |
||||||
|
cond2.set_parameter( |
||||||
|
'options', [ |
||||||
|
'DBOptions.db_write_buffer_size', |
||||||
|
'bloom_bits', |
||||||
|
'rate_limiter_bytes_per_sec' |
||||||
|
] |
||||||
|
) |
||||||
|
cond2.set_parameter( |
||||||
|
'evaluate', |
||||||
|
'(int(options[2]) * int(options[1]) * int(options[0]))==0' |
||||||
|
) |
||||||
|
# mix of CFOptions and DBOptions |
||||||
|
cond3 = Condition('opt-cond-3') |
||||||
|
cond3 = OptionCondition.create(cond3) |
||||||
|
cond3.set_parameter( |
||||||
|
'options', [ |
||||||
|
'DBOptions.db_write_buffer_size', # 0 |
||||||
|
'CFOptions.num_levels', # 5, 7 |
||||||
|
'bloom_bits' # 4 |
||||||
|
] |
||||||
|
) |
||||||
|
cond3.set_parameter( |
||||||
|
'evaluate', 'int(options[2])*int(options[0])+int(options[1])>6' |
||||||
|
) |
||||||
|
self.db_options.check_and_trigger_conditions([cond1, cond2, cond3]) |
||||||
|
|
||||||
|
cond1_trigger = {'col_fam_A': ['4', '16', '10']} |
||||||
|
self.assertDictEqual(cond1_trigger, cond1.get_trigger()) |
||||||
|
cond2_trigger = {NO_COL_FAMILY: ['0', '4', '1024000']} |
||||||
|
self.assertDictEqual(cond2_trigger, cond2.get_trigger()) |
||||||
|
cond3_trigger = {'default': ['0', '7', '4']} |
||||||
|
self.assertDictEqual(cond3_trigger, cond3.get_trigger()) |
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__': |
||||||
|
unittest.main() |
||||||
@ -0,0 +1,126 @@ |
|||||||
|
# Copyright (c) 2011-present, Facebook, Inc. All rights reserved. |
||||||
|
# This source code is licensed under both the GPLv2 (found in the |
||||||
|
# COPYING file in the root directory) and Apache 2.0 License |
||||||
|
# (found in the LICENSE.Apache file in the root directory). |
||||||
|
|
||||||
|
from advisor.db_stats_fetcher import LogStatsParser, DatabasePerfContext |
||||||
|
from advisor.db_timeseries_parser import NO_ENTITY |
||||||
|
from advisor.rule_parser import Condition, TimeSeriesCondition |
||||||
|
import os |
||||||
|
import time |
||||||
|
import unittest |
||||||
|
from unittest.mock import MagicMock |
||||||
|
|
||||||
|
|
||||||
|
class TestLogStatsParser(unittest.TestCase): |
||||||
|
def setUp(self): |
||||||
|
this_path = os.path.abspath(os.path.dirname(__file__)) |
||||||
|
stats_file = os.path.join( |
||||||
|
this_path, 'input_files/log_stats_parser_keys_ts' |
||||||
|
) |
||||||
|
# populate the keys_ts dictionary of LogStatsParser |
||||||
|
self.stats_dict = {NO_ENTITY: {}} |
||||||
|
with open(stats_file, 'r') as fp: |
||||||
|
for line in fp: |
||||||
|
stat_name = line.split(':')[0].strip() |
||||||
|
self.stats_dict[NO_ENTITY][stat_name] = {} |
||||||
|
token_list = line.split(':')[1].strip().split(',') |
||||||
|
for token in token_list: |
||||||
|
timestamp = int(token.split()[0]) |
||||||
|
value = float(token.split()[1]) |
||||||
|
self.stats_dict[NO_ENTITY][stat_name][timestamp] = value |
||||||
|
self.log_stats_parser = LogStatsParser('dummy_log_file', 20) |
||||||
|
self.log_stats_parser.keys_ts = self.stats_dict |
||||||
|
|
||||||
|
def test_check_and_trigger_conditions_bursty(self): |
||||||
|
# mock fetch_timeseries() because 'keys_ts' has been pre-populated |
||||||
|
self.log_stats_parser.fetch_timeseries = MagicMock() |
||||||
|
# condition: bursty |
||||||
|
cond1 = Condition('cond-1') |
||||||
|
cond1 = TimeSeriesCondition.create(cond1) |
||||||
|
cond1.set_parameter('keys', 'rocksdb.db.get.micros.p50') |
||||||
|
cond1.set_parameter('behavior', 'bursty') |
||||||
|
cond1.set_parameter('window_sec', 40) |
||||||
|
cond1.set_parameter('rate_threshold', 0) |
||||||
|
self.log_stats_parser.check_and_trigger_conditions([cond1]) |
||||||
|
expected_cond_trigger = { |
||||||
|
NO_ENTITY: {1530896440: 0.9767546362322214} |
||||||
|
} |
||||||
|
self.assertDictEqual(expected_cond_trigger, cond1.get_trigger()) |
||||||
|
# ensure that fetch_timeseries() was called once |
||||||
|
self.log_stats_parser.fetch_timeseries.assert_called_once() |
||||||
|
|
||||||
|
def test_check_and_trigger_conditions_eval_agg(self): |
||||||
|
# mock fetch_timeseries() because 'keys_ts' has been pre-populated |
||||||
|
self.log_stats_parser.fetch_timeseries = MagicMock() |
||||||
|
# condition: evaluate_expression |
||||||
|
cond1 = Condition('cond-1') |
||||||
|
cond1 = TimeSeriesCondition.create(cond1) |
||||||
|
cond1.set_parameter('keys', 'rocksdb.db.get.micros.p50') |
||||||
|
cond1.set_parameter('behavior', 'evaluate_expression') |
||||||
|
keys = [ |
||||||
|
'rocksdb.manifest.file.sync.micros.p99', |
||||||
|
'rocksdb.db.get.micros.p50' |
||||||
|
] |
||||||
|
cond1.set_parameter('keys', keys) |
||||||
|
cond1.set_parameter('aggregation_op', 'latest') |
||||||
|
# condition evaluates to FALSE |
||||||
|
cond1.set_parameter('evaluate', 'keys[0]-(keys[1]*100)>200') |
||||||
|
self.log_stats_parser.check_and_trigger_conditions([cond1]) |
||||||
|
expected_cond_trigger = {NO_ENTITY: [1792.0, 15.9638]} |
||||||
|
self.assertIsNone(cond1.get_trigger()) |
||||||
|
# condition evaluates to TRUE |
||||||
|
cond1.set_parameter('evaluate', 'keys[0]-(keys[1]*100)<200') |
||||||
|
self.log_stats_parser.check_and_trigger_conditions([cond1]) |
||||||
|
expected_cond_trigger = {NO_ENTITY: [1792.0, 15.9638]} |
||||||
|
self.assertDictEqual(expected_cond_trigger, cond1.get_trigger()) |
||||||
|
# ensure that fetch_timeseries() was called |
||||||
|
self.log_stats_parser.fetch_timeseries.assert_called() |
||||||
|
|
||||||
|
def test_check_and_trigger_conditions_eval(self): |
||||||
|
# mock fetch_timeseries() because 'keys_ts' has been pre-populated |
||||||
|
self.log_stats_parser.fetch_timeseries = MagicMock() |
||||||
|
# condition: evaluate_expression |
||||||
|
cond1 = Condition('cond-1') |
||||||
|
cond1 = TimeSeriesCondition.create(cond1) |
||||||
|
cond1.set_parameter('keys', 'rocksdb.db.get.micros.p50') |
||||||
|
cond1.set_parameter('behavior', 'evaluate_expression') |
||||||
|
keys = [ |
||||||
|
'rocksdb.manifest.file.sync.micros.p99', |
||||||
|
'rocksdb.db.get.micros.p50' |
||||||
|
] |
||||||
|
cond1.set_parameter('keys', keys) |
||||||
|
cond1.set_parameter('evaluate', 'keys[0]-(keys[1]*100)>500') |
||||||
|
self.log_stats_parser.check_and_trigger_conditions([cond1]) |
||||||
|
expected_trigger = {NO_ENTITY: { |
||||||
|
1530896414: [9938.0, 16.31508], |
||||||
|
1530896440: [9938.0, 16.346602], |
||||||
|
1530896466: [9938.0, 16.284669], |
||||||
|
1530896492: [9938.0, 16.16005] |
||||||
|
}} |
||||||
|
self.assertDictEqual(expected_trigger, cond1.get_trigger()) |
||||||
|
self.log_stats_parser.fetch_timeseries.assert_called_once() |
||||||
|
|
||||||
|
|
||||||
|
class TestDatabasePerfContext(unittest.TestCase): |
||||||
|
def test_unaccumulate_metrics(self): |
||||||
|
perf_dict = { |
||||||
|
"user_key_comparison_count": 675903942, |
||||||
|
"block_cache_hit_count": 830086, |
||||||
|
} |
||||||
|
timestamp = int(time.time()) |
||||||
|
perf_ts = {} |
||||||
|
for key in perf_dict: |
||||||
|
perf_ts[key] = {} |
||||||
|
start_val = perf_dict[key] |
||||||
|
for ix in range(5): |
||||||
|
perf_ts[key][timestamp+(ix*10)] = start_val + (2 * ix * ix) |
||||||
|
db_perf_context = DatabasePerfContext(perf_ts, 10, True) |
||||||
|
timestamps = [timestamp+(ix*10) for ix in range(1, 5, 1)] |
||||||
|
values = [val for val in range(2, 15, 4)] |
||||||
|
inner_dict = {timestamps[ix]: values[ix] for ix in range(4)} |
||||||
|
expected_keys_ts = {NO_ENTITY: { |
||||||
|
'user_key_comparison_count': inner_dict, |
||||||
|
'block_cache_hit_count': inner_dict |
||||||
|
}} |
||||||
|
self.assertDictEqual(expected_keys_ts, db_perf_context.keys_ts) |
||||||
Loading…
Reference in new issue