Use -Wno-invalid-offsetof instead of dangerous offset_of hack (#9563)

Summary:
After https://github.com/facebook/rocksdb/issues/9515 added a unique_ptr to Status, we see some
warnings-as-error in some internal builds like this:

```
stderr: rocksdb/src/db/compaction/compaction_job.cc:2839:7: error:
offset of on non-standard-layout type 'struct CompactionServiceResult'
[-Werror,-Winvalid-offsetof]
     {offsetof(struct CompactionServiceResult, status),
      ^                                        ~~~~~~
```

I see three potential solutions to resolving this:

* Expand our use of an idiom that works around the warning (see offset_of
functions removed in this change, inspired by
https://gist.github.com/graphitemaster/494f21190bb2c63c5516)  However,
this construction is invoking undefined behavior that assumes consistent
layout with no compiler-introduced indirection. A compiler incompatible
with our assumptions will likely compile the code and exhibit undefined
behavior.
* Migrate to something in place of offset, like a function mapping
CompactionServiceResult* to Status* (for the `status` field). This might
be required in the long term.
* **Selected:** Use our new C++17 dependency to use offsetof in a well-defined way
when the compiler allows it. From a comment on
https://gist.github.com/graphitemaster/494f21190bb2c63c5516:

> A final note: in C++17, offsetof is conditionally supported, which
> means that you can use it on any type (not just standard layout
> types) and the compiler will error if it can't compile it correctly.
> That appears to be the best option if you can live with C++17 and
> don't need constexpr support.

The C++17 semantics are confirmed on
https://en.cppreference.com/w/cpp/types/offsetof, so we can suppress the
warning as long as we accept that we might run into a compiler that
rejects the code, and at that point we will find a solution, such as
the more intrusive "migrate" solution above.

Although this is currently only showing in our buck build, it will
surely show up also with make and cmake, so I have updated those
configurations as well.

Also in the buck build, -Wno-expansion-to-defined does not appear to be
needed anymore (both current compiler configurations) so I
removed it.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9563

Test Plan: Tried out buck builds with both current compiler configurations

Reviewed By: riversand963

Differential Revision: D34220931

Pulled By: pdillinger

fbshipit-source-id: d39436008259bd1eaaa87c77be69fb2a5b559e1f
main
Peter Dillinger 3 years ago committed by Facebook GitHub Bot
parent 241b5aa15a
commit e24734f843
  1. 2
      CMakeLists.txt
  2. 5
      Makefile
  3. 6
      TARGETS
  4. 6
      buckifier/targets_cfg.py
  5. 63
      db/compaction/compaction_job.cc
  6. 85
      options/cf_options.cc
  7. 17
      options/customizable_test.cc
  8. 52
      options/options_settable_test.cc

@ -219,7 +219,7 @@ if(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /FC /d2Zi+ /W4 /wd4127 /wd4800 /wd4996 /wd4351 /wd4100 /wd4204 /wd4324") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /FC /d2Zi+ /W4 /wd4127 /wd4800 /wd4996 /wd4351 /wd4100 /wd4204 /wd4324")
else() else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -W -Wextra -Wall -pthread") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -W -Wextra -Wall -pthread")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wsign-compare -Wshadow -Wno-unused-parameter -Wno-unused-variable -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-strict-aliasing") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wsign-compare -Wshadow -Wno-unused-parameter -Wno-unused-variable -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-strict-aliasing -Wno-invalid-offsetof")
if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64") if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wstrict-prototypes") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wstrict-prototypes")
endif() endif()

@ -501,6 +501,11 @@ endif
CFLAGS += $(C_WARNING_FLAGS) $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CCFLAGS) $(OPT) CFLAGS += $(C_WARNING_FLAGS) $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CCFLAGS) $(OPT)
CXXFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CXXFLAGS) $(OPT) -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers CXXFLAGS += $(WARNING_FLAGS) -I. -I./include $(PLATFORM_CXXFLAGS) $(OPT) -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers
# Allow offsetof to work on non-standard layout types. Some compiler could
# completely reject our usage of offsetof, but we will solve that when it
# happens.
CXXFLAGS += -Wno-invalid-offsetof
LDFLAGS += $(PLATFORM_LDFLAGS) LDFLAGS += $(PLATFORM_LDFLAGS)
LIB_OBJECTS = $(patsubst %.cc, $(OBJ_DIR)/%.o, $(LIB_SOURCES)) LIB_OBJECTS = $(patsubst %.cc, $(OBJ_DIR)/%.o, $(LIB_SOURCES))

@ -14,8 +14,10 @@ REPO_PATH = package_name() + "/"
ROCKSDB_COMPILER_FLAGS_0 = [ ROCKSDB_COMPILER_FLAGS_0 = [
"-fno-builtin-memcmp", "-fno-builtin-memcmp",
# Needed to compile in fbcode # Allow offsetof to work on non-standard layout types. Some compiler could
"-Wno-expansion-to-defined", # completely reject our usage of offsetof, but we will solve that when it
# happens.
"-Wno-invalid-offsetof",
# Added missing flags from output of build_detect_platform # Added missing flags from output of build_detect_platform
"-Wnarrowing", "-Wnarrowing",
"-DROCKSDB_NO_DYNAMIC_EXTENSION", "-DROCKSDB_NO_DYNAMIC_EXTENSION",

@ -21,8 +21,10 @@ REPO_PATH = package_name() + "/"
ROCKSDB_COMPILER_FLAGS_0 = [ ROCKSDB_COMPILER_FLAGS_0 = [
"-fno-builtin-memcmp", "-fno-builtin-memcmp",
# Needed to compile in fbcode # Allow offsetof to work on non-standard layout types. Some compiler could
"-Wno-expansion-to-defined", # completely reject our usage of offsetof, but we will solve that when it
# happens.
"-Wno-invalid-offsetof",
# Added missing flags from output of build_detect_platform # Added missing flags from output of build_detect_platform
"-Wnarrowing", "-Wnarrowing",
"-DROCKSDB_NO_DYNAMIC_EXTENSION", "-DROCKSDB_NO_DYNAMIC_EXTENSION",

@ -2506,34 +2506,14 @@ enum BinaryFormatVersion : uint32_t {
kOptionsString = 1, // Use string format similar to Option string format kOptionsString = 1, // Use string format similar to Option string format
}; };
// offset_of is used to get the offset of a class data member
// ex: offset_of(&ColumnFamilyDescriptor::options)
// This call will return the offset of options in ColumnFamilyDescriptor class
//
// This is the same as offsetof() but allow us to work with non standard-layout
// classes and structures
// refs:
// http://en.cppreference.com/w/cpp/concept/StandardLayoutType
// https://gist.github.com/graphitemaster/494f21190bb2c63c5516
static ColumnFamilyDescriptor dummy_cfd("", ColumnFamilyOptions());
template <typename T1>
int offset_of(T1 ColumnFamilyDescriptor::*member) {
return int(size_t(&(dummy_cfd.*member)) - size_t(&dummy_cfd));
}
static CompactionServiceInput dummy_cs_input;
template <typename T1>
int offset_of(T1 CompactionServiceInput::*member) {
return int(size_t(&(dummy_cs_input.*member)) - size_t(&dummy_cs_input));
}
static std::unordered_map<std::string, OptionTypeInfo> cfd_type_info = { static std::unordered_map<std::string, OptionTypeInfo> cfd_type_info = {
{"name", {"name",
{offset_of(&ColumnFamilyDescriptor::name), OptionType::kEncodedString, {offsetof(struct ColumnFamilyDescriptor, name), OptionType::kEncodedString,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"options", {"options",
{offset_of(&ColumnFamilyDescriptor::options), OptionType::kConfigurable, {offsetof(struct ColumnFamilyDescriptor, options),
OptionVerificationType::kNormal, OptionTypeFlags::kNone, OptionType::kConfigurable, OptionVerificationType::kNormal,
OptionTypeFlags::kNone,
[](const ConfigOptions& opts, const std::string& /*name*/, [](const ConfigOptions& opts, const std::string& /*name*/,
const std::string& value, void* addr) { const std::string& value, void* addr) {
auto cf_options = static_cast<ColumnFamilyOptions*>(addr); auto cf_options = static_cast<ColumnFamilyOptions*>(addr);
@ -2567,13 +2547,14 @@ static std::unordered_map<std::string, OptionTypeInfo> cfd_type_info = {
static std::unordered_map<std::string, OptionTypeInfo> cs_input_type_info = { static std::unordered_map<std::string, OptionTypeInfo> cs_input_type_info = {
{"column_family", {"column_family",
OptionTypeInfo::Struct("column_family", &cfd_type_info, OptionTypeInfo::Struct(
offset_of(&CompactionServiceInput::column_family), "column_family", &cfd_type_info,
OptionVerificationType::kNormal, offsetof(struct CompactionServiceInput, column_family),
OptionTypeFlags::kNone)}, OptionVerificationType::kNormal, OptionTypeFlags::kNone)},
{"db_options", {"db_options",
{offset_of(&CompactionServiceInput::db_options), OptionType::kConfigurable, {offsetof(struct CompactionServiceInput, db_options),
OptionVerificationType::kNormal, OptionTypeFlags::kNone, OptionType::kConfigurable, OptionVerificationType::kNormal,
OptionTypeFlags::kNone,
[](const ConfigOptions& opts, const std::string& /*name*/, [](const ConfigOptions& opts, const std::string& /*name*/,
const std::string& value, void* addr) { const std::string& value, void* addr) {
auto options = static_cast<DBOptions*>(addr); auto options = static_cast<DBOptions*>(addr);
@ -2602,31 +2583,33 @@ static std::unordered_map<std::string, OptionTypeInfo> cs_input_type_info = {
return result; return result;
}}}, }}},
{"snapshots", OptionTypeInfo::Vector<uint64_t>( {"snapshots", OptionTypeInfo::Vector<uint64_t>(
offset_of(&CompactionServiceInput::snapshots), offsetof(struct CompactionServiceInput, snapshots),
OptionVerificationType::kNormal, OptionTypeFlags::kNone, OptionVerificationType::kNormal, OptionTypeFlags::kNone,
{0, OptionType::kUInt64T})}, {0, OptionType::kUInt64T})},
{"input_files", OptionTypeInfo::Vector<std::string>( {"input_files", OptionTypeInfo::Vector<std::string>(
offset_of(&CompactionServiceInput::input_files), offsetof(struct CompactionServiceInput, input_files),
OptionVerificationType::kNormal, OptionTypeFlags::kNone, OptionVerificationType::kNormal, OptionTypeFlags::kNone,
{0, OptionType::kEncodedString})}, {0, OptionType::kEncodedString})},
{"output_level", {"output_level",
{offset_of(&CompactionServiceInput::output_level), OptionType::kInt, {offsetof(struct CompactionServiceInput, output_level), OptionType::kInt,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"has_begin", {"has_begin",
{offset_of(&CompactionServiceInput::has_begin), OptionType::kBoolean, {offsetof(struct CompactionServiceInput, has_begin), OptionType::kBoolean,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"begin", {"begin",
{offset_of(&CompactionServiceInput::begin), OptionType::kEncodedString, {offsetof(struct CompactionServiceInput, begin),
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionType::kEncodedString, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
{"has_end", {"has_end",
{offset_of(&CompactionServiceInput::has_end), OptionType::kBoolean, {offsetof(struct CompactionServiceInput, has_end), OptionType::kBoolean,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"end", {"end",
{offset_of(&CompactionServiceInput::end), OptionType::kEncodedString, {offsetof(struct CompactionServiceInput, end), OptionType::kEncodedString,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"approx_size", {"approx_size",
{offset_of(&CompactionServiceInput::approx_size), OptionType::kUInt64T, {offsetof(struct CompactionServiceInput, approx_size),
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionType::kUInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
}; };
static std::unordered_map<std::string, OptionTypeInfo> static std::unordered_map<std::string, OptionTypeInfo>

@ -35,22 +35,8 @@
// OPTIONS files. // OPTIONS files.
namespace ROCKSDB_NAMESPACE { namespace ROCKSDB_NAMESPACE {
// offset_of is used to get the offset of a class data member
// ex: offset_of(&ColumnFamilyOptions::num_levels)
// This call will return the offset of num_levels in ColumnFamilyOptions class
//
// This is the same as offsetof() but allow us to work with non standard-layout
// classes and structures
// refs:
// http://en.cppreference.com/w/cpp/concept/StandardLayoutType
// https://gist.github.com/graphitemaster/494f21190bb2c63c5516
#ifndef ROCKSDB_LITE
static ImmutableCFOptions dummy_cf_options;
template <typename T1>
int offset_of(T1 ImmutableCFOptions::*member) {
return int(size_t(&(dummy_cf_options.*member)) - size_t(&dummy_cf_options));
}
#ifndef ROCKSDB_LITE
static Status ParseCompressionOptions(const std::string& value, static Status ParseCompressionOptions(const std::string& value,
const std::string& name, const std::string& name,
CompressionOptions& compression_opts) { CompressionOptions& compression_opts) {
@ -518,19 +504,20 @@ static std::unordered_map<std::string, OptionTypeInfo>
{0, OptionType::kBoolean, OptionVerificationType::kDeprecated, {0, OptionType::kBoolean, OptionVerificationType::kDeprecated,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"inplace_update_support", {"inplace_update_support",
{offset_of(&ImmutableCFOptions::inplace_update_support), {offsetof(struct ImmutableCFOptions, inplace_update_support),
OptionType::kBoolean, OptionVerificationType::kNormal, OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"level_compaction_dynamic_level_bytes", {"level_compaction_dynamic_level_bytes",
{offset_of(&ImmutableCFOptions::level_compaction_dynamic_level_bytes), {offsetof(struct ImmutableCFOptions,
level_compaction_dynamic_level_bytes),
OptionType::kBoolean, OptionVerificationType::kNormal, OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"optimize_filters_for_hits", {"optimize_filters_for_hits",
{offset_of(&ImmutableCFOptions::optimize_filters_for_hits), {offsetof(struct ImmutableCFOptions, optimize_filters_for_hits),
OptionType::kBoolean, OptionVerificationType::kNormal, OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"force_consistency_checks", {"force_consistency_checks",
{offset_of(&ImmutableCFOptions::force_consistency_checks), {offsetof(struct ImmutableCFOptions, force_consistency_checks),
OptionType::kBoolean, OptionVerificationType::kNormal, OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
// Need to keep this around to be able to read old OPTIONS files. // Need to keep this around to be able to read old OPTIONS files.
@ -538,34 +525,37 @@ static std::unordered_map<std::string, OptionTypeInfo>
{0, OptionType::kInt, OptionVerificationType::kDeprecated, {0, OptionType::kInt, OptionVerificationType::kDeprecated,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"max_write_buffer_number_to_maintain", {"max_write_buffer_number_to_maintain",
{offset_of(&ImmutableCFOptions::max_write_buffer_number_to_maintain), {offsetof(struct ImmutableCFOptions,
max_write_buffer_number_to_maintain),
OptionType::kInt, OptionVerificationType::kNormal, OptionType::kInt, OptionVerificationType::kNormal,
OptionTypeFlags::kNone, 0}}, OptionTypeFlags::kNone, 0}},
{"max_write_buffer_size_to_maintain", {"max_write_buffer_size_to_maintain",
{offset_of(&ImmutableCFOptions::max_write_buffer_size_to_maintain), {offsetof(struct ImmutableCFOptions,
max_write_buffer_size_to_maintain),
OptionType::kInt64T, OptionVerificationType::kNormal, OptionType::kInt64T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"min_write_buffer_number_to_merge", {"min_write_buffer_number_to_merge",
{offset_of(&ImmutableCFOptions::min_write_buffer_number_to_merge), {offsetof(struct ImmutableCFOptions, min_write_buffer_number_to_merge),
OptionType::kInt, OptionVerificationType::kNormal, OptionType::kInt, OptionVerificationType::kNormal,
OptionTypeFlags::kNone, 0}}, OptionTypeFlags::kNone, 0}},
{"num_levels", {"num_levels",
{offset_of(&ImmutableCFOptions::num_levels), OptionType::kInt, {offsetof(struct ImmutableCFOptions, num_levels), OptionType::kInt,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"bloom_locality", {"bloom_locality",
{offset_of(&ImmutableCFOptions::bloom_locality), OptionType::kUInt32T, {offsetof(struct ImmutableCFOptions, bloom_locality),
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionType::kUInt32T, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
{"rate_limit_delay_max_milliseconds", {"rate_limit_delay_max_milliseconds",
{0, OptionType::kUInt, OptionVerificationType::kDeprecated, {0, OptionType::kUInt, OptionVerificationType::kDeprecated,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"compression_per_level", {"compression_per_level",
OptionTypeInfo::Vector<CompressionType>( OptionTypeInfo::Vector<CompressionType>(
offset_of(&ImmutableCFOptions::compression_per_level), offsetof(struct ImmutableCFOptions, compression_per_level),
OptionVerificationType::kNormal, OptionTypeFlags::kNone, OptionVerificationType::kNormal, OptionTypeFlags::kNone,
{0, OptionType::kCompressionType})}, {0, OptionType::kCompressionType})},
{"comparator", {"comparator",
OptionTypeInfo::AsCustomRawPtr<const Comparator>( OptionTypeInfo::AsCustomRawPtr<const Comparator>(
offset_of(&ImmutableCFOptions::user_comparator), offsetof(struct ImmutableCFOptions, user_comparator),
OptionVerificationType::kByName, OptionTypeFlags::kCompareLoose, OptionVerificationType::kByName, OptionTypeFlags::kCompareLoose,
// Serializes a Comparator // Serializes a Comparator
[](const ConfigOptions& opts, const std::string&, const void* addr, [](const ConfigOptions& opts, const std::string&, const void* addr,
@ -592,11 +582,11 @@ static std::unordered_map<std::string, OptionTypeInfo>
/* Use the default match function*/ nullptr)}, /* Use the default match function*/ nullptr)},
{"memtable_insert_with_hint_prefix_extractor", {"memtable_insert_with_hint_prefix_extractor",
OptionTypeInfo::AsCustomSharedPtr<const SliceTransform>( OptionTypeInfo::AsCustomSharedPtr<const SliceTransform>(
offset_of(&ImmutableCFOptions:: offsetof(struct ImmutableCFOptions,
memtable_insert_with_hint_prefix_extractor), memtable_insert_with_hint_prefix_extractor),
OptionVerificationType::kByNameAllowNull, OptionTypeFlags::kNone)}, OptionVerificationType::kByNameAllowNull, OptionTypeFlags::kNone)},
{"memtable_factory", {"memtable_factory",
{offset_of(&ImmutableCFOptions::memtable_factory), {offsetof(struct ImmutableCFOptions, memtable_factory),
OptionType::kCustomizable, OptionVerificationType::kByName, OptionType::kCustomizable, OptionVerificationType::kByName,
OptionTypeFlags::kShared, OptionTypeFlags::kShared,
[](const ConfigOptions& opts, const std::string&, [](const ConfigOptions& opts, const std::string&,
@ -612,7 +602,7 @@ static std::unordered_map<std::string, OptionTypeInfo>
return s; return s;
}}}, }}},
{"memtable", {"memtable",
{offset_of(&ImmutableCFOptions::memtable_factory), {offsetof(struct ImmutableCFOptions, memtable_factory),
OptionType::kCustomizable, OptionVerificationType::kAlias, OptionType::kCustomizable, OptionVerificationType::kAlias,
OptionTypeFlags::kShared, OptionTypeFlags::kShared,
[](const ConfigOptions& opts, const std::string&, [](const ConfigOptions& opts, const std::string&,
@ -627,14 +617,15 @@ static std::unordered_map<std::string, OptionTypeInfo>
} }
return s; return s;
}}}, }}},
{"table_factory", OptionTypeInfo::AsCustomSharedPtr<TableFactory>( {"table_factory",
offset_of(&ImmutableCFOptions::table_factory), OptionTypeInfo::AsCustomSharedPtr<TableFactory>(
OptionVerificationType::kByName, offsetof(struct ImmutableCFOptions, table_factory),
(OptionTypeFlags::kCompareLoose | OptionVerificationType::kByName,
OptionTypeFlags::kStringNameOnly | (OptionTypeFlags::kCompareLoose |
OptionTypeFlags::kDontPrepare))}, OptionTypeFlags::kStringNameOnly |
OptionTypeFlags::kDontPrepare))},
{"block_based_table_factory", {"block_based_table_factory",
{offset_of(&ImmutableCFOptions::table_factory), {offsetof(struct ImmutableCFOptions, table_factory),
OptionType::kCustomizable, OptionVerificationType::kAlias, OptionType::kCustomizable, OptionVerificationType::kAlias,
OptionTypeFlags::kShared | OptionTypeFlags::kCompareLoose, OptionTypeFlags::kShared | OptionTypeFlags::kCompareLoose,
// Parses the input value and creates a BlockBasedTableFactory // Parses the input value and creates a BlockBasedTableFactory
@ -666,7 +657,7 @@ static std::unordered_map<std::string, OptionTypeInfo>
} }
}}}, }}},
{"plain_table_factory", {"plain_table_factory",
{offset_of(&ImmutableCFOptions::table_factory), {offsetof(struct ImmutableCFOptions, table_factory),
OptionType::kCustomizable, OptionVerificationType::kAlias, OptionType::kCustomizable, OptionVerificationType::kAlias,
OptionTypeFlags::kShared | OptionTypeFlags::kCompareLoose, OptionTypeFlags::kShared | OptionTypeFlags::kCompareLoose,
// Parses the input value and creates a PlainTableFactory // Parses the input value and creates a PlainTableFactory
@ -699,35 +690,35 @@ static std::unordered_map<std::string, OptionTypeInfo>
{"table_properties_collectors", {"table_properties_collectors",
OptionTypeInfo::Vector< OptionTypeInfo::Vector<
std::shared_ptr<TablePropertiesCollectorFactory>>( std::shared_ptr<TablePropertiesCollectorFactory>>(
offset_of( offsetof(struct ImmutableCFOptions,
&ImmutableCFOptions::table_properties_collector_factories), table_properties_collector_factories),
OptionVerificationType::kByName, OptionTypeFlags::kNone, OptionVerificationType::kByName, OptionTypeFlags::kNone,
OptionTypeInfo::AsCustomSharedPtr<TablePropertiesCollectorFactory>( OptionTypeInfo::AsCustomSharedPtr<TablePropertiesCollectorFactory>(
0, OptionVerificationType::kByName, OptionTypeFlags::kNone))}, 0, OptionVerificationType::kByName, OptionTypeFlags::kNone))},
{"compaction_filter", {"compaction_filter",
OptionTypeInfo::AsCustomRawPtr<const CompactionFilter>( OptionTypeInfo::AsCustomRawPtr<const CompactionFilter>(
offset_of(&ImmutableCFOptions::compaction_filter), offsetof(struct ImmutableCFOptions, compaction_filter),
OptionVerificationType::kByName, OptionTypeFlags::kAllowNull)}, OptionVerificationType::kByName, OptionTypeFlags::kAllowNull)},
{"compaction_filter_factory", {"compaction_filter_factory",
OptionTypeInfo::AsCustomSharedPtr<CompactionFilterFactory>( OptionTypeInfo::AsCustomSharedPtr<CompactionFilterFactory>(
offset_of(&ImmutableCFOptions::compaction_filter_factory), offsetof(struct ImmutableCFOptions, compaction_filter_factory),
OptionVerificationType::kByName, OptionTypeFlags::kAllowNull)}, OptionVerificationType::kByName, OptionTypeFlags::kAllowNull)},
{"merge_operator", {"merge_operator",
OptionTypeInfo::AsCustomSharedPtr<MergeOperator>( OptionTypeInfo::AsCustomSharedPtr<MergeOperator>(
offset_of(&ImmutableCFOptions::merge_operator), offsetof(struct ImmutableCFOptions, merge_operator),
OptionVerificationType::kByNameAllowFromNull, OptionVerificationType::kByNameAllowFromNull,
OptionTypeFlags::kCompareLoose | OptionTypeFlags::kAllowNull)}, OptionTypeFlags::kCompareLoose | OptionTypeFlags::kAllowNull)},
{"compaction_style", {"compaction_style",
{offset_of(&ImmutableCFOptions::compaction_style), {offsetof(struct ImmutableCFOptions, compaction_style),
OptionType::kCompactionStyle, OptionVerificationType::kNormal, OptionType::kCompactionStyle, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"compaction_pri", {"compaction_pri",
{offset_of(&ImmutableCFOptions::compaction_pri), {offsetof(struct ImmutableCFOptions, compaction_pri),
OptionType::kCompactionPri, OptionVerificationType::kNormal, OptionType::kCompactionPri, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}}, OptionTypeFlags::kNone}},
{"sst_partitioner_factory", {"sst_partitioner_factory",
OptionTypeInfo::AsCustomSharedPtr<SstPartitionerFactory>( OptionTypeInfo::AsCustomSharedPtr<SstPartitionerFactory>(
offset_of(&ImmutableCFOptions::sst_partitioner_factory), offsetof(struct ImmutableCFOptions, sst_partitioner_factory),
OptionVerificationType::kByName, OptionTypeFlags::kAllowNull)}, OptionVerificationType::kByName, OptionTypeFlags::kAllowNull)},
}; };

@ -206,31 +206,22 @@ struct SimpleOptions {
TestCustomizable* cp = nullptr; TestCustomizable* cp = nullptr;
}; };
static SimpleOptions dummy_simple_options;
template <typename T1>
int offset_of(T1 SimpleOptions::*member) {
return static_cast<int>(
reinterpret_cast<uintptr_t>(
std::addressof(dummy_simple_options.*member)) -
reinterpret_cast<uintptr_t>(std::addressof(dummy_simple_options)));
}
static std::unordered_map<std::string, OptionTypeInfo> simple_option_info = { static std::unordered_map<std::string, OptionTypeInfo> simple_option_info = {
#ifndef ROCKSDB_LITE #ifndef ROCKSDB_LITE
{"bool", {"bool",
{offset_of(&SimpleOptions::b), OptionType::kBoolean, {offsetof(struct SimpleOptions, b), OptionType::kBoolean,
OptionVerificationType::kNormal, OptionTypeFlags::kNone}}, OptionVerificationType::kNormal, OptionTypeFlags::kNone}},
{"unique", {"unique",
OptionTypeInfo::AsCustomUniquePtr<TestCustomizable>( OptionTypeInfo::AsCustomUniquePtr<TestCustomizable>(
offset_of(&SimpleOptions::cu), OptionVerificationType::kNormal, offsetof(struct SimpleOptions, cu), OptionVerificationType::kNormal,
OptionTypeFlags::kAllowNull)}, OptionTypeFlags::kAllowNull)},
{"shared", {"shared",
OptionTypeInfo::AsCustomSharedPtr<TestCustomizable>( OptionTypeInfo::AsCustomSharedPtr<TestCustomizable>(
offset_of(&SimpleOptions::cs), OptionVerificationType::kNormal, offsetof(struct SimpleOptions, cs), OptionVerificationType::kNormal,
OptionTypeFlags::kAllowNull)}, OptionTypeFlags::kAllowNull)},
{"pointer", {"pointer",
OptionTypeInfo::AsCustomRawPtr<TestCustomizable>( OptionTypeInfo::AsCustomRawPtr<TestCustomizable>(
offset_of(&SimpleOptions::cp), OptionVerificationType::kNormal, offsetof(struct SimpleOptions, cp), OptionVerificationType::kNormal,
OptionTypeFlags::kAllowNull)}, OptionTypeFlags::kAllowNull)},
#endif // ROCKSDB_LITE #endif // ROCKSDB_LITE
}; };

@ -367,12 +367,6 @@ TEST_F(OptionsSettableTest, DBOptionsAllFieldsSettable) {
delete[] new_options_ptr; delete[] new_options_ptr;
} }
template <typename T1, typename T2>
inline int offset_of(T1 T2::*member) {
static T2 obj;
return int(size_t(&(obj.*member)) - size_t(&obj));
}
// If the test fails, likely a new option is added to ColumnFamilyOptions // If the test fails, likely a new option is added to ColumnFamilyOptions
// but it cannot be set through GetColumnFamilyOptionsFromString(), or the // but it cannot be set through GetColumnFamilyOptionsFromString(), or the
// test is not updated accordingly. // test is not updated accordingly.
@ -386,36 +380,39 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
// options in the excluded set need to appear in the same order as in // options in the excluded set need to appear in the same order as in
// ColumnFamilyOptions. // ColumnFamilyOptions.
const OffsetGap kColumnFamilyOptionsExcluded = { const OffsetGap kColumnFamilyOptionsExcluded = {
{offset_of(&ColumnFamilyOptions::inplace_callback), {offsetof(struct ColumnFamilyOptions, inplace_callback),
sizeof(UpdateStatus(*)(char*, uint32_t*, Slice, std::string*))}, sizeof(UpdateStatus(*)(char*, uint32_t*, Slice, std::string*))},
{offset_of( {offsetof(struct ColumnFamilyOptions,
&ColumnFamilyOptions::memtable_insert_with_hint_prefix_extractor), memtable_insert_with_hint_prefix_extractor),
sizeof(std::shared_ptr<const SliceTransform>)}, sizeof(std::shared_ptr<const SliceTransform>)},
{offset_of(&ColumnFamilyOptions::compression_per_level), {offsetof(struct ColumnFamilyOptions, compression_per_level),
sizeof(std::vector<CompressionType>)}, sizeof(std::vector<CompressionType>)},
{offset_of( {offsetof(struct ColumnFamilyOptions,
&ColumnFamilyOptions::max_bytes_for_level_multiplier_additional), max_bytes_for_level_multiplier_additional),
sizeof(std::vector<int>)}, sizeof(std::vector<int>)},
{offset_of(&ColumnFamilyOptions::memtable_factory), {offsetof(struct ColumnFamilyOptions, memtable_factory),
sizeof(std::shared_ptr<MemTableRepFactory>)}, sizeof(std::shared_ptr<MemTableRepFactory>)},
{offset_of(&ColumnFamilyOptions::table_properties_collector_factories), {offsetof(struct ColumnFamilyOptions,
table_properties_collector_factories),
sizeof(ColumnFamilyOptions::TablePropertiesCollectorFactories)}, sizeof(ColumnFamilyOptions::TablePropertiesCollectorFactories)},
{offset_of(&ColumnFamilyOptions::comparator), sizeof(Comparator*)}, {offsetof(struct ColumnFamilyOptions, comparator), sizeof(Comparator*)},
{offset_of(&ColumnFamilyOptions::merge_operator), {offsetof(struct ColumnFamilyOptions, merge_operator),
sizeof(std::shared_ptr<MergeOperator>)}, sizeof(std::shared_ptr<MergeOperator>)},
{offset_of(&ColumnFamilyOptions::compaction_filter), {offsetof(struct ColumnFamilyOptions, compaction_filter),
sizeof(const CompactionFilter*)}, sizeof(const CompactionFilter*)},
{offset_of(&ColumnFamilyOptions::compaction_filter_factory), {offsetof(struct ColumnFamilyOptions, compaction_filter_factory),
sizeof(std::shared_ptr<CompactionFilterFactory>)}, sizeof(std::shared_ptr<CompactionFilterFactory>)},
{offset_of(&ColumnFamilyOptions::prefix_extractor), {offsetof(struct ColumnFamilyOptions, prefix_extractor),
sizeof(std::shared_ptr<const SliceTransform>)}, sizeof(std::shared_ptr<const SliceTransform>)},
{offset_of(&ColumnFamilyOptions::snap_refresh_nanos), sizeof(uint64_t)}, {offsetof(struct ColumnFamilyOptions, snap_refresh_nanos),
{offset_of(&ColumnFamilyOptions::table_factory), sizeof(uint64_t)},
{offsetof(struct ColumnFamilyOptions, table_factory),
sizeof(std::shared_ptr<TableFactory>)}, sizeof(std::shared_ptr<TableFactory>)},
{offset_of(&ColumnFamilyOptions::cf_paths), sizeof(std::vector<DbPath>)}, {offsetof(struct ColumnFamilyOptions, cf_paths),
{offset_of(&ColumnFamilyOptions::compaction_thread_limiter), sizeof(std::vector<DbPath>)},
{offsetof(struct ColumnFamilyOptions, compaction_thread_limiter),
sizeof(std::shared_ptr<ConcurrentTaskLimiter>)}, sizeof(std::shared_ptr<ConcurrentTaskLimiter>)},
{offset_of(&ColumnFamilyOptions::sst_partitioner_factory), {offsetof(struct ColumnFamilyOptions, sst_partitioner_factory),
sizeof(std::shared_ptr<SstPartitionerFactory>)}, sizeof(std::shared_ptr<SstPartitionerFactory>)},
}; };
@ -538,11 +535,12 @@ TEST_F(OptionsSettableTest, ColumnFamilyOptionsAllFieldsSettable) {
// Test copying to mutabable and immutable options and copy back the mutable // Test copying to mutabable and immutable options and copy back the mutable
// part. // part.
const OffsetGap kMutableCFOptionsExcluded = { const OffsetGap kMutableCFOptionsExcluded = {
{offset_of(&MutableCFOptions::prefix_extractor), {offsetof(struct MutableCFOptions, prefix_extractor),
sizeof(std::shared_ptr<const SliceTransform>)}, sizeof(std::shared_ptr<const SliceTransform>)},
{offset_of(&MutableCFOptions::max_bytes_for_level_multiplier_additional), {offsetof(struct MutableCFOptions,
max_bytes_for_level_multiplier_additional),
sizeof(std::vector<int>)}, sizeof(std::vector<int>)},
{offset_of(&MutableCFOptions::max_file_size), {offsetof(struct MutableCFOptions, max_file_size),
sizeof(std::vector<uint64_t>)}, sizeof(std::vector<uint64_t>)},
}; };

Loading…
Cancel
Save