JMH microbenchmarks for RocksJava (#6241)

Summary: This is the start of some JMH microbenchmarks for RocksJava. Such benchmarks can help us decide on performance improvements of the Java API. At the moment, I have only added benchmarks for various Comparator options, as that is one of the first areas where I want to improve performance. I plan to expand this to many more tests. Details of how to compile and run the benchmarks are in the `README.md`. A run of these on a XEON 3.5 GHz 4vCPU (QEMU Virtual CPU version 2.5+) / 8GB RAM KVM with Ubuntu 18.04, OpenJDK 1.8.0_232, and gcc 8.3.0 produced the following: ``` # Run complete. Total time: 01:43:17 REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell. Benchmark (comparatorName) Mode Cnt Score Error Units ComparatorBenchmarks.put native_bytewise thrpt 25 122373.920 ± 2200.538 ops/s ComparatorBenchmarks.put java_bytewise_adaptive_mutex thrpt 25 17388.201 ± 1444.006 ops/s ComparatorBenchmarks.put java_bytewise_non-adaptive_mutex thrpt 25 16887.150 ± 1632.204 ops/s ComparatorBenchmarks.put java_direct_bytewise_adaptive_mutex thrpt 25 15644.572 ± 1791.189 ops/s ComparatorBenchmarks.put java_direct_bytewise_non-adaptive_mutex thrpt 25 14869.601 ± 2252.135 ops/s ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 116528.735 ± 4168.797 ops/s ComparatorBenchmarks.put java_reverse_bytewise_adaptive_mutex thrpt 25 10651.975 ± 545.998 ops/s ComparatorBenchmarks.put java_reverse_bytewise_non-adaptive_mutex thrpt 25 10514.224 ± 930.069 ops/s ``` Indicating a ~7x difference between comparators implemented natively (C++) and those implemented in Java. Let's see if we can't improve on that in the near future... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6241 Differential Revision: D19290410 Pulled By: pdillinger fbshipit-source-id: 25d44bf3a31de265502ed0c5d8a28cf4c7cb9c0b
5 years ago · 6477075f2c
parent 5709e97a74
commit 6477075f2c
9 changed files with 809 additions and 0 deletions
--- a/java/jmh/LICENSE-HEADER.txt
+++ b/java/jmh/LICENSE-HEADER.txt
@ -0,0 +1,5 @@
+Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ This source code is licensed under both the GPLv2 (found in the
+ COPYING file in the root directory) and Apache 2.0 License
+ (found in the LICENSE.Apache file in the root directory).
+
--- a/java/jmh/README.md
+++ b/java/jmh/README.md
@ -0,0 +1,18 @@
+# JMH Benchmarks for RocksJava
+
+These are micro-benchmarks for RocksJava functionality, using [JMH (Java Microbenchmark Harness)](https://openjdk.java.net/projects/code-tools/jmh/).
+
+## Compiling
+
+**Note**: This uses a specific build of RocksDB that is set in the `<version>` element of the `dependencies` section of the `pom.xml` file. If you are testing local changes you should build and install a SNAPSHOT version of rocksdbjni, and update the `pom.xml` of rocksdbjni-jmh file to test with this.
+
+```bash
+$ mvn package
+```
+
+## Running
+```bash
+$ java -jar target/rocksdbjni-jmh-1.0-SNAPSHOT-benchmarks.jar
+```
+
+NOTE: you can append `-help` to the command above to see all of the JMH runtime options.
--- a/java/jmh/pom.xml
+++ b/java/jmh/pom.xml
@ -0,0 +1,138 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.rocksdb</groupId>
+    <artifactId>rocksdbjni-jmh</artifactId>
+    <version>1.0-SNAPSHOT</version>
+
+    <url>http://rocksdb.org/</url>
+
+    <name>rocksdbjni-jmh</name>
+    <description>JMH Benchmarks for RocksDB Java API</description>
+
+    <organization>
+        <name>Facebook, Inc.</name>
+        <url>https://www.facebook.com</url>
+    </organization>
+
+    <licenses>
+        <license>
+            <name>Apache License 2.0</name>
+            <url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
+            <distribution>repo</distribution>
+        </license>
+        <license>
+            <name>GNU General Public License, version 2</name>
+            <url>http://www.gnu.org/licenses/gpl-2.0.html</url>
+            <distribution>repo</distribution>
+        </license>
+    </licenses>
+
+    <scm>
+        <connection>scm:git:git://github.com/facebook/rocksdb.git</connection>
+        <developerConnection>scm:git:git@github.com:facebook/rocksdb.git</developerConnection>
+        <url>http://github.com/facebook/rocksdb/</url>
+    </scm>
+
+    <properties>
+        <project.build.source>1.7</project.build.source>
+        <project.build.target>1.7</project.build.target>
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+
+        <jmh.version>1.22</jmh.version>
+        <uberjar.name>benchmarks</uberjar.name>
+    </properties>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.rocksdb</groupId>
+            <artifactId>rocksdbjni</artifactId>
+            <version>6.4.6</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.openjdk.jmh</groupId>
+            <artifactId>jmh-core</artifactId>
+            <version>${jmh.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.openjdk.jmh</groupId>
+            <artifactId>jmh-generator-annprocess</artifactId>
+            <version>${jmh.version}</version>
+            <scope>provided</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+                <version>3.8.1</version>
+                <configuration>
+                    <source>${project.build.source}</source>
+                    <target>${project.build.target}</target>
+                    <encoding>${project.build.sourceEncoding}</encoding>
+                </configuration>
+            </plugin>
+
+            <plugin>
+                <groupId>com.mycila</groupId>
+                <artifactId>license-maven-plugin</artifactId>
+                <version>3.0</version>
+                <inherited>true</inherited>
+                <configuration>
+                    <header>LICENSE-HEADER.txt</header>
+                    <failIfMissing>true</failIfMissing>
+                    <aggregate>true</aggregate>
+                    <strictCheck>true</strictCheck>
+                    <excludes>
+                        <exclude>pom.xml</exclude>
+                    </excludes>
+                    <encoding>${project.build.sourceEncoding}</encoding>
+                </configuration>
+            </plugin>
+
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-shade-plugin</artifactId>
+                <version>3.2.1</version>
+                <executions>
+                    <execution>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>shade</goal>
+                        </goals>
+                        <configuration>
+                            <finalName>${project.artifactId}-${project.version}-${uberjar.name}</finalName>
+                            <transformers>
+                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+                                    <mainClass>org.openjdk.jmh.Main</mainClass>
+                                </transformer>
+                            </transformers>
+                            <filters>
+                                <filter>
+                                    <!--
+                                        Shading signed JARs will fail without this.
+                                        http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
+                                    -->
+                                    <artifact>*:*</artifact>
+                                    <excludes>
+                                        <exclude>META-INF/*.SF</exclude>
+                                        <exclude>META-INF/*.DSA</exclude>
+                                        <exclude>META-INF/*.RSA</exclude>
+                                    </excludes>
+                                </filter>
+                            </filters>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+
+        </plugins>
+    </build>
+
+</project>
--- a/java/jmh/src/main/java/org/rocksdb/jmh/ComparatorBenchmarks.java
+++ b/java/jmh/src/main/java/org/rocksdb/jmh/ComparatorBenchmarks.java
@ -0,0 +1,122 @@
+/**
+ * Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ *  This source code is licensed under both the GPLv2 (found in the
+ *  COPYING file in the root directory) and Apache 2.0 License
+ *  (found in the LICENSE.Apache file in the root directory).
+ */
+package org.rocksdb.jmh;
+
+import org.openjdk.jmh.annotations.*;
+import org.rocksdb.*;
+import org.rocksdb.util.BytewiseComparator;
+import org.rocksdb.util.DirectBytewiseComparator;
+import org.rocksdb.util.FileUtils;
+import org.rocksdb.util.ReverseBytewiseComparator;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.rocksdb.util.KVUtils.ba;
+
+@State(Scope.Benchmark)
+public class ComparatorBenchmarks {
+
+  @Param({
+      "native_bytewise",
+      "native_reverse_bytewise",
+      "java_bytewise_adaptive_mutex",
+      "java_bytewise_non-adaptive_mutex",
+      "java_reverse_bytewise_adaptive_mutex",
+      "java_reverse_bytewise_non-adaptive_mutex",
+      "java_direct_bytewise_adaptive_mutex",
+      "java_direct_bytewise_non-adaptive_mutex"
+
+  })
+  public String comparatorName;
+
+  Path dbDir;
+  ComparatorOptions comparatorOptions;
+  AbstractComparator comparator;
+  Options options;
+  RocksDB db;
+
+  @Setup(Level.Trial)
+  public void setup() throws IOException, RocksDBException {
+    RocksDB.loadLibrary();
+
+    dbDir = Files.createTempDirectory("rocksjava-comparator-benchmarks");
+
+    options = new Options()
+            .setCreateIfMissing(true);
+    if (comparatorName == null || "native_bytewise".equals(comparatorName)) {
+      options.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
+    } else if ("native_reverse_bytewise".equals(comparatorName)) {
+      options.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
+    } else if ("java_bytewise_adaptive_mutex".equals(comparatorName)) {
+      comparatorOptions = new ComparatorOptions()
+              .setUseAdaptiveMutex(true);
+      comparator = new BytewiseComparator(comparatorOptions);
+      options.setComparator(comparator);
+    } else if ("java_bytewise_non-adaptive_mutex".equals(comparatorName)) {
+      comparatorOptions = new ComparatorOptions()
+              .setUseAdaptiveMutex(false);
+      comparator = new BytewiseComparator(comparatorOptions);
+      options.setComparator(comparator);
+    } else if ("java_reverse_bytewise_adaptive_mutex".equals(comparatorName)) {
+      comparatorOptions = new ComparatorOptions()
+              .setUseAdaptiveMutex(true);
+      comparator = new ReverseBytewiseComparator(comparatorOptions);
+      options.setComparator(comparator);
+    } else if ("java_reverse_bytewise_non-adaptive_mutex".equals(comparatorName)) {
+      comparatorOptions = new ComparatorOptions()
+              .setUseAdaptiveMutex(false);
+      comparator = new ReverseBytewiseComparator(comparatorOptions);
+      options.setComparator(comparator);
+    } else if ("java_direct_bytewise_adaptive_mutex".equals(comparatorName)) {
+      comparatorOptions = new ComparatorOptions()
+              .setUseAdaptiveMutex(true);
+      comparator = new DirectBytewiseComparator(comparatorOptions);
+      options.setComparator(comparator);
+    } else if ("java_direct_bytewise_non-adaptive_mutex".equals(comparatorName)) {
+      comparatorOptions = new ComparatorOptions()
+              .setUseAdaptiveMutex(false);
+      comparator = new DirectBytewiseComparator(comparatorOptions);
+      options.setComparator(comparator);
+    } else {
+      throw new IllegalArgumentException("Unknown comparator name: " + comparatorName);
+    }
+
+    db = RocksDB.open(options, dbDir.toAbsolutePath().toString());
+  }
+
+  @TearDown(Level.Trial)
+  public void cleanup() throws IOException {
+    db.close();
+    if (comparator != null) {
+      comparator.close();
+    }
+    if (comparatorOptions != null) {
+      comparatorOptions.close();
+    }
+    options.close();
+    FileUtils.delete(dbDir);
+  }
+
+  @State(Scope.Benchmark)
+  public static class Counter {
+    private final AtomicInteger count = new AtomicInteger();
+
+    public int next() {
+      return count.getAndIncrement();
+    }
+  }
+
+
+  @Benchmark
+  public void put(final Counter counter) throws RocksDBException {
+    final int i = counter.next();
+    db.put(ba("key" + i), ba("value" + i));
+  }
+}
--- a/java/jmh/src/main/java/org/rocksdb/jmh/GetBenchmarks.java
+++ b/java/jmh/src/main/java/org/rocksdb/jmh/GetBenchmarks.java
@ -0,0 +1,139 @@
+/**
+ * Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ *  This source code is licensed under both the GPLv2 (found in the
+ *  COPYING file in the root directory) and Apache 2.0 License
+ *  (found in the LICENSE.Apache file in the root directory).
+ */
+package org.rocksdb.jmh;
+
+import org.openjdk.jmh.annotations.*;
+import org.rocksdb.*;
+import org.rocksdb.util.FileUtils;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.rocksdb.util.KVUtils.ba;
+
+@State(Scope.Benchmark)
+public class GetBenchmarks {
+
+  @Param({
+      "no_column_family",
+      "1_column_family",
+      "20_column_families",
+      "100_column_families"
+  })
+  String columnFamilyTestType;
+
+  @Param("100000")
+  int keyCount;
+
+  Path dbDir;
+  DBOptions options;
+  int cfs = 0;  // number of column families
+  private AtomicInteger cfHandlesIdx;
+  ColumnFamilyHandle[] cfHandles;
+  RocksDB db;
+  private final AtomicInteger keyIndex = new AtomicInteger();
+
+  @Setup(Level.Trial)
+  public void setup() throws IOException, RocksDBException {
+    RocksDB.loadLibrary();
+
+    dbDir = Files.createTempDirectory("rocksjava-get-benchmarks");
+
+    options = new DBOptions()
+        .setCreateIfMissing(true)
+        .setCreateMissingColumnFamilies(true);
+
+    final List<ColumnFamilyDescriptor> cfDescriptors = new ArrayList<>();
+    cfDescriptors.add(new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY));
+
+    if ("1_column_family".equals(columnFamilyTestType)) {
+      cfs = 1;
+    } else if ("20_column_families".equals(columnFamilyTestType)) {
+      cfs = 20;
+    } else if ("100_column_families".equals(columnFamilyTestType)) {
+      cfs = 100;
+    }
+
+    if (cfs > 0) {
+      cfHandlesIdx = new AtomicInteger(1);
+      for (int i = 1; i <= cfs; i++) {
+        cfDescriptors.add(new ColumnFamilyDescriptor(ba("cf" + i)));
+      }
+    }
+
+    final List<ColumnFamilyHandle> cfHandlesList = new ArrayList<>(cfDescriptors.size());
+    db = RocksDB.open(options, dbDir.toAbsolutePath().toString(), cfDescriptors, cfHandlesList);
+    cfHandles = cfHandlesList.toArray(new ColumnFamilyHandle[0]);
+
+    // store initial data for retrieving via get
+    for (int i = 0; i < cfs; i++) {
+      for (int j = 0; j < keyCount; j++) {
+        db.put(cfHandles[i], ba("key" + j), ba("value" + j));
+      }
+    }
+
+    try (final FlushOptions flushOptions = new FlushOptions()
+            .setWaitForFlush(true)) {
+      db.flush(flushOptions);
+    }
+  }
+
+  @TearDown(Level.Trial)
+  public void cleanup() throws IOException {
+    for (final ColumnFamilyHandle cfHandle : cfHandles) {
+      cfHandle.close();
+    }
+    db.close();
+    options.close();
+    FileUtils.delete(dbDir);
+  }
+
+  private ColumnFamilyHandle getColumnFamily() {
+    if (cfs == 0) {
+      return cfHandles[0];
+    }  else if (cfs == 1) {
+      return cfHandles[1];
+    } else {
+      int idx = cfHandlesIdx.getAndIncrement();
+      if (idx > cfs) {
+        cfHandlesIdx.set(1); // doesn't ensure a perfect distribution, but it's ok
+        idx = 0;
+      }
+      return cfHandles[idx];
+    }
+  }
+
+  /**
+   * Takes the next position in the index.
+   */
+  private int next() {
+    int idx;
+    int nextIdx;
+    while (true) {
+      idx = keyIndex.get();
+      nextIdx = idx + 1;
+      if (nextIdx >= keyCount) {
+        nextIdx = 0;
+      }
+
+      if (keyIndex.compareAndSet(idx, nextIdx)) {
+        break;
+      }
+    }
+    return idx;
+  }
+
+  @Benchmark
+  public byte[] get() throws RocksDBException {
+    final int keyIdx = next();
+    return db.get(getColumnFamily(), ba("key" + keyIdx));
+  }
+}
--- a/java/jmh/src/main/java/org/rocksdb/jmh/MultiGetBenchmarks.java
+++ b/java/jmh/src/main/java/org/rocksdb/jmh/MultiGetBenchmarks.java
@ -0,0 +1,158 @@
+/**
+ * Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ *  This source code is licensed under both the GPLv2 (found in the
+ *  COPYING file in the root directory) and Apache 2.0 License
+ *  (found in the LICENSE.Apache file in the root directory).
+ */
+package org.rocksdb.jmh;
+
+import org.openjdk.jmh.annotations.*;
+import org.rocksdb.*;
+import org.rocksdb.util.FileUtils;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.rocksdb.util.KVUtils.ba;
+import static org.rocksdb.util.KVUtils.keys;
+
+@State(Scope.Benchmark)
+public class MultiGetBenchmarks {
+
+  @Param({
+      "no_column_family",
+      "1_column_family",
+      "20_column_families",
+      "100_column_families"
+  })
+  String columnFamilyTestType;
+
+  @Param("100000")
+  int keyCount;
+
+  @Param({
+          "10",
+          "100",
+          "1000",
+          "10000",
+  })
+  int multiGetSize;
+
+  Path dbDir;
+  DBOptions options;
+  int cfs = 0;  // number of column families
+  private AtomicInteger cfHandlesIdx;
+  ColumnFamilyHandle[] cfHandles;
+  RocksDB db;
+  private final AtomicInteger keyIndex = new AtomicInteger();
+
+  @Setup(Level.Trial)
+  public void setup() throws IOException, RocksDBException {
+    RocksDB.loadLibrary();
+
+    dbDir = Files.createTempDirectory("rocksjava-multiget-benchmarks");
+
+    options = new DBOptions()
+        .setCreateIfMissing(true)
+        .setCreateMissingColumnFamilies(true);
+
+    final List<ColumnFamilyDescriptor> cfDescriptors = new ArrayList<>();
+    cfDescriptors.add(new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY));
+
+    if ("1_column_family".equals(columnFamilyTestType)) {
+      cfs = 1;
+    } else if ("20_column_families".equals(columnFamilyTestType)) {
+      cfs = 20;
+    } else if ("100_column_families".equals(columnFamilyTestType)) {
+      cfs = 100;
+    }
+
+    if (cfs > 0) {
+      cfHandlesIdx = new AtomicInteger(1);
+      for (int i = 1; i <= cfs; i++) {
+        cfDescriptors.add(new ColumnFamilyDescriptor(ba("cf" + i)));
+      }
+    }
+
+    final List<ColumnFamilyHandle> cfHandlesList = new ArrayList<>(cfDescriptors.size());
+    db = RocksDB.open(options, dbDir.toAbsolutePath().toString(), cfDescriptors, cfHandlesList);
+    cfHandles = cfHandlesList.toArray(new ColumnFamilyHandle[0]);
+
+    // store initial data for retrieving via get
+    for (int i = 0; i < cfs; i++) {
+      for (int j = 0; j < keyCount; j++) {
+        db.put(cfHandles[i], ba("key" + j), ba("value" + j));
+      }
+    }
+
+    try (final FlushOptions flushOptions = new FlushOptions()
+            .setWaitForFlush(true)) {
+      db.flush(flushOptions);
+    }
+  }
+
+  @TearDown(Level.Trial)
+  public void cleanup() throws IOException {
+    for (final ColumnFamilyHandle cfHandle : cfHandles) {
+      cfHandle.close();
+    }
+    db.close();
+    options.close();
+    FileUtils.delete(dbDir);
+  }
+
+  private ColumnFamilyHandle getColumnFamily() {
+    if (cfs == 0) {
+      return cfHandles[0];
+    }  else if (cfs == 1) {
+      return cfHandles[1];
+    } else {
+      int idx = cfHandlesIdx.getAndIncrement();
+      if (idx > cfs) {
+        cfHandlesIdx.set(1); // doesn't ensure a perfect distribution, but it's ok
+        idx = 0;
+      }
+      return cfHandles[idx];
+    }
+  }
+
+  /**
+   * Reserves the next {@inc} positions in the index.
+   *
+   * @param inc the number by which to increment the index
+   * @param limit the limit for the index
+   * @return the index before {@code inc} is added
+   */
+  private int next(final int inc, final int limit) {
+    int idx;
+    int nextIdx;
+    while (true) {
+      idx = keyIndex.get();
+      nextIdx = idx + inc;
+      if (nextIdx >= limit) {
+          nextIdx = inc;
+      }
+
+      if (keyIndex.compareAndSet(idx, nextIdx)) {
+        break;
+      }
+    }
+
+    if (nextIdx >= limit) {
+      return -1;
+    } else {
+      return idx;
+    }
+  }
+
+  @Benchmark
+  public List<byte[]> multiGet10() throws RocksDBException {
+    final int fromKeyIdx = next(multiGetSize, keyCount);
+    final List<byte[]> keys = keys(fromKeyIdx, fromKeyIdx + multiGetSize);
+    return db.multiGetAsList(keys);
+  }
+}
--- a/java/jmh/src/main/java/org/rocksdb/jmh/PutBenchmarks.java
+++ b/java/jmh/src/main/java/org/rocksdb/jmh/PutBenchmarks.java
@ -0,0 +1,112 @@
+/**
+ * Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ *  This source code is licensed under both the GPLv2 (found in the
+ *  COPYING file in the root directory) and Apache 2.0 License
+ *  (found in the LICENSE.Apache file in the root directory).
+ */
+package org.rocksdb.jmh;
+
+import org.openjdk.jmh.annotations.*;
+import org.rocksdb.*;
+import org.rocksdb.util.FileUtils;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.rocksdb.util.KVUtils.ba;
+
+@State(Scope.Benchmark)
+public class PutBenchmarks {
+
+  @Param({
+      "no_column_family",
+      "1_column_family",
+      "20_column_families",
+      "100_column_families"
+  })
+  String columnFamilyTestType;
+
+  Path dbDir;
+  DBOptions options;
+  int cfs = 0;  // number of column families
+  private AtomicInteger cfHandlesIdx;
+  ColumnFamilyHandle[] cfHandles;
+  RocksDB db;
+
+  @Setup(Level.Trial)
+  public void setup() throws IOException, RocksDBException {
+    RocksDB.loadLibrary();
+
+    dbDir = Files.createTempDirectory("rocksjava-put-benchmarks");
+
+    options = new DBOptions()
+        .setCreateIfMissing(true)
+        .setCreateMissingColumnFamilies(true);
+
+    final List<ColumnFamilyDescriptor> cfDescriptors = new ArrayList<>();
+    cfDescriptors.add(new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY));
+
+    if ("1_column_family".equals(columnFamilyTestType)) {
+      cfs = 1;
+    } else if ("20_column_families".equals(columnFamilyTestType)) {
+      cfs = 20;
+    } else if ("100_column_families".equals(columnFamilyTestType)) {
+      cfs = 100;
+    }
+
+    if (cfs > 0) {
+      cfHandlesIdx = new AtomicInteger(1);
+      for (int i = 1; i <= cfs; i++) {
+        cfDescriptors.add(new ColumnFamilyDescriptor(ba("cf" + i)));
+      }
+    }
+
+    final List<ColumnFamilyHandle> cfHandlesList = new ArrayList<>(cfDescriptors.size());
+    db = RocksDB.open(options, dbDir.toAbsolutePath().toString(), cfDescriptors, cfHandlesList);
+    cfHandles = cfHandlesList.toArray(new ColumnFamilyHandle[0]);
+  }
+
+  @TearDown(Level.Trial)
+  public void cleanup() throws IOException {
+    for (final ColumnFamilyHandle cfHandle : cfHandles) {
+      cfHandle.close();
+    }
+    db.close();
+    options.close();
+    FileUtils.delete(dbDir);
+  }
+
+  private ColumnFamilyHandle getColumnFamily() {
+    if (cfs == 0) {
+      return cfHandles[0];
+    }  else if (cfs == 1) {
+      return cfHandles[1];
+    } else {
+      int idx = cfHandlesIdx.getAndIncrement();
+      if (idx > cfs) {
+        cfHandlesIdx.set(1); // doesn't ensure a perfect distribution, but it's ok
+        idx = 0;
+      }
+      return cfHandles[idx];
+    }
+  }
+
+  @State(Scope.Benchmark)
+  public static class Counter {
+    private final AtomicInteger count = new AtomicInteger();
+
+    public int next() {
+      return count.getAndIncrement();
+    }
+  }
+
+  @Benchmark
+  public void put(final ComparatorBenchmarks.Counter counter) throws RocksDBException {
+    final int i = counter.next();
+    db.put(getColumnFamily(), ba("key" + i), ba("value" + i));
+  }
+}
--- a/java/jmh/src/main/java/org/rocksdb/util/FileUtils.java
+++ b/java/jmh/src/main/java/org/rocksdb/util/FileUtils.java
@ -0,0 +1,59 @@
+/**
+ * Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ *  This source code is licensed under both the GPLv2 (found in the
+ *  COPYING file in the root directory) and Apache 2.0 License
+ *  (found in the LICENSE.Apache file in the root directory).
+ */
+package org.rocksdb.util;
+
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+public final class FileUtils {
+    private static final SimpleFileVisitor<Path> DELETE_DIR_VISITOR = new DeleteDirVisitor();
+
+    /**
+     * Deletes a path from the filesystem
+     *
+     * If the path is a directory its contents
+     * will be recursively deleted before it itself
+     * is deleted.
+     *
+     * Note that removal of a directory is not an atomic-operation
+     * and so if an error occurs during removal, some of the directories
+     * descendants may have already been removed
+     *
+     * @param path the path to delete.
+     *
+     * @throws IOException if an error occurs whilst removing a file or directory
+     */
+    public static void delete(final Path path) throws IOException {
+        if (!Files.isDirectory(path)) {
+            Files.deleteIfExists(path);
+        } else {
+            Files.walkFileTree(path, DELETE_DIR_VISITOR);
+        }
+    }
+
+    private static class DeleteDirVisitor extends SimpleFileVisitor<Path> {
+        @Override
+        public FileVisitResult visitFile(final Path file, final BasicFileAttributes attrs) throws IOException {
+            Files.deleteIfExists(file);
+            return FileVisitResult.CONTINUE;
+        }
+
+        @Override
+        public FileVisitResult postVisitDirectory(final Path dir, final IOException exc) throws IOException {
+            if (exc != null) {
+                throw exc;
+            }
+
+            Files.deleteIfExists(dir);
+            return FileVisitResult.CONTINUE;
+        }
+    }
+}
--- a/java/jmh/src/main/java/org/rocksdb/util/KVUtils.java
+++ b/java/jmh/src/main/java/org/rocksdb/util/KVUtils.java
@ -0,0 +1,58 @@
+/**
+ * Copyright (c) 2011-present, Facebook, Inc.  All rights reserved.
+ *  This source code is licensed under both the GPLv2 (found in the
+ *  COPYING file in the root directory) and Apache 2.0 License
+ *  (found in the LICENSE.Apache file in the root directory).
+ */
+package org.rocksdb.util;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public final class KVUtils {
+
+  /**
+   * Get a byte array from a string.
+   *
+   * Assumes UTF-8 encoding
+   *
+   * @param string the string
+   *
+   * @return the bytes.
+   */
+  public static byte[] ba(final String string) {
+    return string.getBytes(UTF_8);
+  }
+
+  /**
+   * Get a string from a byte array.
+   *
+   * Assumes UTF-8 encoding
+   *
+   * @param bytes the bytes
+   *
+   * @return the string.
+   */
+  public static String str(final byte[] bytes) {
+    return new String(bytes, UTF_8);
+  }
+
+  /**
+   * Get a list of keys where the keys are named key1..key1+N
+   * in the range of {@code from} to {@code to} i.e. keyFrom..keyTo.
+   *
+   * @param from the first key
+   * @param to the last key
+   *
+   * @return the array of keys
+   */
+  public static List<byte[]> keys(final int from, final int to) {
+    final List<byte[]> keys = new ArrayList<>(to - from);
+    for (int i = from; i < to; i++) {
+      keys.add(ba("key" + i));
+    }
+    return keys;
+  }
+}