Clarify max_write_buffer_size_to_maintain (#9154)

Summary:
I was unable to figure out the behavior by reading the old doc so attempted to
write it differently.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9154

Reviewed By: riversand963

Differential Revision: D32338843

Pulled By: ajkr

fbshipit-source-id: e1e67720cd92572b195583e5ea2c592180d4fefd

include/rocksdb/advanced_options.h

@@ -253,17 +253,32 @@ struct AdvancedColumnFamilyOptions {
   // ignored.
   int max_write_buffer_number_to_maintain = 0;
 
-  // The total maximum size(bytes) of write buffers to maintain in memory
+  // The target number of write history bytes to hold in memory. Write history
-  // including copies of buffers that have already been flushed. This parameter
+  // comprises the latest write buffers (memtables). To reach the target, write
-  // only affects trimming of flushed buffers and does not affect flushing.
+  // buffers that were most recently flushed to SST files may be retained in
-  // This controls the maximum amount of write history that will be available
+  // memory.
-  // in memory for conflict checking when Transactions are used. The actual
+  //
-  // size of write history (flushed Memtables) might be higher than this limit
+  // This controls the target amount of write history that will be available
-  // if further trimming will reduce write history total size below this
+  // in memory for conflict checking when Transactions are used.
-  // limit. For example, if max_write_buffer_size_to_maintain is set to 64MB,
+  //
-  // and there are three flushed Memtables, with sizes of 32MB, 20MB, 20MB.
+  // This target may be undershot when the CF first opens and has not recovered
-  // Because trimming the next Memtable of size 20MB will reduce total memory
+  // or received enough writes to reach the target. After reaching the target
-  // usage to 52MB which is below the limit, RocksDB will stop trimming.
+  // once, it is guaranteed to never undershoot again. That guarantee is
+  // implemented by retaining flushed write buffers in-memory until the oldest
+  // one can be trimmed without dropping below the target.
+  //
+  // Examples with `max_write_buffer_size_to_maintain` set to 32MB:
+  //
+  // - One mutable memtable of 64MB, one unflushed immutable memtable of 64MB,
+  //   and zero flushed immutable memtables. Nothing trimmable exists.
+  // - One mutable memtable of 16MB, zero unflushed immutable memtables, and
+  //   one flushed immutable memtable of 64MB. Trimming is disallowed because
+  //   dropping the earliest (only) flushed immutable memtable would result in
+  //   write history of 16MB < 32MB.
+  // - One mutable memtable of 24MB, one unflushed immutable memtable of 16MB,
+  //   and one flushed immutable memtable of 16MB. The earliest (only) flushed
+  //   immutable memtable is trimmed because without it we still have
+  //   16MB + 24MB = 40MB > 32MB of write history.
   //
   // When using an OptimisticTransactionDB:
   // If this value is too low, some transactions may fail at commit time due