From 8848ec92dd782d3448b874c8f57b17be7a7457bc Mon Sep 17 00:00:00 2001 From: Peter Dillinger Date: Tue, 30 May 2023 16:42:49 -0700 Subject: [PATCH] Better management of unreleased HISTORY (#11481) Summary: See new NOTE in HISTORY.md and unreleased_history/README.txt Pull Request resolved: https://github.com/facebook/rocksdb/pull/11481 Test Plan: some manual testing on my CentOS 8 system Reviewed By: jaykorean Differential Revision: D46233342 Pulled By: pdillinger fbshipit-source-id: daf59cf3dc907f450b469090dcc481a30a7d7c0d --- HISTORY.md | 12 +-- unreleased_history/README.txt | 73 +++++++++++++++ unreleased_history/add.sh | 26 ++++++ unreleased_history/behavior_changes/.gitkeep | 0 .../behavior_changes/sst_read_micros.md | 1 + unreleased_history/bug_fixes/.gitkeep | 0 unreleased_history/new_features/.gitkeep | 0 .../new_features/shared_lock_buckets.md | 1 + .../new_features/tiered_volatile_cache.md | 1 + .../new_features/wait_for_compact.md | 1 + .../new_features/writebatchrelease.md | 1 + .../public_api_changes/.gitkeep | 0 unreleased_history/release.sh | 92 +++++++++++++++++++ 13 files changed, 197 insertions(+), 11 deletions(-) create mode 100644 unreleased_history/README.txt create mode 100755 unreleased_history/add.sh create mode 100644 unreleased_history/behavior_changes/.gitkeep create mode 100644 unreleased_history/behavior_changes/sst_read_micros.md create mode 100644 unreleased_history/bug_fixes/.gitkeep create mode 100644 unreleased_history/new_features/.gitkeep create mode 100644 unreleased_history/new_features/shared_lock_buckets.md create mode 100644 unreleased_history/new_features/tiered_volatile_cache.md create mode 100644 unreleased_history/new_features/wait_for_compact.md create mode 100644 unreleased_history/new_features/writebatchrelease.md create mode 100644 unreleased_history/public_api_changes/.gitkeep create mode 100755 unreleased_history/release.sh diff --git a/HISTORY.md b/HISTORY.md index 4418f02b6..7d667d962 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,15 +1,5 @@ # Rocksdb Change Log -## Unreleased -### New Features -* Add a new option OptimisticTransactionDBOptions::shared_lock_buckets that enables sharing mutexes for validating transactions between DB instances, for better balancing memory efficiency and validation contention across DB instances. Different column families and DBs also now use different hash seeds in this validation, so that the same set of key names will not contend across DBs or column families. -* Add `WriteBatch::Release()` that releases the batch's serialized data to the caller. -* Add an API NewTieredVolatileCache() in include/rocksdb/cache.h to allocate an instance of a block cache with a primary block cache tier and a compressed secondary cache tier. A cache of this type distributes memory reservations against the block cache, such as WriteBufferManager, table reader memory etc., proportionally across both the primary and compressed secondary cache. - -### Public API Changes -* Add `WaitForCompact()` to wait for all flush and compactions jobs to finish. Jobs to wait include the unscheduled (queued, but not scheduled yet). - -### Behavior changes -* Statistics `rocksdb.sst.read.micros` scope is expanded to all SST reads except for file ingestion and column family import (some compaction reads were previously excluded). +> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` ## 8.3.0 (05/19/2023) ### New Features diff --git a/unreleased_history/README.txt b/unreleased_history/README.txt new file mode 100644 index 000000000..1d641285d --- /dev/null +++ b/unreleased_history/README.txt @@ -0,0 +1,73 @@ +Adding release notes +-------------------- + +When adding release notes for the next release, add a file to one of these +directories: + +unreleased_history/new_features +unreleased_history/behavior_changes +unreleased_history/public_api_changes +unreleased_history/bug_fixes + +with a unique name that makes sense for your change, preferably using the .md +extension for syntax highlighting. + +There is a script to help, as in + +$ unreleased_history/add.sh unreleased_history/bug_fixes/crash_in_feature.md + +or simply + +$ unreleased_history/add.sh + +will take you through some prompts. + +The file should usually contain one line of markdown, and "* " is not +required, as it will automatically be inserted later if not included at the +start of the first line in the file. Extra newlines or missing trailing +newlines will also be corrected. + +The only times release notes should be added directly to HISTORY are if +* A release is being amended or corrected after it is already "cut" but not +tagged, which should be rare. +* A single commit contains a noteworthy change and a patch release version bump + + +Ordering of entries +------------------- + +Within each group, entries will be included using ls sort order, so important +entries could start their file name with a small three digit number like +100pretty_important.md. + +The ordering of groups such as new_features vs. public_api_changes is +hard-coded in unreleased_history/release.sh + + +Updating HISTORY.md with release notes +-------------------------------------- + +The script unreleased_history/release.sh does this. Run the script before +updating version.h to the next develpment release, so that the script will pick +up the version being released. You might want to start with + +$ DRY_RUN=1 unreleased_history/release.sh | less + +to check for problems and preview the output. Then run + +$ unreleased_history/release.sh + +which will git rm some files and modify HISTORY.md. You still need to commit the +changes, or revert with the command reported in the output. + + +Why not update HISTORY.md directly? +----------------------------------- + +First, it was common to hit unnecessary merge conflicts when adding entries to +HISTORY.md, which slowed development. Second, when a PR was opened before a +release cut and landed after the release cut, it was easy to add the HISTORY +entry to the wrong version's history. This new setup completely fixes both of +those issues, with perhaps slighly more initial work to create each entry. +There is also now an extra step in using `git blame` to map a release note +to its source code implementation, but that is a relatively rare operation. diff --git a/unreleased_history/add.sh b/unreleased_history/add.sh new file mode 100755 index 000000000..13f55c972 --- /dev/null +++ b/unreleased_history/add.sh @@ -0,0 +1,26 @@ +#! /usr/bin/env bash + +set -e +set -o pipefail + +if [ "$1" ]; then + # Target file specified on command line + TARGET="$1" +else + # Interactively choose a group and file name + DIRS="`find unreleased_history/ -mindepth 1 -maxdepth 1 -type d`" + echo "Choose a group for new release note:" + echo "$DIRS" | grep -nEo '[^/]+$' + echo -n "Enter a number: " + while [ ! "$DIRNUM" ]; do read -r DIRNUM; done + DIR="$(echo "$DIRS" | head -n "$DIRNUM" | tail -1)" + echo "Choose a file name for new release note (e.g. improved_whatever.md):" + while [ ! "$FNAME" ]; do read -re FNAME; done + # Replace spaces with underscores + TARGET="$(echo "$DIR/$FNAME" | tr ' ' '_')" +fi + +# Edit/create the file +${EDITOR:-nano} "$TARGET" +# Add to version control (easy to forget!) +git add "$TARGET" diff --git a/unreleased_history/behavior_changes/.gitkeep b/unreleased_history/behavior_changes/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/unreleased_history/behavior_changes/sst_read_micros.md b/unreleased_history/behavior_changes/sst_read_micros.md new file mode 100644 index 000000000..99924f225 --- /dev/null +++ b/unreleased_history/behavior_changes/sst_read_micros.md @@ -0,0 +1 @@ +* Statistics `rocksdb.sst.read.micros` scope is expanded to all SST reads except for file ingestion and column family import (some compaction reads were previously excluded). diff --git a/unreleased_history/bug_fixes/.gitkeep b/unreleased_history/bug_fixes/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/unreleased_history/new_features/.gitkeep b/unreleased_history/new_features/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/unreleased_history/new_features/shared_lock_buckets.md b/unreleased_history/new_features/shared_lock_buckets.md new file mode 100644 index 000000000..5c6905333 --- /dev/null +++ b/unreleased_history/new_features/shared_lock_buckets.md @@ -0,0 +1 @@ +* Add a new option OptimisticTransactionDBOptions::shared_lock_buckets that enables sharing mutexes for validating transactions between DB instances, for better balancing memory efficiency and validation contention across DB instances. Different column families and DBs also now use different hash seeds in this validation, so that the same set of key names will not contend across DBs or column families. diff --git a/unreleased_history/new_features/tiered_volatile_cache.md b/unreleased_history/new_features/tiered_volatile_cache.md new file mode 100644 index 000000000..49bc1f90b --- /dev/null +++ b/unreleased_history/new_features/tiered_volatile_cache.md @@ -0,0 +1 @@ +Add an API NewTieredVolatileCache() in include/rocksdb/cache.h to allocate an instance of a block cache with a primary block cache tier and a compressed secondary cache tier. A cache of this type distributes memory reservations against the block cache, such as WriteBufferManager, table reader memory etc., proportionally across both the primary and compressed secondary cache. diff --git a/unreleased_history/new_features/wait_for_compact.md b/unreleased_history/new_features/wait_for_compact.md new file mode 100644 index 000000000..3fd7cf6f1 --- /dev/null +++ b/unreleased_history/new_features/wait_for_compact.md @@ -0,0 +1 @@ +Add `WaitForCompact()` to wait for all flush and compactions jobs to finish. Jobs to wait include the unscheduled (queued, but not scheduled yet). diff --git a/unreleased_history/new_features/writebatchrelease.md b/unreleased_history/new_features/writebatchrelease.md new file mode 100644 index 000000000..d0a3c015d --- /dev/null +++ b/unreleased_history/new_features/writebatchrelease.md @@ -0,0 +1 @@ +Add `WriteBatch::Release()` that releases the batch's serialized data to the caller. diff --git a/unreleased_history/public_api_changes/.gitkeep b/unreleased_history/public_api_changes/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/unreleased_history/release.sh b/unreleased_history/release.sh new file mode 100755 index 000000000..4dbcec609 --- /dev/null +++ b/unreleased_history/release.sh @@ -0,0 +1,92 @@ +#! /usr/bin/env bash + +set -e + +if [ ! -d unreleased_history ]; then + echo "Can't find unreleased_history/ directory" + exit 1 +fi + +GIT_PATHS="unreleased_history/ HISTORY.md" +if [ ! "$DRY_RUN" ]; then + # Check for uncommitted changes + UNCOMMITTED="$(git diff -- $GIT_PATHS)" + if [ "$UNCOMMITTED" ]; then + echo 'Uncommitted changes to files to be modified. Please commit first to' + echo 'ensure a clean revert path. You can always `git commit -a --amend`' + echo 'to add more changes to your commit.' + exit 2 + fi +fi + +# Add first part of existing HISTORY file to new version +awk '{ print } /NOTE/ { exit(0) }' < HISTORY.md > HISTORY.new + +# And a blank line separator +echo >> HISTORY.new + +# Add new version header +awk '/#define ROCKSDB_MAJOR/ { major = $3 } + /#define ROCKSDB_MINOR/ { minor = $3 } + /#define ROCKSDB_PATCH/ { patch = $3 } + END { print "## " major "." minor "." patch " (" strftime("%F") ")" }' < include/rocksdb/version.h >> HISTORY.new + +function process_file () { + # use awk to correct extra or missing newlines, missing '* ' on first line + awk '/./ { if (notfirstline || $1 == "*") print; + else print "* " $0; + notfirstline=1; }' < $1 >> HISTORY.new + echo git rm $1 + if [ ! "$DRY_RUN" ]; then + git rm $1 + fi +} + +PROCESSED_DIRECTORIES="" + +function process_dir () { + PROCESSED_DIRECTORIES="$PROCESSED_DIRECTORIES $1" + # ls will sort the files, including the permanent header file + FILES="$(ls unreleased_history/$1/)" + if [ "$FILES" ]; then + echo "### $2" >> HISTORY.new + for FILE in $FILES; do + process_file "unreleased_history/$1/$FILE" + done + echo >> HISTORY.new + echo "Saved entries from $1" + else + echo "Nothing new in $1" + fi +} + +# Process dirs and files +process_dir new_features "New Features" +process_dir public_api_changes "Public API Changes" +process_dir behavior_changes "Behavior Changes" +process_dir bug_fixes "Bug Fixes" + +# Check for unexpected files or dirs at top level. process_dir/process_file +# will deal with contents of these directories +EXPECTED_REGEX="[^/]*[.]sh|README[.]txt|$(echo $PROCESSED_DIRECTORIES | tr ' ' '|')" +UNEXPECTED="$(find unreleased_history/ -mindepth 1 -maxdepth 1 -regextype egrep -not -regex "[^/]*/($EXPECTED_REGEX)")" +if [ "$UNEXPECTED" ]; then + echo "Unexpected files I don't know how to process:" + echo "$UNEXPECTED" + rm HISTORY.new + exit 3 +fi + +# Add rest of existing HISTORY file to new version (collapsing newlines) +awk '/./ { if (note) pr=1 } + /NOTE/ { note=1 } + { if (pr) print }' < HISTORY.md >> HISTORY.new + +if [ "$DRY_RUN" ]; then + echo '===========================================' + diff -U3 HISTORY.md HISTORY.new || true + rm HISTORY.new +else + mv HISTORY.new HISTORY.md + echo "Done. Revert command: git checkout HEAD -- $GIT_PATHS" +fi