node/benchmark

Node.js Core Benchmarks

This folder contains code and data used to measure performance of different Node.js implementations and different ways of writing JavaScript run by the built-in JavaScript engine.

For a detailed guide on how to write and run benchmarks in this directory, see the guide on benchmarks.

Table of Contents

• File tree structure
• node:bench evaluation tools
• Common API

File tree structure

Directories

Benchmarks testing the performance of a single node submodule are placed into a directory with the corresponding name, so that they can be executed by submodule or individually. Benchmarks that span multiple submodules may either be placed into the misc directory or into a directory named after the feature they benchmark. E.g. benchmarks for various new ECMAScript features and their pre-ES2015 counterparts are placed in a directory named es. Fixtures that are not specific to a certain benchmark but can be reused throughout the benchmark suite should be placed in the fixtures directory.

Other Top-level files

The top-level files include common dependencies of the benchmarks and the tools for launching benchmarks and visualizing their output. The actual benchmark scripts should be placed in their corresponding directories.

• _benchmark_progress.js: implements the progress bar displayed when running compare.js and scatter.js
• _cli.js: parses the command line arguments passed to compare.js, run.js and scatter.js
• _cli.R: parses the command line arguments passed to compare.R
• _http-benchmarkers.js: selects and runs external tools for benchmarking the http subsystem.
• bar.R: R script for visualizing the output of benchmarks with bar plots.
• common.js: see Common API.
• compare.js: command line tool for comparing performance between different Node.js binaries.
• compare-node-bench.js: parallel comparison tool for explicit node:bench files. It does not change or invoke compare.js.
• compare.R: R script for statistically analyzing the output of compare.js
• run.js: command line tool for running individual benchmark suite(s).
• scatter.js: command line tool for comparing the performance between different parameters in benchmark configurations, for example to analyze the time complexity. Pass --analyze to summarize the results without R.
• scatter-node-bench.js: parallel scatter-data tool for an explicit node:bench file. It does not change or invoke scatter.js.
• scatter.R: R script for visualizing the output of scatter.js with scatter plots.

node:bench evaluation tools

The compare-node-bench.js and scatter-node-bench.js tools run explicit node:bench files without changing the existing benchmark framework or its tools. Each repeated observation for a benchmark identity is collected by a separate process invocation with one measured sample. Benchmarks declared in the same file still execute serially in that process and can share JIT, garbage collector, heap, and cache state. This differs from legacy configuration-level process isolation and must be considered when comparing the frameworks.

Compare two binaries and analyze the compatible CSV using compare.R:

./node benchmark/compare-node-bench.js \
  --old ./node-main --new ./node-pr --runs 30 -- \
  benchmark/crypto/_create-hash.node-bench.js > compare-node-bench.csv
Rscript benchmark/compare.R < compare-node-bench.csv

Pass --analyze to run the same Welch analysis inline. --max-regression N implies --analyze and makes the command fail only when the Holm-Bonferroni adjusted one-sided p-value against the N% threshold is below 0.05 and the full 95% confidence interval is worse than -N%. Requiring both conditions prevents a noisy point estimate from failing a regression gate.

./node benchmark/compare-node-bench.js \
  --old ./node-main --new ./node-pr --runs 30 \
  --max-regression 5 -- benchmark/crypto/_create-hash.node-bench.js

Collect parameter data for the parallel buffer benchmark and plot it using scatter.R:

./node benchmark/scatter-node-bench.js --node ./node --runs 30 -- \
  benchmark/buffers/_buffer-compare-offset.node-bench.js \
  > scatter-node-bench.csv
Rscript benchmark/scatter.R --xaxis size --category method \
  --plot scatter-node-bench.png < scatter-node-bench.csv

Pass --analyze with an x-axis parameter to summarize the samples without R. The output includes mean and median confidence intervals, skew warnings, an optional bar chart, and Mann-Whitney U and Cliff's delta comparisons between consecutive x-axis values. Use --category for a second grouping parameter and --no-chart to omit the chart.

Because configurations in one file share a process, inline analysis averages aggregated configurations into one value per outer process. Consecutive x-axis comparisons use alternating, disjoint process sets so the unpaired Mann-Whitney test does not treat correlated values as independent samples.

./node benchmark/scatter-node-bench.js --runs 30 --analyze \
  --xaxis size --category method -- \
  benchmark/buffers/_buffer-compare-offset.node-bench.js

A file passed to scatter-node-bench.js must use one logical benchmark name. Parameter values distinguish its configurations. The tool rejects unstable identities and names or parameters that would merge unrelated CSV groups.

The underscore-prefixed benchmark files are parallel ports used to compare the measurement frameworks. Legacy discovery ignores them, so the original files remain the source benchmarks for run.js, compare.js, and scatter.js. The ports use the platform-specific original relative filename as their benchmark name and preserve parameter column names to keep CSV grouping compatible. For a direct framework comparison, collect the same number of runs from an original benchmark with scatter.js and from its port with scatter-node-bench.js, then compare their rate distributions.

Common API

The common.js module is used by benchmarks for consistency across repeated tasks. It has a number of helpful functions and properties to help with writing benchmarks.

createBenchmark(fn, configs[, options])

See the guide on writing benchmarks.

default_http_benchmarker

The default benchmarker used to run HTTP benchmarks. See the guide on writing HTTP benchmarks.

PORT

The default port used to run HTTP benchmarks. See the guide on writing HTTP benchmarks.

sendResult(data)

Used in special benchmarks that can't use createBenchmark and the object it returns to accomplish what they need. This function reports timing data to the parent process (usually created by running compare.js, run.js or scatter.js).