The reserved root `${os.devNull}/vfs`, which holds the mount points of
all virtual file systems, could not be read: fs calls on it fell through
to the real file system, so nothing could list what was mounted.
While any file system is mounted, serve the root as a read-only
directory. It lists every mount point by the last segment of its path, a
recursive listing descends into each mounted file system, and paths
under it that no mount serves report ENOENT. Creating, removing or
changing entries in it fails with EROFS. When nothing is mounted it does
not exist, as before.
Add vfs.vfsBase(), which returns the path of that directory, so that a
program can read it without spelling the path out.
A mount point cannot be removed or renamed, nor replaced by a rename:
rmdir() and rename() fail with EBUSY, and a recursive rm() empties the
file system and then fails the same way. Before, rmdir() of an empty
mount point reported success without doing anything.
The callback and promise forms of readdir() with `withFileTypes` now
report each Dirent's parentPath as a host path, as readdirSync() did,
instead of the provider-relative one, and split recursive names such as
`dir/file.txt` into their directory and base name. A recursive listing
joins subdirectories with the host separator rather than `/`, which
mixed separators on Windows. realpath() of a mount point no longer
returns it with a trailing separator.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66140
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Getting the values recorded during an interval requires calling
`reset()`, which removes them for every other user of the histogram,
or copying and subtracting histograms, which silently produces a wrong
result when the source was reset in between.
Add `histogram.diff(other)`, which returns a new read-only `Histogram`
containing the values recorded after `other`, an earlier snapshot of
the histogram, was taken. Neither histogram is changed. Unlike
`subtract()`, it verifies that both histograms have the same layout,
and it throws instead of clamping when `other` contains values that
the histogram does not.
Add `histogram.resetCount`, the number of calls to `reset()` and
`subtract()`, which `snapshot()` copies. `diff()` throws when the
counts differ, so a reset between two snapshots is detected even when
every bucket has since grown past its previous count.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: OpenCode
PR-URL: https://github.com/nodejs/node/pull/66099
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Cloning a histogram with `structuredClone()` or `postMessage()` shares
the native histogram instead of copying it. Capturing the state of a
histogram at a point in time, while other code keeps recording into
it, requires serializing it with `export()` and parsing the result
with `importHistogram()`.
Add `histogram.snapshot()`, which returns a new, independent
`Histogram` containing a copy of the histogram's configuration,
recorded values, `exceeds` count, and EWMA state, without the
serialization round trip. Values cannot be recorded into the returned
histogram. The method is available on all histograms, including
`RecordableHistogram` and `ELDHistogram` instances.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: OpenCode
PR-URL: https://github.com/nodejs/node/pull/66099
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Histograms exported by Node.js v26.9.0 use format version 1, which
rejects unknown keys on import. Adding fields to version 1 data would
therefore break importing it in v26.9.0, even though that release
claims to support version 1.
Introduce format version 2. It has the same layout as version 1, but
unknown keys are ignored on import, so fields can be added to it later
without changing the version again, while older releases reject it
based on the version rather than on the new fields.
`histogram.export()` now produces version 2 data. `importHistogram()`
accepts versions 1 and 2, and imports version 1 data, as well as data
without a version, with the original semantics.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66098
Reviewed-By: Xuguang Mei <meixuguang@gmail.com>
--vfs-mount mounted a source without running it, and shared one ordered
list of sources with --vfs-load, so neither option could say which entry
it had contributed: the entry point was recovered from the position of
--vfs-load among the mounts, in a list NODE_OPTIONS could prepend to.
Nothing needs more than one mount from the command line: a program that
wants more can mount them itself through node:vfs, where it also gets
the instance.
Remove --vfs-mount, leaving --vfs-load with the single source it mounts
and runs, and reserve layer 0 for that source, numbering the file
systems a program mounts itself from 1. The source is then at the same
mount point in every thread, whatever else that thread mounts -
including a thread where a --require preload mounted a file system of
its own first - so a path into it stays valid in a worker.
A worker still does not run that entry point: it inherits the source but
not the decision to load from it. A worker created with its own execArgv
inherits neither, so the documentation now says that such a worker must
be given --experimental-vfs and --vfs-load again to run a script from
the mount, and that --experimental-vfs is also what makes node:vfs
available to the worker's own code.
ERR_VFS_INVALID_TARGET now names --vfs-load as the source's origin, and
the startup test moves to test-vfs-load.js, with the cases that covered
mounting without loading removed and cases for the reserved mount point
added.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66162
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
ZipProvider threw ERR_METHOD_NOT_IMPLEMENTED for readdir() with
`{ recursive: true }`, although an archive already stores every member
under its full path, so a recursive listing is the same scan of the
entry names as a flat one, keeping the whole path below the directory
instead of only its first segment.
List every member below the directory, and every directory their paths
pass through, whether the archive holds an entry for it or only implies
it. Names are relative to the listed directory and joined with `/`, as
MemoryProvider returns them, so VirtualFileSystem reports the right
parentPath for each Dirent. A name that is both a member and a
directory is listed once, as a directory, as a flat listing already
did.
The entries are now collected in a map rather than looked up in an
array, so listing a directory no longer takes time quadratic in the
number of its children.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66127
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Expose SQLite's virtual table API through a new
`database.createModule(name, options)` method, wrapping
`sqlite3_create_module_v2()`. This enables read-only virtual tables
backed by JavaScript data sources, usable either as an eponymous table
(`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING
module_name`. Hidden columns pass parameters using table-valued
function syntax (`SELECT * FROM module_name(param1, param2)`).
`options` accepts `columns`, `rows`, `directOnly`, and
`useBigIntArguments`. Column types are validated against INTEGER,
TEXT, REAL, BLOB, and ANY, and column names are quoted when building
the `sqlite3_declare_vtab()` schema.
Rebased from https://github.com/nodejs/node/pull/61544, which was
opened by byteforge38 and became inactive. Changes on top of that
work:
- xColumn reports the value each hidden column was constrained to,
rather than NULL. SQLite treats xBestIndex's `omit` as a hint, so it
may recheck a constraint it already handed to xFilter; against NULL
that recheck rejected every row, and `gs(1, 3) WHERE start = 1`
returned no rows.
- xBestIndex lowers estimatedCost as it consumes constraints. With a
constant cost the planner was free to pick the unconstrained plan and
recheck afterwards, so a correlated parameter such as
`FROM t, gs(t.a, t.a + 1)` also returned no rows.
- Violations of the iteration protocol report a SQLite error instead of
calling PropagateJSError with no JavaScript exception pending. That
left `.all()` returning undefined and `exec()` reporting success.
- xBestIndex passes the constrained hidden-column indices to xFilter
through idxStr rather than an int bitmask, which previously aliased
for parameter indices at or above the width of an int.
- xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it
close() from inside rows(), an iterator's next(), or a row getter
finalized the statement that SQLite was still stepping, crashing the
process.
- xClose calls the iterator's return() method so generator `finally`
blocks run when SQLite stops stepping early, as it does for LIMIT or
a `break` out of a for...of loop. It is skipped while tearing down
from ~StatementSync or ~DatabaseSync, which run from garbage
collection callbacks where JavaScript cannot be executed; an
abandoned generator does not run `finally` in JavaScript either. It
is also skipped when an error is already pending, so that error still
reaches the caller.
- VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match
UserDefinedFunction instead of a raw pointer.
- createModule() rejects being called from an authorizer callback.
- Documents that values yielded by rows() follow the usual conversion
rules, so a number is stored as REAL and a BigInt as INTEGER even
when a column declares INTEGER, since virtual tables do not apply
column affinity to the values they return.
Refs: https://github.com/nodejs/node/pull/61544
Refs: https://github.com/nodejs/node/issues/63826
Fixes: https://github.com/nodejs/node/issues/61539
Co-authored-by: byteforge38 <stormcraft318@gmail.com>
Signed-off-by: Trevor Burnham <trevorburnham@gmail.com>
Assisted-by: Claude Opus 5
PR-URL: https://github.com/nodejs/node/pull/65787
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
`performance.nodeTiming.uvMetricsInfo` returns the libuv event loop
metrics as numbers, which are only exact up to
`Number.MAX_SAFE_INTEGER`. Add `uvMetricsInfoBigInt`, which returns
the same metrics as bigints backed by `uint64_t` storage, carrying the
full 64-bit range reported by libuv.
A single native call fills both a `Float64Array` and a
`BigUint64Array`, so `uvMetricsInfo` does not pay for bigint
allocation and conversion. The new property is omitted from
`toJSON()`, as `JSON.stringify()` cannot serialize bigints.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66094
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
When "vfsArchive" names a ZIP archive in the SEA configuration (with
"useVfs": true), --build-sea embeds the archive verbatim as one
reserved asset, and at runtime the SEA virtual file system mounts the
existing ZipProvider over a zero-copy view of the embedded archive
instead of the SEAProvider. The main script is injected into the
in-memory archive index as a stored entry, so the mounted tree looks
the same as with "assets".
The archive can be produced with any ZIP tool or with the ZIP support
in node:zlib; no ZIP serialization logic is added to the build side.
This trades asset read speed (entries are inflated when opened) for a
substantially smaller executable when the assets are compressible.
Also make FindSingleExecutableResource() return the deserialized
SeaResource by reference instead of by value: the copy included the
whole assets map, and getAsset() calls it once per asset read, so
every fs operation served by the SEA virtual file system paid a cost
linear in the number of bundled assets - with 8192 assets, reading
each of them took seconds instead of milliseconds.
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/65810
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
`node:ffi` dispatches native calls through one of three call paths: the
V8 Fast API with a generated trampoline, a per-function shared buffer,
or the generic libffi invoker. The public documentation only describes
the Fast API path and a "generic" fallback, never mentions the shared
buffer path, and leaves the selection order implicit.
Add a "Call paths" section to doc/api/ffi.md that explains the selection
order, what each path does, and when a signature qualifies for it, and
point the existing per-architecture limits table at it. Add a call path
overview table to the FFI Fast API internals contributor guide and
cross-link the two documents.
Signed-off-by: Soul Lee <alus20x@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66086
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Add `Buffer.stringLength(input[, encoding])`, the counterpart of
`Buffer.byteLength()`: it returns the number of UTF-16 code units that
`buf.toString(encoding)` would produce, without decoding.
For UTF-8 the count is computed with simdutf. Invalid input is counted
with the same maximal-subpart replacement that the decoder applies, so
the result always matches `toString().length`. The other encodings are
computed from `byteLength` alone.
This lets code that accumulates streamed input check the result against
`buffer.constants.MAX_STRING_LENGTH` and size its memory budget before
decoding.
Refs: https://github.com/nodejs/node/issues/66062
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/66064
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
Requiring `node:trace_events` and calling `createTracing()` aborted
the process, and `getEnabledCategories()` dereferenced a null pointer,
when Node.js runs on an embedder's own platform
(`kNoInitializeNodeV8Platform`): no `tracing::Agent` exists then, and
`lib/trace_events.js` only checked the compile-time `hasTracing` flag
and `ownsProcessState` before handing the null agent to the binding.
Have the binding report whether an agent exists and throw the existing
`ERR_TRACE_EVENTS_UNAVAILABLE` when it does not, as for a
`--without-v8-platform` build.
Refs: https://github.com/nodejs/node/pull/19803
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65954
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
The current documentation states that `initialDelay` and `interval`
are divided by 1000 and rounded down, but does not explicitly mention
that values less than 1000ms will result in 0 seconds. This can lead
to confusion when users pass values like 400ms expecting sub-second
keepalive delays.
Add a concrete example showing that 400ms rounds down to 0 seconds.
Refs: https://github.com/nodejs/node/issues/57712
Signed-off-by: haramjeong <04harams77@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65869
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ulises Gascón <ulisesgascongonzalez@gmail.com>
Mounting a virtual file system requires calling vfs.mount() from inside
the program, so a program cannot itself be served from one: something
already running has to mount the VFS first.
Add two startup flags. --vfs-mount=<source> mounts a directory or an
archive as a virtual file system, and may be repeated.
--vfs-load=<source> mounts that source the same way and additionally
runs the entry point and all subsequent require()/import resolution
against it rather than against the real file system, so an application
can be run straight out of a directory or a ZIP archive:
node --experimental-vfs --vfs-load=my-app.zip
Both options append to one list, so mounts happen in the order written:
node --experimental-vfs --vfs-mount=a --vfs-load=b --vfs-mount=c
mounts a, b and c in that order and runs b. Mounting the same source
twice mounts it twice, at two mount points, and the entry point comes
from the mount --vfs-load contributed rather than the earlier one.
The provider backing a source is chosen from the source itself rather
than from its name: a directory is served by RealFSProvider and a file
whose bytes are a ZIP archive by ZipProvider, so an archive can carry
any extension. vfs.registerProvider() registers a provider for formats
there is no built-in for; selection is deferred until -r and --import
preloads have run, so a preloaded module can register one before its
source is claimed.
The entry point comes from the mount the way `node <directory>` takes
one, from package.json "main" or index.js. Nothing is consumed as an
entry point argument, so every positional reaches the program unchanged
from argv[2] on, and argv[1] reports the named source rather than the
reserved mount point, which is an opaque implementation detail.
--vfs-load may be given at most once, and is not permitted in
NODE_OPTIONS: which entry point runs is the command line's decision, and
NODE_OPTIONS is parsed first, so an environment variable could otherwise
redirect any invocation. --vfs-mount is permitted there. Workers inherit
the same mounts in the same order but not the loading: a worker mounts
what the parent mounted and runs its own entry point.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/65748
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
The operating system's dynamic loader cannot open a library that lives
in a mounted virtual file system: the reserved mount path has no real
inode. Native addons already handle this in require(): the loader
hands their bytes to process.dlopen(), which loads them from a
private, self-cleaning image - an anonymous in-memory memfd on Linux.
Make ffi.dlopen() and new DynamicLibrary() do the same transparently.
Mirroring the fs handler integration, the VFS hook installer sets a
library reader into node:ffi while at least one VFS is mounted and
clears it when the last one unmounts; DynamicLibrary consults it
before every load, so the dependency points from the VFS into ffi and
ffi never loads any VFS code. The reader hands the library's bytes to
the native constructor, which loads them from the same kind of image,
released right after the load, while library.path keeps reporting the
virtual path. Libraries on the real file system are unaffected and
load directly, and pay only a null check while no VFS is mounted.
The AddonImage materializer moves from an anonymous namespace in
node_binding.cc to node_binding.h so that node_ffi.cc can reuse it.
On Windows the image is now written and closed before the load,
because the loader shares read alone and a retained writable
delete-on-close handle failed the load with ERROR_SHARING_VIOLATION;
since a mapped image cannot be unlinked there, it is kept with the
module it loaded as and both are released at process exit. On POSIX
the image still never outlives the constructor call, so nothing is
left for dlclose() to clean up.
Also fix the VFS dlopen hook forwarding a missing flags argument as
undefined, which process.dlopen() coerces to 0 - not a valid
dlopen(2) mode - so loading any addon from the real file system
failed with EINVAL while a VFS was mounted.
Co-authored-by: Philipp Dunkel <pipobscure@users.noreply.github.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/65909
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
Often closely related to util.debounce use cases, this adds an
implementation of throttle(fn) .. returns a function that limits
the number of concurrent invocations of fn.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/65899
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
I found myself using debounce quite a bit recently while
testing some recent other additions (quic and dtls testing,
perf_hooks improvements, etc). I was using an npm dependency
right up until I realized just how generally useful it is
to actually have it Just There. So, since it was a holiday
and I just felt like it... util.debounce(...)
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65899
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Flags such --trace-event-file or any other flag that specifies a
directory are subject to permission model rules, but a "bypass"
isn't considered a vulnerability while it doesn't pose a
risk to the user application
Signed-off-by: RafaelGSS <rafael.nunu@hotmail.com>
PR-URL: https://github.com/nodejs/node/pull/66004
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
The current recordable `Histogram` does not maintain any sense
of time. Samples accumulate indefinitely as they are collected.
This implements a relatively simple sliding-window mechanism
that can be count or time based. The key benefit is that the
window remains fixed/bounded while samples are recorded.
The sliding window is chunk based. Internally, it maintains
a ring buffer of a fixed number of individual histograms.
When snapshot() is called, those are materialized into a
single combined histogram. As the window slides, older
chunks (and all of the samples they hold) are dropped from
the window so the window drops however many samples happened
to be in that chunk. This does mean that precision of the
window is determined by the chunk size.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/65825
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Previously, ClientRequest silently truncated responses that exceeded
request.maxHeadersCount while llhttp continued using omitted headers.
Reject these responses with HPE_HEADER_OVERFLOW so parser state and
exposed headers cannot diverge. This is a semver-major behavior change.
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/65010
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Aviv Keller <me@aviv.sh>
Implements quantile-respectful density estimate calcuation
on Histogram. Helps with tail-focused latency analysis
without retaining raw samples.
Baking this directly into Node.js, based on a 1 million
sample, 1k bin workload, this impl is roughly 150-325x
faster than performing the equivalent in Rscript.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/65806
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Return the private key, end-entity certificate, and any other
non-matching certificates from a PKCS#12 (.p12/.pfx) bundle as a
KeyObject and X509Certificate instances.
Node.js already parses PKCS#12 in SecureContext::LoadPKCS12, which backs
tls's `pfx` option, but the results are consumed directly into an
SSL_CTX and never reach JavaScript. Callers who need the key or the
certificates for anything other than an immediate TLS connection have to
shell out to `openssl pkcs12` or take a userland dependency.
SecureContext::LoadPKCS12 now uses this same parsing logic.
The binding wraps d2i_PKCS12_bio() and PKCS12_parse() and follows their
semantics, matching the existing TLS path: the first private key is
returned, the end-entity certificate is the one associated with that
key, and any remaining certificates are returned through
`additionalCertificates`. A bundle containing no private key reports
`certificate` as null and returns its certificates through
`additionalCertificates`.
Absent and empty passphrases are kept distinct, since OpenSSL treats
them differently. Bundles that require OpenSSL's legacy provider throw
ERR_CRYPTO_UNSUPPORTED_OPERATION, reusing the error added for the TLS
path.
Signed-off-by: bmuenzenmeyer <brian.muenzenmeyer@gmail.com>
Co-authored-by: Filip Skokan <panva.ip@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/65627
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
René