A single executable application built with "vfsArchive" mounted its
archive at the next free layer, like any file system a program mounts
itself, and only in the main thread. A worker had no mount at all, so
a path into the archive, such as the main script's __filename, could
not be loaded there.
Mount the archive at the layer reserved for --vfs-load instead, so it
is at the same mount point in every thread, and mount it in each
worker as well, with the main script placed at its root just as in
the main thread. The worker reads the main script through a new
getMainCode() binding, since no main script is handed to it. Unlike
the --vfs-load source, the archive needs no provider that a preload
could register, so its mount in a worker is not deferred past
--import.
Since the archive takes that mount point, reject --vfs-load alongside
it: --build-sea refuses it in "execArgv", and the executable refuses it
at startup when it comes from --node-options or from the execArgv of a
worker.
Also fix the example in vfs.md, which showed a file system a program
mounts itself at layer 0, the layer reserved for these mounts.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66614
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
pipeToSync() does not fail the writer when endSync() returns -1, so
that a caller can still close it asynchronously. A caller that cannot,
e.g. because the writer is sync-only and has no end(), would be left
with a writer that is neither closed nor failed.
Add a failOnIncompleteClose option (a Node.js extension) that fails
the writer with the thrown ERR_INVALID_STATE error in that case.
preventFail takes precedence over it.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
After writing every chunk, pipeToSync() treated endSync() returning -1
like any other error: it threw ERR_INVALID_STATE and, unless
preventFail was set, called writer.fail() with it. -1 only means that
the writer cannot close synchronously, e.g. a push() writer whose
consumer has not drained it yet. All of the data had been accepted,
but failing the writer discarded it, so the consumer saw an error
instead of the end of the stream, and the caller could not recover.
pipeToSync() still throws ERR_INVALID_STATE in that case, since it
never falls back to the async end(), but it no longer fails the
writer. The caller can still close it, e.g. with `await writer.end()`.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Document behaviors that were previously only visible in the code:
- the writer "closing" state after end()/endSync(),
- writes made from argument conversion being ordered first,
- zero-length push() writes not being buffered,
- broadcast.cancel() also closing the paired writer,
- merge() not waiting for the other sources' cleanup on error,
- the options argument passed to the tap() callback,
- FileHandle writer() ordering of un-awaited writes, and the lazy
locking of FileHandle pull() and pullSync().
Also link the WinterTC Iterable Streams API draft and list the exports
that are Node.js extensions to it.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
push(), duplex(), broadcast(), share() and shareSync() rejected an
explicit budget below 16384 bytes with ERR_OUT_OF_RANGE. The spec
(push() step 3, broadcast() step 1 and share() step 2) only requires
the implementation-defined default to be at least 16384 bytes; an
explicit budget is used as given. Small budgets are also useful in
tests and in memory-constrained code.
Accept any explicit budget of at least 1 byte, and validate it in one
place. Defaults are unchanged. The validation tests are updated to the
new lower bound; note that WebIDL conversion truncates fractions, so
1.5 is now a valid budget of 1 and 0.5 is used to exercise the
rejection instead.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
pull() threw synchronously when given an already-aborted signal. The
spec (Iterable Streams, Stream.pull() step 5) requires it to return an
iterable that throws the abort reason when read. This also matches how
broadcast.push() and share.pull() already handle a pre-aborted signal.
Once the signal aborted, the pull that observed the abort rejected,
but later pulls resolved { done: true } because the pipeline is an
async generator, which completes after throwing. The spec (step 7)
requires future pulls to reject with the abort reason as well, so
that a stream that was cancelled is never reported as having ended
cleanly.
Return an iterator that rejects every read with the abort reason once
the signal has aborted the pipeline, without starting the pipeline if
the signal was already aborted. Pipelines without a signal are not
affected. The two tests that asserted the synchronous throw now check
the rejection instead, and the documentation is updated.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
share() and shareSync() buffer each batch pulled from the source as a
single entry, and the 'drop-oldest' policy evicts whole entries until
the buffer is below the budget. from() and fromSync() combine up to 128
values of a sync source into one batch, so a single entry can be many
times larger than the budget. Evicting it discarded every chunk in it,
including chunks that slower consumers had not read yet: a consumer
could lose the entire stream while a faster consumer read all of it.
When the policy is 'drop-oldest', split batches that are larger than
the budget into consecutive entries that are each smaller than it, so
eviction keeps the newest chunks that fit within the budget.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
With 'drop-newest', a shareSync() consumer that needed to pull while
the buffer was at the budget discarded one entry from the source and
then returned { done: true } even though the source was not exhausted.
for...of loops, and anything else that trusts the iterator protocol,
silently stopped consuming. There is no correct alternative in a
synchronous context: the slowest consumer cannot advance while another
consumer's next() is running, so the call can neither wait for budget
nor keep discarding until budget is released.
Reject 'drop-newest' in shareSync() with ERR_INVALID_ARG_VALUE, as is
already done for 'unbounded'. The two tests that asserted the previous
"done but not detached" behavior are replaced by one that checks the
rejection. Also document how 'unbounded' and 'drop-newest' make the
async share() wait for the slowest consumer.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When a shareSync() consumer needed to pull while the buffer was at or
above the budget under the 'strict' policy, it threw ERR_OUT_OF_RANGE
but stayed registered. Neither for...of nor a pullSync() transform
pipeline calls return() when next() throws, so the abandoned consumer
kept its cursor forever, pinned every entry pulled afterwards, and made
the remaining consumers fail with ERR_OUT_OF_RANGE as well.
Detach the consumer before throwing, as the async share() already does
(see 1ad67bc66b), and document the behavior for both.
The spec notes that a rejected strict pull does not terminate the
consumer's iterator so that it may be retried. That is not safe with
the common iteration patterns, and will be raised with the spec
editors.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When the last consumer of a share() or shareSync() detached, the min
cursor fell back to the end of the buffer, so every buffered entry was
trimmed, including entries that the detaching consumer had not read.
The source had already produced that data, so consumers that attached
later silently skipped it.
Keep the buffer while there are no consumers, as broadcast() already
does, so late-joining consumers start at the oldest entry still in the
buffer. Document that the source stays open until the share is
cancelled or disposed.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
A failing todo subtest does not fail its parent, so the parent suite was
recorded as passed in the rerun state file along with its children. On
the next run the suite was replayed synthetically and the failing todo
subtest was marked as passed instead of being rerun.
Do not record a passed test whose descendant failed, so the suite body
runs again while passing siblings are still replayed from their own
entries. Document that a failing todo test counts as not yet passed.
Signed-off-by: Barak <26878518+barakor@users.noreply.github.com>
PR-URL: https://github.com/nodejs/node/pull/66605
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Fix several issues with `Utf8Stream` flushing:
* `flush()` now writes buffered data regardless of `minLength`
and invokes the callback only after pending writes complete,
including when `minLength` is zero and a write is in flight.
* Multiple concurrent `flush()` calls are tracked correctly and
`end()` waits for pending flushes before closing.
* `flushSync()` throws `ERR_INVALID_STATE` if called while an
asynchronous write is in progress instead of corrupting output.
* Periodic flushes no longer stack up while a flush is pending.
* `fsync` is skipped for stdout/stderr file descriptors.
* `reopen()`, `end()` and `destroy()` behave correctly when the
stream is destroyed while still opening.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/66473
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Every consumer in node:stream/iter retains what it reads, so there is
no way to read a streamable to completion while keeping nothing. To
clear a stream, callers write `for await (const _ of source) {}`. This
is apparent throughout multiple test suites, including the stream/iter
tests themselves.
These are especially useful for QUIC where a receiver doesn't want the
payload, but still has to read it to completion to relieve
backpressure. bytes() does that too, but allocates the whole payload to
discard it.
dump() pulls every batch and drops it, so peak memory is one batch
regardless of volume. It takes the same signal and limit options as
the other consumers, rejects if the source errors mid-stream, and
fulfills with undefined. dumpSync() is the synchronous form.
The name follows undici's body.dump(). drain() was the obvious
alternative but this module already exports ondrain() and
drainableProtocol() for writer-side backpressure, which is the
opposite side of the pipe.
Assisted-by: Claude Opus 5
Signed-off-by: Ethan Arrowood <ethan@arrowood.dev>
PR-URL: https://github.com/nodejs/node/pull/65598
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tim Perry <pimterry@gmail.com>
`session.close()` and `session.destroy()` only checked that
`options.code` was a bigint or a number. Numbers were then converted
to `uint64_t` unchecked, so fractions were truncated and negative,
non-finite or too large values were undefined behavior. Bigints were
only checked to fit in 64 bits, but QUIC error codes are
variable-length integers limited to 62 bits, so larger codes cannot
be encoded in the `CONNECTION_CLOSE` frame.
Reject codes that are not integers between `0` and `2n ** 62n - 1n`,
the range `QuicError` already enforces, before any close or destroy
side effects.
Signed-off-by: Christian Aurich Zanettini Martins <christian.aurichzm@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66302
Reviewed-By: Tim Perry <pimterry@gmail.com>
Add websocketMask(source, mask, output[, offset[, length]]) and
websocketUnmask(buffer, mask), which XOR data with a repeating 4-byte
key. This is the masking that WebSocket clients apply to every frame
they send (RFC 6455, Section 5.3), and that servers undo on every frame
they receive.
Userland WebSocket implementations do this either with a byte-by-byte
JS loop (undici, and ws without optional dependencies) or with the
bufferutil native addon. bufferutil is installed for only about 3% of
ws downloads and has no linux-arm64 prebuild, so almost all users run
the JS loop. The signature matches bufferutil, so existing users can
switch with a feature check, as ws did for buffer.isUtf8().
The implementation uses a V8 fast API call and processes 8-byte words,
which the compiler vectorizes. It handles overlapping source and output
views, and treats every view type as raw bytes. It is about 20x faster
than the JS loop at 1 KiB and 40x faster at 64 KiB, and faster than
bufferutil at every size from 32 bytes up.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/66335
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matthew Aitken <maitken033380023@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Allow providers to be layered at one mount point. Reads search layers
in priority order, while writes copy lower-layer files to the first
provider and deletions hide lower copies without changing them.
Assisted-by: pi
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/66235
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Allow safe integer numbers as int64 and uint64 arguments alongside
bigint values. Reject negative numbers for uint64 and numbers outside
the safe integer range. Keep 64-bit return values as bigint.
Apply validation and conversion across the Fast API, shared-buffer,
and generic argument conversion paths.
Add coverage for Number and BigInt boundaries, invalid inputs, and
single-argument calls before and after optimization.
Signed-off-by: HoonDongKang <d159123@naver.com>
Assisted-by: Codex:Astra-medium
PR-URL: https://github.com/nodejs/node/pull/66197
Fixes: https://github.com/nodejs/node/issues/66198
Reviewed-By: Daeyeon Jeong <daeyeon.dev@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Stopping a Worker while it is running an FFI callback aborted the whole
process with "Callbacks cannot throw an exception". This happened on
worker.terminate(), on process.exit() inside the callback, and when the
main thread exited while the Worker was in a callback, since exit
terminates all Workers.
All three stop the Worker by terminating execution, and InvokeCallback
treated the termination as a thrown exception. Check HasTerminated()
first and return a zeroed result so the native caller can unwind.
Callbacks that throw still abort.
Signed-off-by: Trivikram Kamat <16024985+trivikr@users.noreply.github.com>
Assisted-by: claude:opus-5.5
PR-URL: https://github.com/nodejs/node/pull/66389
Fixes: https://github.com/nodejs/node/issues/66388
Reviewed-By: Daeyeon Jeong <daeyeon.dev@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
With the FreeEnvironment() fix for sibling Environments and the handle
cleanup depth tracked per thread, only the Environment being freed
loses JavaScript while FreeEnvironment() runs the shared loop. Callbacks
of the other Environments on that loop run their JavaScript as usual.
Update embedding.md and the comment in node.h, which still describe
JavaScript as disallowed on the whole isolate.
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66239
Refs: https://github.com/nodejs/node/pull/65977
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
zstdCompress() writes its input and ends the frame in separate calls,
so zstd cannot infer the input size the way it does for
zstdCompressSync(), and sizes its tables for an unbounded stream.
Default pledgedSrcSize to the input's byte length. The output is now
identical to zstdCompressSync() and several times faster at higher
levels. An explicit pledgedSrcSize, or a string with a custom
defaultEncoding, keeps the current behavior.
Signed-off-by: James Ross <james@jross.me>
PR-URL: https://github.com/nodejs/node/pull/66358
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Add non-throwing counterparts of http.validateHeaderName() and
http.validateHeaderValue() that return a boolean instead of throwing.
Rejecting an invalid header with the existing validators costs a few
microseconds, because an error object and its stack trace are created,
compared to ~20ns for the boolean check. Userland HTTP implementations
such as undici (fetch Headers, request options) therefore keep private
copies of the token and field-value tables from _http_common. These new
functions let them reuse the core implementation.
isValidHeaderValue() accepts an optional `httpValidation` option
('strict' or 'relaxed') that has the same meaning as the option of the
same name on http.createServer() and http.request().
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66334
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Tim Perry <pimterry@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
When --process-timeout expires, --report-on-process-timeout asked
every Worker for a subreport and waited without a time limit. A Worker
blocked in a synchronous native call never answers, so the watchdog
force-exited the process before the report was written. That left a
truncated, invalid JSON file, and the forced-exit message was glued
onto the "Writing Node.js report to file" line.
For reports triggered by --process-timeout, wait at most two seconds
for Worker subreports and leave out Worker threads that have not
responded by then. The subreport state is now shared with the interrupt
callbacks, so a Worker that answers late does not touch freed memory.
Other report triggers are unchanged.
Signed-off-by: Trivikram Kamat <16024985+trivikr@users.noreply.github.com>
Assisted-by: claude:opus-5.5
PR-URL: https://github.com/nodejs/node/pull/66304
Fixes: https://github.com/nodejs/node/issues/66303
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Necessarily semver-major.
When `--permission` is on, every env var not matched by
`--allow-env` is removed at startup. It takes names,
prefix patterns (`PREFIX_*`), or `*`, repeatable or
comma-sep'd.
There are a range of env vars that Node.js itself uses,
and a default range that are generally known to be safe
in common usage. These are never scrubbed. These include
things like `NODE_OPTIONS`, `NODE_EXTRA_CA_CERTS`, `PATH`,
`HOME`, etc. `NODE_ENV` is not in the defaults and must
be allowed explicitly.
Proxy vars (`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`) are
also not in the defaults since they can carry credentials.
When `--use-env-proxy` or `NODE_USE_ENV_PROXY` is set and
any of them were removed, a single warning naming them is
emitted.
Env vars can be dropped at runtime after reading using
`permission.drop()`. This is a stronger protection than
using `process.env.FOO = undefined` because it will
scrub the env var also from the environment block.
On Linux, the removed entries are overwritten in the
initial environment block. fs reads of any other
process's /proc/<pid>/environ, ancestors included, are
denied regardless of `--allow-fs-read`. A process's own
is readable only with `--allow-env=*`. Symlinks are
resolved before the check so paths like
/dev/fd/../../<ppid>/environ are caught. The check only
canonicalizes paths that statfs() reports are on procfs.
On Windows, removal also clears the C runtime's copy
of the environ using _wputenv_s.
Reading a removed name returns undefined, warns once per
name, and publishes to a diagnostics channel.
Env file keys are allowed. If the user had reason to pass
in an env file the assumption is they meant to allow them.
File-source config (node.config.json and NODE_OPTIONS
from a .env file) can only narrow the allow list.
Embedders must call ScrubProcessEnvironment() themselves
on startup. This is left up to the embedder to determine
the exact timing but needs to be called before startup
actually happens.
Child processes are started with `--allow-env=*`. Those
either receive the explicit env they were started with
or only the env they inherit from the parent. Since the
parent process is scrubbed, and the child cannot read
any other process's /proc/<pid>/environ, it should never
see more than the parent can.
Main part of the impl was done by hand. Docs, tests,
verification pass, and cleanup nits were automated.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/66132
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Chris Swithinbank