A single executable application built with "vfsArchive" mounted its
archive at the next free layer, like any file system a program mounts
itself, and only in the main thread. A worker had no mount at all, so
a path into the archive, such as the main script's __filename, could
not be loaded there.
Mount the archive at the layer reserved for --vfs-load instead, so it
is at the same mount point in every thread, and mount it in each
worker as well, with the main script placed at its root just as in
the main thread. The worker reads the main script through a new
getMainCode() binding, since no main script is handed to it. Unlike
the --vfs-load source, the archive needs no provider that a preload
could register, so its mount in a worker is not deferred past
--import.
Since the archive takes that mount point, reject --vfs-load alongside
it: --build-sea refuses it in "execArgv", and the executable refuses it
at startup when it comes from --node-options or from the execArgv of a
worker.
Also fix the example in vfs.md, which showed a file system a program
mounts itself at layer 0, the layer reserved for these mounts.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66709
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
pull() passes every stateless transform call a new
`{ __proto__: null, signal }` options object, which V8 creates in
dictionary mode, once per batch and transform.
Create the options with a TransformOptions constructor instead, for
stateful transforms as well so that both forms receive the same kind of
object. Each call still gets its own object, as the pipeline requires.
Its prototype is a single frozen object with no %Object.prototype% in
its chain, so a transform cannot pass state to other transforms through
it, and with a non-enumerable `constructor` so that util.inspect()
prints `TransformOptions { signal }`.
With 300,000 single-chunk batches, pull() is about 2% faster with one
stateless transform and about 8% faster with four.
This is observable: the options object's prototype is no longer null.
A new test covers the options contract, and the documentation now
describes it, including that pullSync() passes transforms no options.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
pipeToSync() threw ERR_INVALID_ARG_TYPE before writing anything when
the writer had no endSync() method and preventClose was not set.
endSync() is optional: the spec (pipeToSync() step 7) only calls it if
the writer has it, and pipeTo() already treats it that way. A writer
without endSync() now receives the data and is not closed. pipeToSync()
still never falls back to the async end().
This also fixes the from-sync-writev case of
benchmark/streams/iter-from-batching.js, whose writer has no endSync().
testPipeToSyncNoEndSync asserted the previous rejection and now checks
that the data is written and end() is not called. The documentation of
the writer requirements is corrected as well: only writeSync() is
required.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
A single executable application built with "vfsArchive" mounted its
archive at the next free layer, like any file system a program mounts
itself, and only in the main thread. A worker had no mount at all, so
a path into the archive, such as the main script's __filename, could
not be loaded there.
Mount the archive at the layer reserved for --vfs-load instead, so it
is at the same mount point in every thread, and mount it in each
worker as well, with the main script placed at its root just as in
the main thread. The worker reads the main script through a new
getMainCode() binding, since no main script is handed to it. Unlike
the --vfs-load source, the archive needs no provider that a preload
could register, so its mount in a worker is not deferred past
--import.
Since the archive takes that mount point, reject --vfs-load alongside
it: --build-sea refuses it in "execArgv", and the executable refuses it
at startup when it comes from --node-options or from the execArgv of a
worker.
Also fix the example in vfs.md, which showed a file system a program
mounts itself at layer 0, the layer reserved for these mounts.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66614
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
pipeToSync() does not fail the writer when endSync() returns -1, so
that a caller can still close it asynchronously. A caller that cannot,
e.g. because the writer is sync-only and has no end(), would be left
with a writer that is neither closed nor failed.
Add a failOnIncompleteClose option (a Node.js extension) that fails
the writer with the thrown ERR_INVALID_STATE error in that case.
preventFail takes precedence over it.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
After writing every chunk, pipeToSync() treated endSync() returning -1
like any other error: it threw ERR_INVALID_STATE and, unless
preventFail was set, called writer.fail() with it. -1 only means that
the writer cannot close synchronously, e.g. a push() writer whose
consumer has not drained it yet. All of the data had been accepted,
but failing the writer discarded it, so the consumer saw an error
instead of the end of the stream, and the caller could not recover.
pipeToSync() still throws ERR_INVALID_STATE in that case, since it
never falls back to the async end(), but it no longer fails the
writer. The caller can still close it, e.g. with `await writer.end()`.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Document behaviors that were previously only visible in the code:
- the writer "closing" state after end()/endSync(),
- writes made from argument conversion being ordered first,
- zero-length push() writes not being buffered,
- broadcast.cancel() also closing the paired writer,
- merge() not waiting for the other sources' cleanup on error,
- the options argument passed to the tap() callback,
- FileHandle writer() ordering of un-awaited writes, and the lazy
locking of FileHandle pull() and pullSync().
Also link the WinterTC Iterable Streams API draft and list the exports
that are Node.js extensions to it.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
push(), duplex(), broadcast(), share() and shareSync() rejected an
explicit budget below 16384 bytes with ERR_OUT_OF_RANGE. The spec
(push() step 3, broadcast() step 1 and share() step 2) only requires
the implementation-defined default to be at least 16384 bytes; an
explicit budget is used as given. Small budgets are also useful in
tests and in memory-constrained code.
Accept any explicit budget of at least 1 byte, and validate it in one
place. Defaults are unchanged. The validation tests are updated to the
new lower bound; note that WebIDL conversion truncates fractions, so
1.5 is now a valid budget of 1 and 0.5 is used to exercise the
rejection instead.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
pull() threw synchronously when given an already-aborted signal. The
spec (Iterable Streams, Stream.pull() step 5) requires it to return an
iterable that throws the abort reason when read. This also matches how
broadcast.push() and share.pull() already handle a pre-aborted signal.
Once the signal aborted, the pull that observed the abort rejected,
but later pulls resolved { done: true } because the pipeline is an
async generator, which completes after throwing. The spec (step 7)
requires future pulls to reject with the abort reason as well, so
that a stream that was cancelled is never reported as having ended
cleanly.
Return an iterator that rejects every read with the abort reason once
the signal has aborted the pipeline, without starting the pipeline if
the signal was already aborted. Pipelines without a signal are not
affected. The two tests that asserted the synchronous throw now check
the rejection instead, and the documentation is updated.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
share() and shareSync() buffer each batch pulled from the source as a
single entry, and the 'drop-oldest' policy evicts whole entries until
the buffer is below the budget. from() and fromSync() combine up to 128
values of a sync source into one batch, so a single entry can be many
times larger than the budget. Evicting it discarded every chunk in it,
including chunks that slower consumers had not read yet: a consumer
could lose the entire stream while a faster consumer read all of it.
When the policy is 'drop-oldest', split batches that are larger than
the budget into consecutive entries that are each smaller than it, so
eviction keeps the newest chunks that fit within the budget.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
With 'drop-newest', a shareSync() consumer that needed to pull while
the buffer was at the budget discarded one entry from the source and
then returned { done: true } even though the source was not exhausted.
for...of loops, and anything else that trusts the iterator protocol,
silently stopped consuming. There is no correct alternative in a
synchronous context: the slowest consumer cannot advance while another
consumer's next() is running, so the call can neither wait for budget
nor keep discarding until budget is released.
Reject 'drop-newest' in shareSync() with ERR_INVALID_ARG_VALUE, as is
already done for 'unbounded'. The two tests that asserted the previous
"done but not detached" behavior are replaced by one that checks the
rejection. Also document how 'unbounded' and 'drop-newest' make the
async share() wait for the slowest consumer.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When a shareSync() consumer needed to pull while the buffer was at or
above the budget under the 'strict' policy, it threw ERR_OUT_OF_RANGE
but stayed registered. Neither for...of nor a pullSync() transform
pipeline calls return() when next() throws, so the abandoned consumer
kept its cursor forever, pinned every entry pulled afterwards, and made
the remaining consumers fail with ERR_OUT_OF_RANGE as well.
Detach the consumer before throwing, as the async share() already does
(see 1ad67bc66b), and document the behavior for both.
The spec notes that a rejected strict pull does not terminate the
consumer's iterator so that it may be retried. That is not safe with
the common iteration patterns, and will be raised with the spec
editors.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When the last consumer of a share() or shareSync() detached, the min
cursor fell back to the end of the buffer, so every buffered entry was
trimmed, including entries that the detaching consumer had not read.
The source had already produced that data, so consumers that attached
later silently skipped it.
Keep the buffer while there are no consumers, as broadcast() already
does, so late-joining consumers start at the oldest entry still in the
buffer. Document that the source stays open until the share is
cancelled or disposed.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
A failing todo subtest does not fail its parent, so the parent suite was
recorded as passed in the rerun state file along with its children. On
the next run the suite was replayed synthetically and the failing todo
subtest was marked as passed instead of being rerun.
Do not record a passed test whose descendant failed, so the suite body
runs again while passing siblings are still replayed from their own
entries. Document that a failing todo test counts as not yet passed.
Signed-off-by: Barak <26878518+barakor@users.noreply.github.com>
PR-URL: https://github.com/nodejs/node/pull/66605
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Fix several issues with `Utf8Stream` flushing:
* `flush()` now writes buffered data regardless of `minLength`
and invokes the callback only after pending writes complete,
including when `minLength` is zero and a write is in flight.
* Multiple concurrent `flush()` calls are tracked correctly and
`end()` waits for pending flushes before closing.
* `flushSync()` throws `ERR_INVALID_STATE` if called while an
asynchronous write is in progress instead of corrupting output.
* Periodic flushes no longer stack up while a flush is pending.
* `fsync` is skipped for stdout/stderr file descriptors.
* `reopen()`, `end()` and `destroy()` behave correctly when the
stream is destroyed while still opening.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/66473
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Every consumer in node:stream/iter retains what it reads, so there is
no way to read a streamable to completion while keeping nothing. To
clear a stream, callers write `for await (const _ of source) {}`. This
is apparent throughout multiple test suites, including the stream/iter
tests themselves.
These are especially useful for QUIC where a receiver doesn't want the
payload, but still has to read it to completion to relieve
backpressure. bytes() does that too, but allocates the whole payload to
discard it.
dump() pulls every batch and drops it, so peak memory is one batch
regardless of volume. It takes the same signal and limit options as
the other consumers, rejects if the source errors mid-stream, and
fulfills with undefined. dumpSync() is the synchronous form.
The name follows undici's body.dump(). drain() was the obvious
alternative but this module already exports ondrain() and
drainableProtocol() for writer-side backpressure, which is the
opposite side of the pipe.
Assisted-by: Claude Opus 5
Signed-off-by: Ethan Arrowood <ethan@arrowood.dev>
PR-URL: https://github.com/nodejs/node/pull/65598
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Tim Perry <pimterry@gmail.com>
`session.close()` and `session.destroy()` only checked that
`options.code` was a bigint or a number. Numbers were then converted
to `uint64_t` unchecked, so fractions were truncated and negative,
non-finite or too large values were undefined behavior. Bigints were
only checked to fit in 64 bits, but QUIC error codes are
variable-length integers limited to 62 bits, so larger codes cannot
be encoded in the `CONNECTION_CLOSE` frame.
Reject codes that are not integers between `0` and `2n ** 62n - 1n`,
the range `QuicError` already enforces, before any close or destroy
side effects.
Signed-off-by: Christian Aurich Zanettini Martins <christian.aurichzm@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66302
Reviewed-By: Tim Perry <pimterry@gmail.com>
Add websocketMask(source, mask, output[, offset[, length]]) and
websocketUnmask(buffer, mask), which XOR data with a repeating 4-byte
key. This is the masking that WebSocket clients apply to every frame
they send (RFC 6455, Section 5.3), and that servers undo on every frame
they receive.
Userland WebSocket implementations do this either with a byte-by-byte
JS loop (undici, and ws without optional dependencies) or with the
bufferutil native addon. bufferutil is installed for only about 3% of
ws downloads and has no linux-arm64 prebuild, so almost all users run
the JS loop. The signature matches bufferutil, so existing users can
switch with a feature check, as ws did for buffer.isUtf8().
The implementation uses a V8 fast API call and processes 8-byte words,
which the compiler vectorizes. It handles overlapping source and output
views, and treats every view type as raw bytes. It is about 20x faster
than the JS loop at 1 KiB and 40x faster at 64 KiB, and faster than
bufferutil at every size from 32 bytes up.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/66335
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matthew Aitken <maitken033380023@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Allow providers to be layered at one mount point. Reads search layers
in priority order, while writes copy lower-layer files to the first
provider and deletions hide lower copies without changing them.
Assisted-by: pi
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/66235
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Allow safe integer numbers as int64 and uint64 arguments alongside
bigint values. Reject negative numbers for uint64 and numbers outside
the safe integer range. Keep 64-bit return values as bigint.
Apply validation and conversion across the Fast API, shared-buffer,
and generic argument conversion paths.
Add coverage for Number and BigInt boundaries, invalid inputs, and
single-argument calls before and after optimization.
Signed-off-by: HoonDongKang <d159123@naver.com>
Assisted-by: Codex:Astra-medium
PR-URL: https://github.com/nodejs/node/pull/66197
Fixes: https://github.com/nodejs/node/issues/66198
Reviewed-By: Daeyeon Jeong <daeyeon.dev@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Stopping a Worker while it is running an FFI callback aborted the whole
process with "Callbacks cannot throw an exception". This happened on
worker.terminate(), on process.exit() inside the callback, and when the
main thread exited while the Worker was in a callback, since exit
terminates all Workers.
All three stop the Worker by terminating execution, and InvokeCallback
treated the termination as a thrown exception. Check HasTerminated()
first and return a zeroed result so the native caller can unwind.
Callbacks that throw still abort.
Signed-off-by: Trivikram Kamat <16024985+trivikr@users.noreply.github.com>
Assisted-by: claude:opus-5.5
PR-URL: https://github.com/nodejs/node/pull/66389
Fixes: https://github.com/nodejs/node/issues/66388
Reviewed-By: Daeyeon Jeong <daeyeon.dev@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Rafael Gonzaga