When a net.Server is configured with a `blockList`, the peer address is
looked up with getpeername() and checked before the connection is
accepted. If the peer address cannot be obtained (for example, the
connection was reset before getpeername(), causing
uv_tcp_getpeername() to return ENOTCONN), the access-control check was
silently skipped and the connection proceeded, allowing a blocked peer
to reach the application's 'connection' handler (fail open).
Fail closed: when a BlockList is configured but the peer address cannot
be determined, close the connection instead of bypassing the check. A
connection whose peer address cannot be obtained cannot be proven to be
allowed, so it must be refused.
Ref: https://github.com/nodejs/node/issues/66422
Ref: https://github.com/libuv/libuv/pull/5337
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Assisted-by: pi
PR-URL: https://github.com/nodejs/node/pull/66640
Refs: https://github.com/nodejs/node/issues/66422
Refs: https://github.com/libuv/libuv/pull/5337
Reviewed-By: Tim Perry <pimterry@gmail.com>
Constructing a TransformStream allocated five closures for the sink and
source algorithms of its two sides, and each side adopted the start
promise through a wrapper promise plus a thenable job, so a stream
without a start() left about three kilobytes pending in the microtask
queue until the started steps ran. Per-stream creation bursts spend
most of their time copying that graph through the scavenger.
The sink and source algorithms are now shared functions that reach the
transform stream through a field on their controller state (the
controller is passed to the close, abort and cancel algorithms for
that), and the post-start steps of both sides are delivered by one
reaction chain on a shared promise, taking the same microtask hops as
the spec's start promise adoption: three after construction for a
non-thenable start result, two after the adopting promise settles for a
thenable one. The readable and writable transfer state records are
materialized on first transfer instead of per stream.
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/66643
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Mattias Buelens <mattias@buelens.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
decodeFormComponent() already finds the first percent character, but
hasPercentHex() scans the plain prefix again. Start the check at the
known position to avoid repeating that work.
Replacing plus signs with spaces preserves character positions, so
escape validation and decoding behavior remain unchanged.
Signed-off-by: Roberto Bianchi <roberto.bianchi@spendesk.com>
PR-URL: https://github.com/nodejs/node/pull/66696
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
A single executable application built with "vfsArchive" mounted its
archive at the next free layer, like any file system a program mounts
itself, and only in the main thread. A worker had no mount at all, so
a path into the archive, such as the main script's __filename, could
not be loaded there.
Mount the archive at the layer reserved for --vfs-load instead, so it
is at the same mount point in every thread, and mount it in each
worker as well, with the main script placed at its root just as in
the main thread. The worker reads the main script through a new
getMainCode() binding, since no main script is handed to it. Unlike
the --vfs-load source, the archive needs no provider that a preload
could register, so its mount in a worker is not deferred past
--import.
Since the archive takes that mount point, reject --vfs-load alongside
it: --build-sea refuses it in "execArgv", and the executable refuses it
at startup when it comes from --node-options or from the execArgv of a
worker.
Also fix the example in vfs.md, which showed a file system a program
mounts itself at layer 0, the layer reserved for these mounts.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66709
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
To stay cancellable while a source is pending, from() waits for every
value of an async source through waitForNormalization(). For each value
it created a PromiseWithResolvers(), raced it against the value with
SafePromiseRace(), which wraps both in new promises, and ran an async
function with try/finally. This was the largest per-batch cost of
normalizing an async source.
Wait with a single promise and a single reaction on the value instead,
and reject that promise directly on cancellation. The outcome is
unchanged: the value's result, its rejection, or the cancellation
reason, whichever comes first, and the cancellation reason if the
normalization was cancelled by the time the value fulfills.
With an async generator yielding 16-byte chunks, pipeTo() is about
1.7x faster and iterating from() about 1.75x faster.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
stream/iter registered its one-time 'abort' listeners with a new
`{ __proto__: null, once: true }` options object every time, which can
be once per chunk (abortableNext()) or per waiting write. Use a single
shared kNullOnceOption instead. It is frozen, because signals can come
from user code and a patched addEventListener() must not be able to
change the options for every later registration.
The difference is small, since the listener registration itself costs
much more: with 300,000 16-byte chunks, pull() with a signal is about
1.7% faster, push() writes that wait with a signal about 3% faster, and
pipeTo() and bytes() with a signal are unchanged.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
When a write filled the classic Writable, fromWritable() recorded that
it needed to drain, but only listened for 'drain' once a later write
was queued or something waited for drain. If the Writable emitted
'drain' before that, for example because its write callback ran on a
microtask or with process.nextTick(), the event was missed and the flag
was never cleared: the next write() or writev() never settled, canWrite
stayed false and ondrain() never resolved.
Listen for 'drain' as soon as a write returns false, and keep listening
until it is emitted.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
pull() passes every stateless transform call a new
`{ __proto__: null, signal }` options object, which V8 creates in
dictionary mode, once per batch and transform.
Create the options with a TransformOptions constructor instead, for
stateful transforms as well so that both forms receive the same kind of
object. Each call still gets its own object, as the pipeline requires.
Its prototype is a single frozen object with no %Object.prototype% in
its chain, so a transform cannot pass state to other transforms through
it, and with a non-enumerable `constructor` so that util.inspect()
prints `TransformOptions { signal }`.
With 300,000 single-chunk batches, pull() is about 2% faster with one
stateless transform and about 8% faster with four.
This is observable: the options object's prototype is no longer null.
A new test covers the options contract, and the documentation now
describes it, including that pullSync() passes transforms no options.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
The records queued when a stream/iter read, write or drain has to wait,
and merge()'s ready-queue entries, were `{ __proto__: null, ... }`
literals, which V8 creates in dictionary mode. They can be created once
per chunk: whenever the consumer is ahead of the producer, every read
waits, and with a full budget every write does.
Create them with constructors whose prototype is an empty, frozen,
null-prototype object: PendingRequest and PendingWrite (push(),
broadcast() and fromWritable()), QueuedWrite (fromWritable()) and
MergeEntry (merge()). fromWritable() drain waiters now settle through
resolve(false) instead of a per-waiter close() closure. merge() tells
error entries apart by their missing iterator rather than by a kind
string.
When every push() read waits for data, or every write waits behind a
full budget, reading or writing 16-byte chunks is about 30% faster.
fromWritable() with queued writes is about 18% faster, and merge() of
two sources about 9% faster.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
createBatchEntry() and recordChunk() snapshot every chunk in a
seven-property `{ __proto__: null, ... }` literal, and every batch gets
a `{ __proto__: null, views, byteLength }` record. V8 creates such
literals in dictionary mode, which is expensive for objects created
for every chunk.
Create them with constructors whose prototype is an empty, frozen,
null-prototype object instead, as for IterResult. They have fast
properties and a single shape, and are only used internally.
Writing 1e6 16-byte chunks into a push() stream and reading them back
is about 6x faster with writeSync(), 3.7x faster with writevSync() (4
chunks per call) and 1.7x faster with string chunks.
benchmark/streams/iter-throughput-share-sync.js improves by 67-93%,
iter-throughput-share.js by 4-8%, and iter-throughput-broadcast.js
with 4 consumers by 8%. pipeTo() with small chunks is about 1.5x
faster.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
Every write() and writeSync() allocated a `{ __proto__: null, context }`
options object for the WebIDL chunk conversion, and for a Uint8Array
chunk a second, six-property one with [AllowShared] and
[AllowResizable]. Both are created in V8 dictionary mode.
Return Uint8Array chunks directly from the WriterChunk converter: with
[AllowShared] and [AllowResizable], the Uint8Array conversion cannot
reject a value isUint8Array() accepts and returns the same object. Use
shared, frozen conversion contexts for chunks, chunk sequences and
write options; the converters only read them to build error messages.
Writing 1e6 16-byte Uint8Array chunks into a push() stream and reading
them back is about 27% faster with writeSync() and with writevSync()
(4 chunks per call). String chunks are unaffected. Behavior and error
messages are unchanged.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
The iterators of push(), share(), shareSync(), broadcast() consumers
and from() normalization created a `{ __proto__: null, done, value }`
literal for every result. V8 creates such literals in dictionary mode,
which makes them several times more expensive to create and read than
ordinary objects.
Create them with an IterResult constructor whose prototype is a single
frozen, null-prototype object instead. Results have fast properties
and a single shape, and still have no %Object.prototype% in their
prototype chain, so a polluted Object.prototype.then still cannot turn
a result into a thenable. Creating and reading a result is about 5x
faster in a microbenchmark.
benchmark/streams/iter-throughput-share-sync.js improves by 7% to 42%
(more with more consumers) and iter-throughput-share.js by 4-6%;
pipeTo() with small chunks is about 2.5% faster.
This is observable: results are no longer null-prototype objects, so
deepStrictEqual() comparisons against `{ __proto__: null, ... }` no
longer match, and util.inspect() prints them as
`IterResult { done, value }` (the prototype has a non-enumerable
`constructor` for that purpose). The five tests that compared results
that way now compare their own properties, and a new test covers the
result contract and the prototype pollution case.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
pipeToSync() threw ERR_INVALID_ARG_TYPE before writing anything when
the writer had no endSync() method and preventClose was not set.
endSync() is optional: the spec (pipeToSync() step 7) only calls it if
the writer has it, and pipeTo() already treats it that way. A writer
without endSync() now receives the data and is not closed. pipeToSync()
still never falls back to the async end().
This also fixes the from-sync-writev case of
benchmark/streams/iter-from-batching.js, whose writer has no endSync().
testPipeToSyncNoEndSync asserted the previous rejection and now checks
that the data is written and end() is not called. The documentation of
the writer requirements is corrected as well: only writeSync() is
required.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
Object literals with `__proto__: null` are created in V8 dictionary
mode. Create the objects that live as long as a stream and are used for
every chunk with ObjectSetPrototypeOf() instead, as was done for the
share and broadcast consumer state, so that they keep fast properties:
- the iterators returned by push(), pull(), share(), shareSync() and
broadcast() consumers, and the pull() consumer-cleanup wrapper,
- the iterators and the cancellation context used by from()
normalization,
- the async wrapper share() uses for sync sources.
Objects created per call or per chunk (iterator results, options bags,
promise resolver records, single-use iterables) keep the literal form:
for those, setting the prototype after creation costs more than it
saves, about 2x slower in a create-and-read microbenchmark. The
fromWritable() writer is also unchanged, since V8 keeps object literals
with accessors in dictionary mode regardless.
With 200,000 16-byte chunks, pipeTo() is about 3.5% faster and pull()
with a transform or a signal about 1-1.5% faster. In
benchmark/streams/iter-throughput-share*.js, share() and shareSync()
improve by 1-4.5%; no benchmark regressed significantly.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66648
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
When an asynchronous randomInt() call finds the cache empty, it queues
itself and starts randomFill() on the shared cache buffer. A synchronous
call made before that job completes refills the same buffer with
randomFillSync() and returns values from offset 0. The completion
callback then reset the offset to 0, so the queued call and later calls
returned the bytes that synchronous calls had already used. The
threadpool job and randomFillSync() could also write the buffer at the
same time.
Fill a separate buffer asynchronously and copy it into the cache once
the job is done.
Fixes: https://github.com/nodejs/node/issues/66595
Refs: https://github.com/nodejs/node/pull/35110
Signed-off-by: Jiawen Geng <technicalcute@gmail.com>
Assisted-by: claude:opus-5.5
PR-URL: https://github.com/nodejs/node/pull/66597
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
URLSearchParams.sort() copies both merge runs into temporary arrays.
Buffer only the left run and read the right run from the output array,
removing one scratch-array allocation and the right-run copying.
While left-side elements remain, the output cursor stays behind unread
right-side elements. Once the left run is exhausted, the right remainder
is already in place. Taking the left key on equality preserves
stability. The insertion-sort path below 50 pairs is unchanged.
Signed-off-by: Roberto Bianchi <roberto.bianchi@spendesk.com>
PR-URL: https://github.com/nodejs/node/pull/66665
Reviewed-By: Xuguang Mei <meixuguang@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Ilyas Shabi <ilyasshabi94@gmail.com>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
A source without pull() has nothing observable left to do in its
post-start step, since the started flag only gates calls into the pull
algorithm. Set the flag right away instead of from a microtask, so a
push-style ReadableStream allocates neither the closure nor the task
and is not kept alive until the next microtask checkpoint.
pipeTo and tee hold the only references to their reader and writer, so
their [[closedPromise]] records are never observed as promises. Install
the watchers as the records themselves, as pipeTo's ready hook already
does, instead of materializing a promise plus reaction per side, and
hand the erroring/release probes one shared pending promise. The tee's
cancel promise is likewise materialized by the first branch cancel.
Microtask ordering is unchanged: each hook enqueues its watcher at the
position the promise reaction would have had.
node benchmark/compare.js --runs 20 over benchmark/webstreams (46 rows,
all others within the confidence interval):
webstreams/creation.js kind='ReadableStream' *** +172.40%
webstreams/creation.js kind='ReadableStream.tee' *** +20.47%
webstreams/creation.js kind='ReadableStreamBYOBReader' *** +15.66%
webstreams/creation.js kind='ReadableStreamDefaultReader' *** +14.53%
webstreams/lifecycle.js kind='pipe-to' (40 runs) ** +10.98%
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/66424
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
Reviewed-By: James M Snell <jasnell@gmail.com>
A single executable application built with "vfsArchive" mounted its
archive at the next free layer, like any file system a program mounts
itself, and only in the main thread. A worker had no mount at all, so
a path into the archive, such as the main script's __filename, could
not be loaded there.
Mount the archive at the layer reserved for --vfs-load instead, so it
is at the same mount point in every thread, and mount it in each
worker as well, with the main script placed at its root just as in
the main thread. The worker reads the main script through a new
getMainCode() binding, since no main script is handed to it. Unlike
the --vfs-load source, the archive needs no provider that a preload
could register, so its mount in a worker is not deferred past
--import.
Since the archive takes that mount point, reject --vfs-load alongside
it: --build-sea refuses it in "execArgv", and the executable refuses it
at startup when it comes from --node-options or from the execArgv of a
worker.
Also fix the example in vfs.md, which showed a file system a program
mounts itself at layer 0, the layer reserved for these mounts.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
PR-URL: https://github.com/nodejs/node/pull/66614
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
AbortSignal.any() copies the source WeakRefs of a composite input into
the new composite, so nested composites share them. Those WeakRefs were
also used as the unregister tokens for the registrations that prune a
composite's sources once they are collected. When an inner composite
was collected first, its cleanup therefore also unregistered the outer
composite from the same sources. An observed outer composite then
stayed in gcPersistentSignals, together with its listeners, even after
all of its sources were gone and nothing could abort it anymore.
Use each composite's own WeakRef as the unregister token instead, so
collecting one composite only removes its own registrations.
Fixes: https://github.com/nodejs/node/issues/66612
Refs: https://github.com/nodejs/node/pull/62367
Assisted-by: Claude Code
Signed-off-by: journaltraces <kxenk@protonmail.com>
PR-URL: https://github.com/nodejs/node/pull/66617
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>
traceCallback() defaults to a frozen context but writes the callback's
result or error to it. Allocate a fresh context when undefined is passed
and subscribers are present, matching traceSync() and tracePromise().
Set the prototype after allocation to avoid dictionary-mode properties.
Assisted-by: pi
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/66481
Reviewed-By: Bryan English <bryan@bryanenglish.com>
pipeToSync() does not fail the writer when endSync() returns -1, so
that a caller can still close it asynchronously. A caller that cannot,
e.g. because the writer is sync-only and has no end(), would be left
with a writer that is neither closed nor failed.
Add a failOnIncompleteClose option (a Node.js extension) that fails
the writer with the thrown ERR_INVALID_STATE error in that case.
preventFail takes precedence over it.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
After writing every chunk, pipeToSync() treated endSync() returning -1
like any other error: it threw ERR_INVALID_STATE and, unless
preventFail was set, called writer.fail() with it. -1 only means that
the writer cannot close synchronously, e.g. a push() writer whose
consumer has not drained it yet. All of the data had been accepted,
but failing the writer discarded it, so the consumer saw an error
instead of the end of the stream, and the caller could not recover.
pipeToSync() still throws ERR_INVALID_STATE in that case, since it
never falls back to the async end(), but it no longer fails the
writer. The caller can still close it, e.g. with `await writer.end()`.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
The ERR_INVALID_ARG_TYPE messages for async iterable and promise inputs
read "must be an a synchronous input (not AsyncIterable)", because the
error formatter adds "an" to expected-type strings that contain
uppercase letters. Rephrase them in lowercase.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
The spec for stream/iter Writers (write(), step 2) requires writes to
a closed writer to reject with a TypeError, as the push(), broadcast()
and FileHandle writers do. The QUIC stream writer rejected write() and
writev() with a plain ERR_INVALID_STATE Error. Use its TypeError
variant; the error code is unchanged.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
The spec (Writer write(), step 2) requires writes to a closed writer to
reject with a TypeError. The fromWritable() adapter rejected write()
and writev() with ERR_STREAM_WRITE_AFTER_END, which is a plain Error,
unlike the other stream/iter writers.
Add a TypeError variant of ERR_STREAM_WRITE_AFTER_END and use it, so
the error code is unchanged.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
push(), duplex(), broadcast(), share() and shareSync() rejected an
explicit budget below 16384 bytes with ERR_OUT_OF_RANGE. The spec
(push() step 3, broadcast() step 1 and share() step 2) only requires
the implementation-defined default to be at least 16384 bytes; an
explicit budget is used as given. Small budgets are also useful in
tests and in memory-constrained code.
Accept any explicit budget of at least 1 byte, and validate it in one
place. Defaults are unchanged. The validation tests are updated to the
new lower bound; note that WebIDL conversion truncates fractions, so
1.5 is now a valid budget of 1 and 0.5 is used to exercise the
rejection instead.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
bytes(), text(), arrayBuffer(), array() and their sync variants kept a
snapshot object for every collected chunk, plus a batch entry and its
views array, so they can detect chunks that were resized or detached
before the result is assembled. For streams of many small chunks this
dominated memory use: collecting 1,000,000 one-byte chunks peaked at
about 490 MB of heap for 1 MB of data.
A non-empty view of a fixed-length, non-shared ArrayBuffer can only
change by its buffer being detached, which makes its byteLength 0, so
recording its byteLength is enough. Keep a full snapshot only for
empty views and views of resizable or shared buffers. The same input
now peaks at about 60 MB and is collected about five times faster.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When pipeTo() or pipeToSync() failed, they called writer.fail(error)
and then rethrew the error. If fail() itself threw, its exception
replaced the error that made the pipe fail, which was then lost.
Call fail() on a best-effort basis and always surface the original
error.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
pull() threw synchronously when given an already-aborted signal. The
spec (Iterable Streams, Stream.pull() step 5) requires it to return an
iterable that throws the abort reason when read. This also matches how
broadcast.push() and share.pull() already handle a pre-aborted signal.
Once the signal aborted, the pull that observed the abort rejected,
but later pulls resolved { done: true } because the pipeline is an
async generator, which completes after throwing. The spec (step 7)
requires future pulls to reject with the abort reason as well, so
that a stream that was cancelled is never reported as having ended
cleanly.
Return an iterator that rejects every read with the abort reason once
the signal has aborted the pipeline, without starting the pipeline if
the signal was already aborted. Pipelines without a signal are not
affected. The two tests that asserted the synchronous throw now check
the rejection instead, and the documentation is updated.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
The protocol lookup used by from(), fromSync(), and the
Broadcast/Share helpers only considered values with typeof 'object',
so a function implementing, e.g., Symbol.for('Stream.toStreamable') was
rejected with ERR_INVALID_ARG_TYPE. Functions are objects and the spec
does not exclude them; the iteration protocol checks already accept
them.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When an async source yielded a value that needed normalizing, such as
a nested async iterable, from() collected the resulting chunks and
only yielded them once 128 had accumulated or the value was fully
consumed. A slow nested stream therefore delivered nothing until it
ended, an endless one with fewer than 128 chunks in flight delivered
nothing at all, and the chunks piled up in memory meanwhile. This
affected every API built on from(), e.g. when concatenating streams
with `async function*() { yield fromReadable(a); yield fromReadable(b); }`.
Yield whatever has been collected right before waiting on a promise or
on a nested async iterable. Chunks that are produced together are still
batched (up to the same bound).
testFromBoundsNestedAsyncIterable asserted that the first batch from an
endless nested async iterable held exactly 128 chunks; it now checks
that the batch is non-empty and bounded, which is what it guards.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When an async source yielded an already-batched Uint8Array[] value,
from() passed it through as-is, however large it was. Every other
input shape (a sync source, or an array passed to from() or
fromSync() directly) splits such batches into batches of at most 128
chunks, which bounds the memory transforms must allocate per batch.
Apply the same bound to async sources. Batches within the bound are
still passed through without copying.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
If a shareSync() source read from a consumer of the same share while
producing a value, the nested read re-entered the source iterator.
For generators this threw "Generator is already running" from inside
the nested read, which recorded that error as the share's source
error while the outer read was still in progress, leaving every
consumer in an error state.
Fail the nested read with ERR_INVALID_STATE before touching the
share's state. The source sees the error and may handle it; if it lets
it escape, it becomes the source error as with any other source
failure.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
share() and shareSync() buffer each batch pulled from the source as a
single entry, and the 'drop-oldest' policy evicts whole entries until
the buffer is below the budget. from() and fromSync() combine up to 128
values of a sync source into one batch, so a single entry can be many
times larger than the budget. Evicting it discarded every chunk in it,
including chunks that slower consumers had not read yet: a consumer
could lose the entire stream while a faster consumer read all of it.
When the policy is 'drop-oldest', split batches that are larger than
the budget into consecutive entries that are each smaller than it, so
eviction keeps the newest chunks that fit within the budget.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
With 'drop-newest', a shareSync() consumer that needed to pull while
the buffer was at the budget discarded one entry from the source and
then returned { done: true } even though the source was not exhausted.
for...of loops, and anything else that trusts the iterator protocol,
silently stopped consuming. There is no correct alternative in a
synchronous context: the slowest consumer cannot advance while another
consumer's next() is running, so the call can neither wait for budget
nor keep discarding until budget is released.
Reject 'drop-newest' in shareSync() with ERR_INVALID_ARG_VALUE, as is
already done for 'unbounded'. The two tests that asserted the previous
"done but not detached" behavior are replaced by one that checks the
rejection. Also document how 'unbounded' and 'drop-newest' make the
async share() wait for the slowest consumer.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When a shareSync() consumer needed to pull while the buffer was at or
above the budget under the 'strict' policy, it threw ERR_OUT_OF_RANGE
but stayed registered. Neither for...of nor a pullSync() transform
pipeline calls return() when next() throws, so the abandoned consumer
kept its cursor forever, pinned every entry pulled afterwards, and made
the remaining consumers fail with ERR_OUT_OF_RANGE as well.
Detach the consumer before throwing, as the async share() already does
(see 1ad67bc66b), and document the behavior for both.
The spec notes that a rejected strict pull does not terminate the
consumer's iterator so that it may be retried. That is not safe with
the common iteration patterns, and will be raised with the spec
editors.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When the last consumer of a share() or shareSync() detached, the min
cursor fell back to the end of the buffer, so every buffered entry was
trimmed, including entries that the detaching consumer had not read.
The source had already produced that data, so consumers that attached
later silently skipped it.
Keep the buffer while there are no consumers, as broadcast() already
does, so late-joining consumers start at the oldest entry still in the
buffer. Document that the source stays open until the share is
cancelled or disposed.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
pull() and pullSync() locked the handle as soon as they were called,
but only released the lock from inside the iteration. An iterable that
was created but never consumed therefore left the handle locked
forever, so every later pull(), pullSync() and writer() call failed
with ERR_INVALID_STATE. pullSync() also took a reference on the handle
eagerly, and returning an iterator that had not started unlocked the
handle even if another consumer held the lock.
Take the lock (and the reference) when iteration actually starts, as
the documentation already describes ("locked while the iterable is
being consumed"). Iterating after the handle has been closed now fails
with ERR_INVALID_STATE instead of reading from a stale descriptor.
testPullLocking is updated accordingly: a second iterable may be
created while the first is unconsumed, but consuming it while the
first is being consumed still fails.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
When writer() was created without a `start` offset, every write() and
writev() call targeted the file descriptor's current position, and
nothing prevented several of them from being in flight at once.
Overlapping un-awaited writes then raced on the shared file offset
(and partial writes were completed in follow-up syscalls), silently
writing data at the wrong offsets while the reported byte count and
the final file size still looked correct.
Issue async writes one at a time, in call order. A write that is still
queued when the writer fails, or whose signal aborts while it is
queued, is no longer started.
Assisted-by: OpenCode
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/66483
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
A failing todo subtest does not fail its parent, so the parent suite was
recorded as passed in the rerun state file along with its children. On
the next run the suite was replayed synthetically and the failing todo
subtest was marked as passed instead of being rerun.
Do not record a passed test whose descendant failed, so the suite body
runs again while passing siblings are still replayed from their own
entries. Document that a failing todo test counts as not yet passed.
Signed-off-by: Barak <26878518+barakor@users.noreply.github.com>
PR-URL: https://github.com/nodejs/node/pull/66605
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Matteo Collina