- Add support for --build-snapshot-config which allows passing
snapshot configurations via a JSON configuration file.
- Add support for node::SnapshotConfig in the embedder API
The initial configurable options are:
- "builder" (SnapshotConfig::builder_script_path): path to the
builder script.
- "withoutCodeCache" (SnapshotFlags::kWithoutCodeCache): disable
code cache generation.
PR-URL: https://github.com/nodejs/node/pull/50453
Refs: https://github.com/nodejs/node/issues/42566
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
This patch implements a debugging flag that dumps the current
stack trace when a promise is created or resolved. To reduce
noise we ignore before and after events (as the execution is
less interesting) and use the per-isolate hook to avoid extra
JS frames. This flag can assist work in reducing unnecessary
promise overhead.
PR-URL: https://github.com/nodejs/node/pull/50899
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
- Make processBindingAllowList a separate list from
runtimeDeprecatedList and legacyWrapperList instead of being
an umbrella one, so it's easier to see the stages the bindings
are in.
- Cache process.binding() results so we don't need to mutate
runtimeDeprecatedList.
PR-URL: https://github.com/nodejs/node/pull/50773
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: James M Snell <jasnell@gmail.com>
We previously used a text that appears to be an excerpt of
https://zh.wikipedia.org/wiki/%E5%8D%97%E8%B6%8A%E5%9B%BD
and can have copyright/license complications. It may
also include some geopolitical nuances. The text has been
repeated through out the code base without much reuse.
This patch consolidates the fixtures by adding a common helper
string as `fixtures.utf8TestText` which is identical to a copy
in test/fixtures/utf8_test_text.txt. It also updates the text
to a copy of 蘭亭集序, It was chosen because:
1. It's a well-known Chinese classical piece written in 353 CE
and therefore in public domain. The string is copied from
https://zh.wikisource.org/zh-hant/%E8%98%AD%E4%BA%AD%E9%9B%86%E5%BA%8F
which contains a disclaimer of copyright for this reason.
2. The text is in suitable length for general UTF8 string
read/write tests (including punctuations, 389 code points and
1167 bytes).
3. This is also commonly used as reference text for Chinese text
layout tests.
4. It's a timeless and harmless preface for a collection of poems,
written by a uncontroversial figure who passed away >1600 years
ago and contains no geopolitical nuances. Background and an
English translation of this text can be found at
https://en.wikipedia.org/wiki/Lantingji_Xu
PR-URL: https://github.com/nodejs/node/pull/50732
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
This is a ESM benchmark, rewrite it so that we are directly
benchmarking the ESM import.meta paths and using number of
loads for op/s calculation, instead of doing it in startup
benchmarks and nesting number of process/workers spawn
for op/s calculation.
PR-URL: https://github.com/nodejs/node/pull/50683
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
When --node-builtin-modules-path is used, we read and create
new strings for builtins in each realm which increases the memory
usage. As a result GC may not be able to keep up with the
allocation done in the loop in the test.
As a workaround, give GC a bit more time by waiting for a timer
in the loop.
PR-URL: https://github.com/nodejs/node/pull/50735
Refs: https://github.com/nodejs/node/issues/50726
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
This benchmarks the startup of various CLI tools that are already
checked into the source code. We use --version because the output
tends to be minimal and fewer operations are done to generate
these so that the startup cost is still dominated by a more
indispensible part of the CLI.
PR-URL: https://github.com/nodejs/node/pull/50684
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
There is currently no reliable way to detect this leak because:
1. We cannot reliably get a reference to the channel from the
API to detect finalization without creating another strong reference.
2. This test does gc() and then checks memory usage - however the
use of gc() disables code aging which can actually lead to increased
memory usage overall, as it is not intended to be used to lower
memory usage in the first place.
3. The implementation of diagnostics channels relies on ephemeron gc
which is inefficient, it's not reliable to use the typical "create
a lot of objects and see if it crashes" trick to check leaks.
Skip the test for now until we find a way to test it reliably.
To avoid flakiness in the CI, it's better to remove an unreliable
test altogether.
PR-URL: https://github.com/nodejs/node/pull/50327
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Users cannot access any API that can be used to return a module or
module namespace in this callback without --experimental-vm-modules
anyway, so this would eventually lead to a rejection. This patch
rejects in this case with our own error message and use a constant
host-defined option for the rejection, so that scripts with the
same source can still be compiled using the compilation cache
if no `import()` is actually called in the script.
PR-URL: https://github.com/nodejs/node/pull/50137
Refs: https://github.com/nodejs/node/issues/35375
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Instead of using the public versions of the vm APIs internally,
use the internal versions so that we can skip unnecessary
argument validation.
The public versions would need special care to the generation
of host-defined options to hit the isolate compilation cache
when imporModuleDynamically isn't used, while internally it's
almost always used, so this allows us to handle the host-defined
options separately.
PR-URL: https://github.com/nodejs/node/pull/50137
Refs: https://github.com/nodejs/node/issues/35375
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
To improve determinism of snapshot generation, add --predictable
to the V8 flags used to initialize a process launched to generate
snapshot. Also add a kGeneratePredictableSnapshot flag
to ProcessInitializationFlags for this and moves the configuration
of these flags into node::InitializeOncePerProcess() so that
it can be shared by embedders.
PR-URL: https://github.com/nodejs/node/pull/48749
Refs: https://github.com/nodejs/build/issues/3043
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Previously to throw errors from C++ land, sync versions of the fs
were created by copying C++ code from the original implementation
and moving JS code to a separate file. This can lead to several
problems:
1. By moving code to a new file for the sake of moving, it would
be harder to use git blame to trace changes and harder to backport
changes to older branches.
2. Scattering the async and sync versions of fs methods in
different files makes it harder to keep them in sync and
share code in the prologues and epilogues.
3. Having two copies of code doing almost the same thing results
in duplication and can be prone to out-of-sync problems when the
prologue and epilogue get updated.
4. There is a minor cost to startup in adding an additional file.
This can add up even with the help of snapshots.
This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4
and introduces C++ helpers SyncCallAndThrowIf() and
SyncCallAndThrowOnError() so that the original implementations
can be easily tweaked to allow throwing from C++ and stop 3.
PR-URL: https://github.com/nodejs/node/pull/49913
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously when checking the initial timing we did a lot of checks
after accessing and copying timing.duration and before we check
that timing.duration is roughly the same as performance.now(),
which can lead to flakes if the overhead of the checking is
big enough. Update the test to check timing.duration against
performance.now() as soon as possible when it's copied instead
of computed.
:#
PR-URL: https://github.com/nodejs/node/pull/49892
Refs: https://github.com/nodejs/reliability/issues/676
Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Richard Lau <rlau@redhat.com>
This patch:
- Documents `--with-icu-default-data-dir` and its precedence
- Elaborates a bit more about the format of the name of the expected
data file.
PR-URL: https://github.com/nodejs/node/pull/49666
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Previously when we fail to initialize ICU data, the error message is
```
could not initialize ICU (check NODE_ICU_DATA or --icu-data-dir
parameters)
```
This patch updates it to something similar to:
```
U_FILE_ACCESS_ERROR: Could not initialize ICU. Check the directory
specified by NODE_ICU_DATA or --icu-data-dir contains icudt73l.dat
and it's readable
```
Where the expected data file name is the same as U_ICUDATA_NAME.
PR-URL: https://github.com/nodejs/node/pull/49666
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Previously we simply create a lot of the target objects and check
if the process crash due to OOM. Due to how we use emphemeron GC
to handle memory management, which is inefficient but necessary
for correctness, the tests can produce false positives as
the GC isn't efficient enough to catch up with a very fast
heap growth.
This patch uses a new checkIfCollectable() utility to terminate the
test early once we detect that any of the target object can actually
be garbage collected. This should lower the chance of false positives.
As a drive-by this also allows us to use setImmediate() to grow the
heap even faster to make the tests run faster.
PR-URL: https://github.com/nodejs/node/pull/49671
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
There were a few bugs in the original test that went unnoticed
because with the bug the test did not actually get run anymore.
This patch fixes the argument computation by accounting filtering
of the arguments, and uses spawnSyncAndExit{WithoutError} in
the test to enable better logging when the test fails.
PR-URL: https://github.com/nodejs/node/pull/49506
Fixes: https://github.com/nodejs/node/issues/49501
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
It is flaky due to the same cause of test-child-process-pipe-dataflow
being flaky - cygwin quirks - so skip it on Windows too.
Drive-by: remove the skip mark of test-child-process-pipe-dataflow
in the status file and directly skip it in the test with a comment.
PR-URL: https://github.com/nodejs/node/pull/49621
Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Previously when managing the importModuleDynamically callback of
vm.compileFunction(), we use an ID number as the host defined option
and maintain a per-Environment ID -> CompiledFnEntry map to retain
the top-level referrer function returned by vm.compileFunction() in
order to pass it back to the callback, but it would leak because with
how we used v8::Persistent to maintain this reference, V8 would not
be able to understand the cycle and would just think that the
CompiledFnEntry was supposed to live forever. We made an attempt
to make that reference known to V8 by making the CompiledFnEntry weak
and using a private symbol to make CompiledFnEntry strongly
references the top-level referrer function in
https://github.com/nodejs/node/pull/46785, but that turned out to be
unsound, because the there's no guarantee that the top-level function
must be alive while import() can still be initiated from that
function, since V8 could discard the top-level function and only keep
inner functions alive, so relying on the top-level function to keep
the CompiledFnEntry alive could result in use-after-free which caused
a revert of that fix.
With this patch we use a symbol in the host defined options instead of
a number, because with the stage-3 symbol-as-weakmap-keys proposal
we could directly use that symbol to keep the referrer alive using a
WeakMap. As a bonus this also keeps the other kinds of referrers
alive as long as import() can still be initiated from that
Script/Module, so this also fixes the long-standing crash caused by
vm.Script being GC'ed too early when its importModuleDynamically
callback still needs it.
PR-URL: https://github.com/nodejs/node/pull/48510
Refs: https://github.com/nodejs/node/issues/44211
Refs: https://github.com/nodejs/node/issues/42080
Refs: https://github.com/nodejs/node/issues/47096
Refs: https://github.com/nodejs/node/issues/43205
Refs: https://github.com/nodejs/node/issues/38695
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Original commit message:
Reland "[api] allow v8::Data as internal field"
This is a reland of commit 0aa622e12893e9921c01a34ce9507b544e599c4a
The original patch tried to run a test that calls exit() in the
fatal error handler in parallel, which would not work. This marked
the test with TEST() to avoid running it in parallel.
Original change's description:
> [api] allow v8::Data as internal field
>
> Previously only v8::Value can be stored as internal fields.
> In some cases, however, it's necessary for the embedder to
> tie the lifetime of a v8::Data with the lifetime of a
> JS object, and that v8::Data may not be a v8::Value, as
> it can be something returned from the V8 API. One way to
> keep the v8::Data alive may be to use a v8::Persistent<v8::Data>
> but that can easily lead to leaks.
>
> This patch changes v8::Object::GetInternalField() and
> v8::Object::SetInernalField() to accept v8::Data instead of just
> v8::Value, so that v8::Data can kept alive by a JS object in
> a way that the GC can be aware of to address this problem.
> This is a breaking change for embedders
> using v8::Object::GetInternalField() as it changes the return
> type. Since most v8::Value subtypes only support direct casts
> from v8::Value but not v8::Data, calls like
>
> object->GetInternalField(index).As<v8::External>()
>
> needs to be updated to cast the value to v8::Value first:
>
> object->GetInternalField(index).As<v8::Value>().As<v8::External>()
>
> Bug: v8:14120
> Change-Id: I731c958d1756b9d5ee4a3e78813416cd60d1b7ca
> Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4707972
> Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
> Commit-Queue: Joyee Cheung <joyee@igalia.com>
> Cr-Commit-Position: refs/heads/main@{#89718}
Bug: v8:14120
Change-Id: I3e45d09b5c300d5eefc73e380ef21ac2bd61760c
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4834471
Commit-Queue: Joyee Cheung <joyee@igalia.com>
Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/main@{#89824}
Refs: https://github.com/v8/v8/commit/93b1a74cbc9b5382a145051b9475433bc6107f1e
PR-URL: https://github.com/nodejs/node/pull/49419
Refs: https://github.com/v8/v8/commit/0aa622e12893e9921c01a34ce9507b544e599c4a
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
This makes it easier to locate indeterminism in the snapshot, with
the following command:
$ ./configure --write-snapshot-as-array-literals
$ make V=
$ mv out/Release/obj/gen/node_snapshot.cc ./node_snapshot.cc
$ make V=
$ diff out/Release/obj/gen/node_snapshot.cc ./node_snapshot.cc
PR-URL: https://github.com/nodejs/node/pull/49312
Refs: https://github.com/nodejs/build/issues/3043
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
We do not actually need to deserialize the context and the whole
environment to compile the code cache, since code cache are not
context-dependent anyway, deserializing just the isolate snapshot is
enough.
PR-URL: https://github.com/nodejs/node/pull/49288
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
PromiseWrap has been removed in
https://github.com/nodejs/node/pull/39135 and we do not have any
internal object setting the internal field at 0 as a promise
(we always set the first field as an aligned pointer to
the embedder ID). As result GetAssignedPromiseWrapAsyncId()
always just returns AsyncWrap::kInvalidAsyncId and turn
the removed block into noops. So the block just can be removed.
PR-URL: https://github.com/nodejs/node/pull/49335
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
V8 now requires code cache to be compiled from an isolate with the
same RO space layout as the one that's going to deserialize the
cache, so for a binary built with snapshot, we need to compile
the code cache using a deserialized isolate.
Drive-by: ignore "useCodeCache" when "useSnapshot" is true because
the compilation would've been done during build time anyway in
that case, and print a warning for it.
PR-URL: https://github.com/nodejs/node/pull/49226
Refs: https://github.com/nodejs/node-v8/issues/252
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Replaces expectSyncExit() and expectSyncExitWithoutError(). Since
we usually just check the child process right after its spawned,
these shorthands also takes care of the spawning. This makes the
tests more concise.
PR-URL: https://github.com/nodejs/node/pull/49200
Reviewed-By: LiviaMedeiros <livia@cirno.name>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Previously this can be flaky because the there could be a delay
of the deallocation after gc() is invoked. Use gcUntil() to run
the GC multiple times to make the test more robust.
PR-URL: https://github.com/nodejs/node/pull/49168
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
V8 now no longer supports serializing code cache in an isolate
with unfinalized read-only space. So guard the code cache regeneration
with the `is_building_snapshot()` flag. When the isolate is created
for snapshot generation, the code cache is going to be serialized
separately anyway, so there is no need to do it in the builtin loader.
PR-URL: https://github.com/nodejs/node/pull/49108
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously the ContextifyContext holds a MicrotaskQueueWrap which in
turns holds a MicrotaskQueue in a shared pointer. The indirection is
actually unnecessary, we can directly hold the MicrotaskQueue via
a unique pointer in ContextifyContext, the lifetime would still
remain the same but the graph would be simpler, and this removes
the additional JS -> C++ to create the wrapper object.
PR-URL: https://github.com/nodejs/node/pull/48982
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
These can be used to check the state and the output of a child
process launched with `spawnSync()`. They log additional information
about the child process when the check fails to facilitate debugging
test failures.
PR-URL: https://github.com/nodejs/node/pull/49020
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
We previously only return startup data for the first slot for
BaseObjects because we can already serialize all the necessary
information in one go, but slots that do not get special startup
data would be serialized verbatim which means that the pointer
addresses are going to be part of the snapshot blob, resulting
in indeterminism.
This patch updates the serialization routines and capture information
for both of the two slots - the first slot with type information
and memory management type (which we can use in the future for
cppgc-managed objects) and the second slot with data about the
object itself. This way the embeedder slots can be serialized
in a reproducible manner in the snapshot.
PR-URL: https://github.com/nodejs/node/pull/48996
Refs: https://github.com/nodejs/build/issues/3043
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
- Build a static table of octal strings and use it instead of
building octal strings repeatedly during printing.
- Print a newline and an offset for every 64 bytes in the case
of printing array literals so it's easier to locate
variation in snapshot blobs.
- Rework the printing routines so that the differences are only
made in a WriteByteVectorLiteral routine. We can update this
for compression support in the future.
- Rename Snapshot::Generate() that write the data as C++ source
instead of a blob as Snaphost::GenerateAsSource() for clarity,
and move the file stream operations into it to streamline
error handling.
PR-URL: https://github.com/nodejs/node/pull/48851
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Joyee Cheung