This reduces the duplication in the code base, also makes
makes the log messages more concise since the `[--trace-env]`
prefix should provide enough indication about the fact that
it's about environment variable access.
PR-URL: https://github.com/nodejs/node/pull/56293
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This implements --trace-env, --trace-env-js-stack and
--trace-env-native-stack CLI options which can be used to find
out what environment variables are accessed and where they are
accessed.
PR-URL: https://github.com/nodejs/node/pull/55604
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This patch implements automatic on-disk code caching that can be enabled
via an environment variable NODE_COMPILE_CACHE.
When set, whenever Node.js compiles a CommonJS or a ECMAScript Module,
it will use on-disk [V8 code cache][] persisted in the specified
directory to speed up the compilation. This may slow down the first
load of a module graph, but subsequent loads of the same module graph
may get a significant speedup if the contents of the modules do not
change. Locally, this speeds up loading of
test/fixtures/snapshot/typescript.js from ~130ms to ~80ms.
To clean up the generated code cache, simply remove the directory.
It will be recreated the next time the same directory is used for
`NODE_COMPILE_CACHE`.
Compilation cache generated by one version of Node.js may not be used
by a different version of Node.js. Cache generated by different versions
of Node.js will be stored separately if the same directory is used
to persist the cache, so they can co-exist.
Caveat: currently when using this with V8 JavaScript code coverage, the
coverage being collected by V8 may be less precise in functions that are
deserialized from the code cache. It's recommended to turn this off when
running tests to generate precise coverage.
Implementation details:
There is one cache file per module on disk. The directory layout
is:
- Compile cache directory (from NODE_COMPILE_CACHE)
- 8b23c8fe: CRC32 hash of CachedDataVersionTag + NODE_VERESION
- 2ea3424d:
- 10860e5a: CRC32 hash of filename + module type
- 431e9adc: ...
- ...
Inside the cache file, there is a header followed by the actual
cache content:
```
[uint32_t] code size
[uint32_t] code hash
[uint32_t] cache size
[uint32_t] cache hash
... compile cache content ...
```
When reading the cache file, we'll also check if the code size
and code hash match the code that the module loader is loading
and whether the cache size and cache hash match the file content
read. If they don't match, or if V8 rejects the cache passed,
we'll ignore the mismatch cache, and regenerate the cache after
compilation succeeds and rewrite it to disk.
PR-URL: https://github.com/nodejs/node/pull/52535
Refs: https://github.com/nodejs/node/issues/47472
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
In many cases we can query the environment variable directly instead
of encoding it into UTF8 first and then decoding it again from
UTF8 via an intermediate V8 string.
Drive-by: pass per_process::system_environment explicitly when
the real environment variables are supposed to be used instead of
relying on fallback with defaults.
PR-URL: https://github.com/nodejs/node/pull/51829
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
- Add support for --build-snapshot-config which allows passing
snapshot configurations via a JSON configuration file.
- Add support for node::SnapshotConfig in the embedder API
The initial configurable options are:
- "builder" (SnapshotConfig::builder_script_path): path to the
builder script.
- "withoutCodeCache" (SnapshotFlags::kWithoutCodeCache): disable
code cache generation.
PR-URL: https://github.com/nodejs/node/pull/50453
Refs: https://github.com/nodejs/node/issues/42566
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Fix double free reported by coverity. ToBufferEndian()
in node_i18n.cc was the only caller of Buffer::New() passing
in a MaybeStackBuffer. Coverity reported a double free
because there were paths in which the src buffer would
be deleted by both the destruction of the MaybeStackBuffer and
by the Buffer which was done even in failure cases for
Buffer::New().
Signed-off-by: Michael Dawson <midawson@redhat.com>
PR-URL: https://github.com/nodejs/node/pull/51046
Reviewed-By: James M Snell <jasnell@gmail.com>
This patch implements a debugging flag that dumps the current
stack trace when a promise is created or resolved. To reduce
noise we ignore before and after events (as the execution is
less interesting) and use the per-isolate hook to avoid extra
JS frames. This flag can assist work in reducing unnecessary
promise overhead.
PR-URL: https://github.com/nodejs/node/pull/50899
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
https://github.com/nodejs/node/pull/45888 took the environment
creation code out of the scope covered by the v8::TryCatch
that we use to print early failures during environment creation.
So e.g. when adding something that would fail in node.js, we get
```
node:internal/options:554: Uncaught Error: Should not query options before bootstrapping is done
```
This patch restores that by adding another v8::TryCatch for it:
```
node:internal/options:20
({ options: optionsMap } = getCLIOptions());
^
Error: Should not query options before bootstrapping is done
at getCLIOptionsFromBinding (node:internal/options:20:32)
at getOptionValue (node:internal/options:45:19)
at node:internal/bootstrap/node:433:29
```
PR-URL: https://github.com/nodejs/node/pull/46533
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Add experimental support for loading snapshots in the embedder API
by adding a public opaque wrapper for our `SnapshotData` struct and
allowing embedders to pass it to the relevant setup functions.
Where applicable, use these helpers to deduplicate existing code
in Node.js’s startup path.
This has shown a 40 % startup performance increase for a real-world
application, even with the somewhat limited current support for
built-in modules.
The documentation includes a note about no guarantees for API or
ABI stability for this feature while it is experimental.
PR-URL: https://github.com/nodejs/node/pull/45888
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
daae938f32 broke addons which create their own `Isolate`
instances, because enabling the shared-readonly-heap feature
of V8 requires all snapshots used for different `Isolate`s to
be identical. Usage of addons that do this has probably
decreased quite a bit since Worker threads were introduced
in Node.js, but it’s still a valid use case, and in any case
the breakage was probably not intentional (although the referenced
commit did require test changes because of this issue).
This commit addresses this issue partially by caching the
V8 snapshot parameters and ignoring ones passed in from users
in `NewIsolate()` when this feature is enabled, and makes
the `NodeMainInstance` snapshot-based isolate creation
also re-use this code.
PR-URL: https://github.com/nodejs/node/pull/45885
Reviewed-By: James M Snell <jasnell@gmail.com>
Add an ExitCode enum class and use it throughout the code base
instead of hard-coding the exit codes everywhere. At the moment,
the exit codes used in many places do not actually conform to
what the documentation describes. With the new enums (which
are also available to the JS land as constants in an internal
binding) we could migrate to a more consistent usage of the
codes, and eventually expose the constants to the user land
when they are stable enough.
PR-URL: https://github.com/nodejs/node/pull/44746
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Access to the global object from within a vm context is intercepted
so it's slow, therefore we should try to avoid unnecessary access
to it during the initialization of vm contexts.
- Remove the Atomics.wake deletion as V8 now does not install it
anymore.
- Move the Intl.v8BreakIterator deletion into the snapshot.
- Do not query the Object prototype if --disable-proto is not set.
This should speed up the creation of vm contexts by about ~12%.
PR-URL: https://github.com/nodejs/node/pull/44252
Refs: https://github.com/nodejs/node/issues/44014
Refs: https://github.com/nodejs/node/issues/37476
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This patch stores the metadata about the Node.js binary
into the SnapshotData and adds fields denoting how the
snapshot was generated, on what platform it was
generated as well as the V8 cached data version flag.
Instead of simply crashing when the metadata doesn't
match, Node.js now prints an error message and exit with
1 for the customized snapshot, or ignore the snapshot
and start from scratch if it's the default one.
PR-URL: https://github.com/nodejs/node/pull/44132
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
The term "native module" dates back to some of the oldest code
in the code base. Within the context of Node.js core it usually
refers to modules that are native to Node.js (e.g. fs, http),
but it can cause confusion for people who don't work on this
part of the code base, as "native module" can also refer to
native addons - which is even the case in some of the API
docs and error messages.
This patch tries to make the usage of these terms more consistent.
Now within the context of Node.js core:
- JavaScript scripts that are built-in to Node.js are now referred
to as "built-in(s)". If they are available as modules,
they can also be referred to as "built-in module(s)".
- Dynamically-linked shared objects that are loaded into
the Node.js processes are referred to as "addons".
We will try to avoid using the term "native modules" because it could
be ambiguous.
Changes in this patch:
File names:
- node_native_module.h -> node_builtins.h,
- node_native_module.cc -> node_builtins.cc
C++ binding names:
- `native_module` -> `builtins`
`node::Environment`:
- `native_modules_without_cache` -> `builtins_without_cache`
- `native_modules_with_cache` -> `builtins_with_cache`
- `native_modules_in_snapshot` -> `builtins_in_cache`
- `native_module_require` -> `builtin_module_require`
`node::EnvSerializeInfo`:
- `native_modules` -> `builtins
`node::native_module::NativeModuleLoader`:
- `native_module` namespace -> `builtins` namespace
- `NativeModuleLoader` -> `BuiltinLoader`
- `NativeModuleRecordMap` -> `BuiltinSourceMap`
- `NativeModuleCacheMap` -> `BuiltinCodeCacheMap`
- `ModuleIds` -> `BuiltinIds`
- `ModuleCategories` -> `BuiltinCategories`
- `LoadBuiltinModuleSource` -> `LoadBuiltinSource`
`loader.js`:
- `NativeModule` -> `BuiltinModule` (the `NativeModule` name used in
`process.moduleLoadList` is kept for compatibility)
And other clarifications in the documentation and comments.
PR-URL: https://github.com/nodejs/node/pull/44135
Fixes: https://github.com/nodejs/node/issues/44036
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
So far, process initialization has been a bit all over the place
in Node.js. `InitializeNodeWithArgs()` is our main public API
for this, but inclusion of items in it vs. `InitializeOncePerProcess()`
and `PlatformInit()` has been random at best. Likewise,
some pieces of initialization have been guarded by
`NODE_SHARED_MODE`, but also fairly randomly and without
any meaningful connection to shared library usage.
This leaves embedders in a position to cherry-pick some of
the initialization code into their own code to make their
application behave like typical Node.js applications to the
degree to which they desire it.
Electron takes an alternative route and makes direct use of
`InitializeOncePerProcess()` already while it is a private
API, with a `TODO` to add it to the public API in Node.js.
This commit addresses that `TODO`, and `TODO`s around the
`NODE_SHARED_MODE` usage. Specifically:
- `InitializeOncePerProcess()` and `TearDownOncePerProcess()`
are added to the public API.
- The `flags` option of these functions are merged with the
`flags` option for `InitializeNodeWithArgs()`, since they
essentially share the same semantics.
- The return value of the function is made an abstract class,
rather than a struct, for easier API/ABI stability.
- Initialization code from `main()` is brought into these
functions (since that makes sense in general).
- Add a `TODO` for turning `InitializeNodeWithArgs()` into
a small wrapper around `InitializeOncePerProcess()` and
eventually removing it (at least one major release cycle
each, presumably).
- Remove `NODE_SHARED_MODE` guards and replace them with
runtime options.
PR-URL: https://github.com/nodejs/node/pull/44121
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
This patch:
- Make NativeModuleLoader::LookupAndCompile() detect parameters based
on module IDs. This allows us to compile more builtins when
generating the embedded bootstrap, including
- internal/per_context/*
- internal/bootstrap/*
- internal/main/*
- Move pre_execution.js to lib/internal/process as it needs to be
compiled as a regular built-in module, unlike other scripts
in lib/internal/bootstrap
- Move markBootstrapComplete() to the performance binding instead of
making it a function-wrapper-based global to reduce number of
special cases.
PR-URL: https://github.com/nodejs/node/pull/44018
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
This patch introduces `--build-snapshot` and `--snapshot-blob` options
for creating and using user land snapshots.
For the initial iteration, user land CJS modules and ESM are not yet
supported in the snapshot, so only one single file can be snapshotted
(users can bundle their applications into a single script with their
bundler of choice to build a snapshot though).
A subset of builtins should already work, and support for more builtins
are being added. This PR includes tests checking that the TypeScript
compiler and the marked markdown renderer (and the builtins they use)
can be snapshotted and deserialized.
To generate a snapshot using `snapshot.js` as entry point and write the
snapshot blob to `snapshot.blob`:
```
$ echo "globalThis.foo = 'I am from the snapshot'" > snapshot.js
$ node --snapshot-blob snapshot.blob --build-snapshot snapshot.js
```
To restore application state from `snapshot.blob`, with `index.js` as
the entry point script for the deserialized application:
```
$ echo "console.log(globalThis.foo)" > index.js
$ node --snapshot-blob snapshot.blob index.js
I am from the snapshot
```
Users can also use the `v8.startupSnapshot` API to specify an entry
point at snapshot building time, thus avoiding the need of an additional
entry script at deserialization time:
```
$ echo "require('v8').startupSnapshot.setDeserializeMainFunction(() => console.log('I am from the snapshot'))" > snapshot.js
$ node --snapshot-blob snapshot.blob --build-snapshot snapshot.js
$ node --snapshot-blob snapshot.blob
I am from the snapshot
```
Note that this patch only adds functionality to the `node` executable
for building run-time user-land snapshots, the generated snapshot is
stored into a separate file on disk. Building a single binary with both
Node.js and an embedded snapshot has already been possible with the
`--node-snapshot-main` option to the `configure` script if the user
compiles Node.js from source. It would be a different task to enable the
`node` executable to produce a single binary that contains both Node.js
and an embedded snapshot without building Node.js from source, which
should be layered on top of the SEA (Single Executable Apps) initiative.
Known limitations/bugs that are being fixed in the upstream:
- V8 hits a DCHECK when deserializing certain mutated globals, e.g.
`Error.stackTraceLimit` (it should work fine in the release build,
however): https://chromium-review.googlesource.com/c/v8/v8/+/3319481
- Layout of V8's read-only heap can be inconsistent after
deserialization, resulting in memory corruption:
https://bugs.chromium.org/p/v8/issues/detail?id=12921
PR-URL: https://github.com/nodejs/node/pull/38905
Refs: https://github.com/nodejs/node/issues/35711
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
According to the chrome trace event format document, works that
are performed on one single thread should be traced with sync
duration events. In this way, these events can be grouped under
one thread and the trace event viewer can estimate the CPU usage
of that thread.
PR-URL: https://github.com/nodejs/node/pull/42977
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Node.js unofficially supports a shared library variant where the
main node executable is a thin wrapper around node.dll/libnode.so.
The key benefit of this is to support embedding Node.js in other
applications.
Since Node.js 12 there have been a number of issues preventing the
shared library build from working correctly, primarily on Windows:
* A number of functions used executables such as `mksnapshot` are
not exported from `libnode.dll` using a `NODE_EXTERN` attribute
* A dependency on the `Winmm` system library is missing
* Incorrect defines on executable targets leads to `node.exe`
claiming to export a number of functions that are actually in
`libnode.dll`
* Because `node.exe` attempts to export symbols, `node.lib` gets
generated causing native extensions to try to link against
`node.exe` not `libnode.dll`.
* Similarly, because `node.dll` was renamed to `libnode.dll`,
native extensions don't know to look for `libnode.lib` rather
than `node.lib`.
* On macOS an RPATH is added to find `libnode.dylib` relative to
`node` in the same folder. This works fine from the
`out/Release` folder but not from an installed prefix, where
`node` will be in `bin/` and `libnode.dylib` will be in `lib/`.
* Similarly on Linux, no RPATH is added so LD_LIBRARY_PATH needs
setting correctly for `bin/node` to find `lib/libnode.so`.
For the `libnode.lib` vs `node.lib` issue there are two possible
options:
1. Ensure `node.lib` from `node.exe` does not get generated, and
instead copy `libnode.lib` to `node.lib`. This means addons
compiled when referencing the correct `node.lib` file will
correctly depend on `libnode.dll`. The down side is that
native addons compiled with stock Node.js will still try to
resolve symbols against node.exe rather than libnode.dll.
2. After building `libnode.dll`, dump the exports using `dumpbin`,
and process this to generate a `node.def` file to be linked into
`node.exe` with the `/DEF:node.def` flag. The export entries
in `node.def` will all read
```
my_symbol=libnode.my_symbol
```
so that `node.exe` will redirect all exported symbols back to
`libnode.dll`. This has the benefit that addons compiled with
stock Node.js will load correctly into `node.exe` from a shared
library build, but means that every embedding executable also
needs to perform this same trick.
I went with the first option as it is the cleaner of the two
solutions in my opinion. Projects wishing to generate a shared
library variant of Node.js can now, for example,
```
.\vcbuild dll package vs
```
to generate a full node installation including `libnode.dll`,
`Release\node.lib`, and all the necessary headers. Native addons
can then be built against the shared library build easily by
specifying the correct `nodedir` option.
For example
```
>npx node-gyp configure --nodedir
C:\Users\User\node\Release\node-v18.0.0-win-x64
...
>npx node-gyp build
...
>dumpbin /dependents build\Release\binding.node
Microsoft (R) COFF/PE Dumper Version 14.29.30136.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file build\Release\binding.node
File Type: DLL
Image has the following dependencies:
KERNEL32.dll
libnode.dll
VCRUNTIME140.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
...
```
PR-URL: https://github.com/nodejs/node/pull/41850
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
This change makes the error codes returned by v8.writeHeapSnapshot()
consistent across all platforms by using the libuv APIs instead of
fopen(), fwrite() and fclose(). This also starts reporting potential
errors that might happen during the write operations.
Signed-off-by: Darshan Sen <raisinten@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/42577
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
There are situations where one wants to invoke a JS callback's ->Call()
from C++ and in particular retain any existing async_context state, but
where it's not obvious that a plain ->Call() would be safe at the point
in question.
Such callsites usually resort to
node::MakeCallback(..., async_context{0, 0}), which unconditionally
pushes the async_context{0, 0} and takes the required provisions for the
->Call() itself such as triggering the tick after its return, if needed.
An example would be the PerformanceObserver invocation from
PerformanceEntry::Notify(): this can get called when coming from JS
through e.g. perf_hooks.performance.mark() and alike, but perhaps also
from nghttp2 (c.f. EmitStatistics() in node_http2.cc).
In the former case, a plain ->Call() would be safe and it would be
desirable to retain the current async_context so that
PerformanceObservers can access it resp. the associated
AsyncLocalStorage. However, in the second case the additional provisions
taken by node::MakeCallback() might potentially be strictly required.
So PerformanceEntry::Notify() bites the bullet and invokes the
PerformanceObservers through node::MakeCallback() unconditionally,
thereby always rendering any possibly preexisting async_context
inaccessible.
Introduce the convenience node::MakeSyncCallback() for such usecases,
which would basically forward to ->Call() if safe and to
node::MakeCallback(..., async_context{0, 0}) otherwise.
Co-Authored-By: ZauberNerd <zaubernerd@zaubernerd.de>
PR-URL: https://github.com/nodejs/node/pull/36343
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Rich Trott <rtrott@gmail.com>
I added it in commit 57231d5286 ("src: notify V8 profiler when we're
idle") from October 2013 as a stop-gap measure to measure CPU time
rather than wall clock time, otherwise processes that spend a lot
of time sleeping in system calls give a false impression of being
very busy.
That fix is not without drawbacks because the idle flag is set before
libuv makes I/O callbacks and cleared again after. I/O callbacks can
result into calls into JS code and executing JS code is as non-idle
as you can get.
In commit 96ffcb9a21 ("src: reduce cpu profiler overhead") from January
2015, I made Node.js block off the SIGPROF signal that V8's CPU profiler
uses before Node.js goes to sleep. The goal of that commit is to reduce
the overhead from EINTR system call wakeups but it also has the pleasant
side effect of fixing what the idle notifier tried to fix.
This commit removes the idle notifier and turns the JS process object
methods into no-ops.
Fixes: https://github.com/nodejs/node/issues/19009
Refs: https://github.com/nodejs/node/pull/33138
PR-URL: https://github.com/nodejs/node/pull/34010
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Joyee Cheung