This brings back several improvements that were reverted by mistake when
landing https://redirect.github.com/nodejs/node/pull/64154
- Update the documentation about how the removal of side effects of
source collection
- Add non-enumerable `requireStack` and `topLevelAwaitLocations`
properties to `ERR_REQUIRE_ASYNC_MODULE`, the latter is only
populated when --experimental-print-required-tla is enabled
- Add "Required module: <url>" to the error message to identify the
required ESM entry point regardless of whether the flag is enabled
- Fix TLA caret column from 0-based to 1-based
- Store module source via a private symbol instead of a public property
- Use `hasAsyncGraph` (post-instantiation) in `throwIfAsyncGraph`
instead of walking the graph before instantiation
- Merge the require stack checking into the
`common.expectRequiredTLAError` helper.
- Removed tests that are made redundant by the snapshot tests
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/64260
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously in order to collect the locations of the TLA, we wait
until right before evalutation to ensure instantiation is
completed so that we can use
v8::Module::GetStalledTopLevelAwaitMessages(). Now we try to
add an additioanl shortcut to the source code in the module wraps
instead during compilation for modules that contain TLAs and
use acron to locate the TLAs when we need to throw
ERR_REQUIRE_AYNSC_MODULE, so we can do this as early as before
instantiation and do not need to run the module again to collect
the locations.
In addition, we now collect the require stack for
ERR_REQUIRE_ASYNC_MODULE too for better metadata in the errors.
Signed-off-by: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/64154
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This adds a embedder_module_hdo for identifying embedder-run
modules in the dynamic import handler and import.meta initializer,
and a SourceTextModuleTypes for customizing source text module
compilation in the JS land via compileSourceTextModule().
Also, refactors the existing embedder module compilation code
to reuse the builtin resolution logic.
PR-URL: https://github.com/nodejs/node/pull/61654
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Aditi Singh <aditisingh1400@gmail.com>
When the source text module is compiled without custom callbacks,
instead of calling into JS land from the per-isolate import.meta
initializer and then back to C++ land to set up lazy data
properties, just do the initialization all in C++ land.
Only import.meta.resolve initialization will call back
into JS land to generate a closure that call the cascaded loader
for resolution.
In addition, simplify the loader structure by merging
allowImportMetaResolve into isForAsyncLoaderHookWorker - the two
are essentially equivalent, as import.meta.resolve is only
allowed in a non-loader-hook worker thread's loader.
PR-URL: https://github.com/nodejs/node/pull/60603
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
ESM resolution and loading is now always synchronous from a
non-loader-hook thread. If no asynchrnous loader hooks are
registered, the resolution/loading is entirely synchronous.
If asynchronous loader hooks are registered, these would be
synchronous on the non-loader-hook thread, and asynchronous
on the loader hook thread.
This avoids several races caused by async/sync loading sharing
the same cache. In particular, asynchronous loader hooks
now works with `require(esm)` - previously it tends to break
due to races.
In addition, when an asynchronous loader hook
returns a promise that never settles, the main thread no longer
silently exits with exit code 13, leaving the code below
any module loading calls silently ignored without being executed.
Instead, it now throws ERR_ASYNC_LOADER_REQUEST_NEVER_SETTLED
which can be caught and handled by the main thread. If the module
request comes from `import()`, the never-settling promise is
now relayed to the result returned by `import()`.
Drive-by: when annotating the error about importing undetectable
named exports from CommonJS, it now no longer reload the source
code of the CommonJS module, and instead reuses format information
cached when the module was loaded for linking.
PR-URL: https://github.com/nodejs/node/pull/60380
Fixes: https://github.com/nodejs/node/issues/59666
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
- This updates the comments that assume loader hooks must be async
- Differentiate the sync/async loader hook paths in naming
`#customizations` is now `#asyncLoaderHooks` to make it clear
it's from the async APIs.
- Differentiate the paths running on the loader hook thread
(affects the loading of async other loader hooks and are async)
v.s. paths on the main thread calling out to code on the loader
hook thread (do not handle loading of other async loader hooks, and
can be sync by blocking).
- `Hooks` is now `AsyncLoaderHooksOnLoaderHookWorker`
- `CustomizedModuleLoader` is now
`AsyncLoaderHooksProxiedToLoaderHookWorker` and moved into
`lib/internal/modules/esm/hooks.js` as it implements the same
interface as `AsyncLoaderHooksOnLoaderHookWorker`
- `HooksProxy` is now `AsyncLoaderHookWorker`
- Adjust the JSDoc accordingly
- Clarify the "loader worker" as the "async loader hook worker"
i.e. when there's no _async_ loader hook registered, there won't
be this worker, to avoid the misconception that this worker
is spawned unconditionally.
- The code run on the loader hook worker to process
`--experimental-loader` is moved into
`lib/internal/modules/esm/worker.js` for clarity.
- The initialization configuration `forceDefaultLoader` is split
into `shouldSpawnLoaderHookWorker` and `shouldPreloadModules`
as those can be separate.
- `--experimental-vm-modules` is now processed during pre-execution
and no longer part of the initialization of the built-in ESM
loader, as it only exposes the vm APIs of ESM, and is unrelated
to built-in ESM loading.
PR-URL: https://github.com/nodejs/node/pull/60278
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This patch implements a "module-sync" exports condition
for packages to supply a sycnrhonous ES module to the
Node.js module loader, no matter it's being required
or imported. This is similar to the "module" condition
that bundlers have been using to support `require(esm)`
in Node.js, and allows dual-package authors to opt into
ESM-first only newer versions of Node.js that supports
require(esm) while avoiding the dual-package hazard.
```json
{
"type": "module",
"exports": {
"node": {
// On new version of Node.js, both require() and import get
// the ESM version
"module-sync": "./index.js",
// On older version of Node.js, where "module" and
// require(esm) are not supported, use the transpiled CJS version
// to avoid dual-package hazard. Library authors can decide
// to drop support for older versions of Node.js when they think
// it's time.
"default": "./dist/index.cjs"
},
// On any other environment, use the ESM version.
"default": "./index.js"
}
}
```
We end up implementing a condition with a different name
instead of reusing "module", because existing code in the
ecosystem using the "module" condition sometimes also expect
the module resolution for these ESM files to work in CJS
style, which is supported by bundlers, but the native
Node.js loader has intentionally made ESM resolution
different from CJS resolution (e.g. forbidding `import
'./noext'` or `import './directory'`), so it would be
semver-major to implement a `"module"` condition
without implementing the forbidden ESM resolution rules.
For now, this just implments a new condition as semver-minor
so it can be backported to older LTS.
Refs: https://webpack.js.org/guides/package-exports/#target-environment-independent-packages
PR-URL: https://github.com/nodejs/node/pull/54648
Fixes: https://github.com/nodejs/node/issues/52173
Refs: https://github.com/joyeecheung/test-module-condition
Refs: https://github.com/nodejs/node/issues/52697
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Unifies the CJS and ESM source map cache map with SourceMapCacheMap
and allows the CJS cache entries to be queried more efficiently with
a source url without iteration on an IterableWeakMap.
Add a test to verify that the CJS source map cache entry can be
reclaimed.
PR-URL: https://github.com/nodejs/node/pull/51711
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This refactors the code that compiles SourceTextModule for the
built-in ESM loader to use a common routine so that it's easier
to customize cache handling for the ESM loader. In addition
this introduces a common symbol for import.meta and import()
so that we don't need to create additional closures as handlers,
since we can get all the information we need from the V8 callback
already. This should reduce the memory footprint of ESM as well.
PR-URL: https://github.com/nodejs/node/pull/52291
Refs: https://github.com/nodejs/node/issues/47472
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Split the `internal/process/esm_loader` file which contains the
singleton cascaded loader:
- The the singleton cascaded loader now directly resides in
`internal/modules/esm/loader`, where the constructor also lives.
This file is the root of most circular dependency of ESM code,
(because components of the loader need the singleton itself),
so this makes the dependency more obvious. Added comments about
loading it lazily to avoid circular dependency.
- The getter to the cascaded loader is also turned into a method
to make the side effect explicit.
- The sequence of `loadESM()` and `handleMainPromise` is now merged
together into `runEntryPointWithESMLoader()` in
`internal/modules/run_main` because this is intended to run entry
points with the ESM loader and not just any module.
- Documents how top-level await is handled.
PR-URL: https://github.com/nodejs/node/pull/51999
Fixes: https://github.com/nodejs/node/issues/42868
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This patch adds support for using
`vm.constants.USE_MAIN_CONTEXT_DEFAULT_LOADER` as
`importModuleDynamically` in all APIs that take the option
except `vm.SourceTextModule`. This allows users to have a shortcut
to support dynamic import() in the compiled code without missing
the compilation cache if they don't need customization of the
loading process. We emit an experimental warning when the
`import()` is actually handled by the default loader through
this option instead of requiring `--experimental-vm-modules`.
In addition this refactors the documentation for
`importModuleDynamically` and adds a dedicated section for it
with examples.
`vm.SourceTextModule` is not supported in this patch because
it needs additional refactoring to handle `initializeImportMeta`,
which can be done in a follow-up.
PR-URL: https://github.com/nodejs/node/pull/51244
Fixes: https://github.com/nodejs/node/issues/51154
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This bootstraps ESM loaders in the ShadowRealm with
`ShadowRealm.prototype.importValue` as its entry point and enables
loading ESM and CJS modules in the ShadowRealm. The module is imported
without a parent URL and resolved with the current process's working
directory.
PR-URL: https://github.com/nodejs/node/pull/48655
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Users cannot access any API that can be used to return a module or
module namespace in this callback without --experimental-vm-modules
anyway, so this would eventually lead to a rejection. This patch
rejects in this case with our own error message and use a constant
host-defined option for the rejection, so that scripts with the
same source can still be compiled using the compilation cache
if no `import()` is actually called in the script.
PR-URL: https://github.com/nodejs/node/pull/50137
Refs: https://github.com/nodejs/node/issues/35375
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Previously when managing the importModuleDynamically callback of
vm.compileFunction(), we use an ID number as the host defined option
and maintain a per-Environment ID -> CompiledFnEntry map to retain
the top-level referrer function returned by vm.compileFunction() in
order to pass it back to the callback, but it would leak because with
how we used v8::Persistent to maintain this reference, V8 would not
be able to understand the cycle and would just think that the
CompiledFnEntry was supposed to live forever. We made an attempt
to make that reference known to V8 by making the CompiledFnEntry weak
and using a private symbol to make CompiledFnEntry strongly
references the top-level referrer function in
https://github.com/nodejs/node/pull/46785, but that turned out to be
unsound, because the there's no guarantee that the top-level function
must be alive while import() can still be initiated from that
function, since V8 could discard the top-level function and only keep
inner functions alive, so relying on the top-level function to keep
the CompiledFnEntry alive could result in use-after-free which caused
a revert of that fix.
With this patch we use a symbol in the host defined options instead of
a number, because with the stage-3 symbol-as-weakmap-keys proposal
we could directly use that symbol to keep the referrer alive using a
WeakMap. As a bonus this also keeps the other kinds of referrers
alive as long as import() can still be initiated from that
Script/Module, so this also fixes the long-standing crash caused by
vm.Script being GC'ed too early when its importModuleDynamically
callback still needs it.
PR-URL: https://github.com/nodejs/node/pull/48510
Refs: https://github.com/nodejs/node/issues/44211
Refs: https://github.com/nodejs/node/issues/42080
Refs: https://github.com/nodejs/node/issues/47096
Refs: https://github.com/nodejs/node/issues/43205
Refs: https://github.com/nodejs/node/issues/38695
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Follows @giltayar's proposed API:
> `register` can pass any data it wants to the loader, which will be
passed to the exported `initialize` function of the loader.
Additionally, if the user of `register` wants to communicate with the
loader, it can just create a `MessageChannel` and pass the port to the
loader as data.
The `register` API is now:
```ts
interface Options {
parentUrl?: string;
data?: any;
transferList?: any[];
}
function register(loader: string, parentUrl?: string): any;
function register(loader: string, options?: Options): any;
```
This API is backwards compatible with the old one (new arguments are
optional and at the end) and allows for passing data into the new
`initialize` hook. If this hook returns data it is passed back to
`register`:
```ts
function initialize(data: any): Promise<any>;
```
**NOTE**: Currently there is no mechanism for a loader to exchange
ownership of something back to the caller.
Refs: https://github.com/nodejs/loaders/issues/147
PR-URL: https://github.com/nodejs/node/pull/48842
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Major functional changes:
- Allow `import()` to work within loaders that require other loaders,
- Unflag the use of `Module.register`.
A new interface `Customizations` has been created in order to unify
`ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and
`CustomizedModuleLoader` all of which now implement it:
```ts
interface LoadResult {
format: ModuleFormat;
source: ModuleSource;
}
interface ResolveResult {
format: string;
url: URL['href'];
}
interface Customizations {
allowImportMetaResolve: boolean;
load(url: string, context: object): Promise<LoadResult>
resolve(
originalSpecifier:
string, parentURL: string,
importAssertions: Record<string, string>
): Promise<ResolveResult>
resolveSync(
originalSpecifier:
string, parentURL: string,
importAssertions: Record<string, string>
) ResolveResult;
register(specifier: string, parentUrl: string): any;
forceLoadHooks(): void;
importMetaInitialize(meta, context, loader): void;
}
```
The `ModuleLoader` class now has `setCustomizations` which takes an
object of this shape and delegates its responsibilities to this object
if present.
Note that two properties `allowImportMetaResolve` and `resolveSync`
exist now as a mechanism for `import.meta.resolve` – since `Hooks`
does not implement `resolveSync` other loaders cannot use
`import.meta.resolve`; `allowImportMetaResolve` is a way of checking
for that case instead of invoking `resolveSync` and erroring.
Fixes https://github.com/nodejs/node/issues/48515
Closes https://github.com/nodejs/node/pull/48439
PR-URL: https://github.com/nodejs/node/pull/48559
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This moves the following utils into modules/esm/utils.js:
- Code related to default conditions
- The callbackMap (which is now created in the module instead of
hanging off the module_wrap binding, since the C++ land
does not need it).
- Per-isolate module callbacks
These are self-contained code that can be included into the
built-in snapshot.
PR-URL: https://github.com/nodejs/node/pull/45849
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Joyee Cheung