This patch speeds up the startup time and reduce the startup memory
footprint by using V8 code cache when comiling builtin modules.
The current approach is demonstrated in the `with-code-cache`
Makefile target (no corresponding Windows target at the moment).
1. Build the binary normally (`src/node_code_cache_stub.cc` is used),
by now `internalBinding('code_cache')` is an empty object
2. Run `tools/generate_code_cache.js` with the binary, which generates
the code caches by reading source code of builtin modules off source
code exposed by `require('internal/bootstrap/cache').builtinSource`
and then generate a C++ file containing static char arrays of the
code cache, using a format similar to `node_javascript.cc`
3. Run `configure` with the `--code-cache-path` option so that
the newly generated C++ file will be used when compiling the
new binary. The generated C++ file will put the cache into
the `internalBinding('code_cache')` object with the module
ids as keys
4. The new binary tries to read the code cache from
`internalBinding('code_cache')` and use it to compile
builtin modules. If the cache is used, it will put the id
into `require('internal/bootstrap/cache').compiledWithCache`
for bookkeeping, otherwise the id will be pushed into
`require('internal/bootstrap/cache').compiledWithoutCache`
This patch also added tests that verify the code cache is
generated and used when compiling builtin modules.
The binary with code cache:
- Is ~1MB bigger than the binary without code cahe
- Consumes ~1MB less memory during start up
- Starts up about 60% faster
PR-URL: https://github.com/nodejs/node/pull/21405
Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Gus Caplan <me@gus.host>
Make test runner capable of skipping tests, which makes it possible
to skip the failing test/message/core_line_numbers.js test.
Make nyc no longer generate compact instrumentation (this causes
significantly different code output, which leads to failing test
assertions).
PR-URL: https://github.com/nodejs/node/pull/20794
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Include the exit code in test failures. This will give us more
information during the currently-puzzling failures that provide no
information in CI such as:
```
03:10:10 not ok 563 parallel/test-fs-truncate
03:10:10 ---
03:10:10 duration_ms: 1.119
03:10:10 severity: fail
03:10:10 stack: |-
03:10:10 ...
```
PR-URL: https://github.com/nodejs/node/pull/19855
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
common.PORT should not be used in parallelized tests. (There can be a
port collision if another tests requests an arbitrary open port from the
operating system and ends up getting common.PORT before a test that uses
common.PORT uses the port.) In such a situation, throw an error.
PR-URL: https://github.com/nodejs/node/pull/17559
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Adds a way to mark a specified test as 'flaky' on fips compliant
systems.
Earlier, the ``tools/test.py`` script supported only 'mode',
'system' and 'arch' for test environment specification. This limits the
ability to specify the behavior of tests and setting pre-determined
behavior of the same on other types of systems. As an example, the
feature request below indicates the need to specify certain tests as
'flaky' on fips compliant systems. It hints at future possibility of a
shared library, which in turn may need a specifier for running tests.
This commit introduces a new item in the ``env`` dict, called ``type``
which defaults to ``simple`` type. It also adds an optional command
line argument ``--type``, which inputs strings. Current functionality
extends to setting ``simple`` or ``fips`` for this ``type`` variable.
However, extending it to further uses is rather simple by adding "if"
conditions at appropriate places in the ``tools/test.py`` script.
PR-URL: https://github.com/nodejs/node/pull/16329
Fixes: https://github.com/nodejs/node/issues/14746
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
`test-make-doc` fails in CI on Raspberry Pi devices where `test-ci-js`
runs the test but does not build the docs. We do not want to build the
docs in these cases. Move the test to the `doctool` suite and add that
suite to `IGNORED_SUITES` in `test.py`. Specify `doctool` as a test
suite to run with `make test` or `make test-ci` but not with the
`make test-ci-js` job run on cross-compiled fanned CI tests like we do
with Raspberry Pi devices in CI.
PR-URL: https://github.com/nodejs/node/pull/16301
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
You can now run suites for subsystem using shorthand, e.g., http.
Switch to black-list of default test folders from white-list.
Tests run by 'make test', 'make coverage', etc., now configurable.
Stop running known_issues suite when collecting test coverage.
PR-URL: https://github.com/nodejs/node/pull/15450
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Async wrap providers tested:
- crypto.randomBytes
- crypto.pbkdf2
- fs event wrap
- fsreqwrap access
- fsreqwrap readFile
- getaddrinforeq wrap
- getnameinforeq wrap
- pipe connect wrap
- query wrap
- pipewrap
- processwrap
- shutdown wrap
- tcpwrap
- udpwrap
- send wrap
- detailed signal wrap
- statwatcher
- timerwrap via setTimeout
- timerwrap via setInterval
- for Immediate
- http parser request
- http parser response
- connection via ssl server
- tls wrap
- write wrap
- ttywrap via readstream
- ttywrap via wriream
- zctx via zlib binding deflate
Embedder API:
- async-event tests
- one test looks at the happy paths
- another ensures that in cases of events emitted in an order that
doesn't make sense, the order is enforced by async hooks throwing a
meaningful error
- embedder enforcement tests are split up since async hook stack
corruption now the process
- therefore we launch a child and check for error output of the offending code
Additional tests:
- tests that show that we can enable/disable hooks inside their lifetime
events
- tests that verify the graph of resources triggering the creation of
other resources
Test Helpers:
- init-hooks:
- returns one collector instance
- when created an async hook is created and the lifetime events are
registered to call the appropriate collector functions
- the collector also exposes `enable` and `disable` functions which call
through to the async hook
- hook checks:
- checks invocations of life time hooks against the actual invocations
that were collected
- in some cases like `destroy` a min/max range of invocations can be
supplied since in these cases the exact number is non-deterministic
- verify graph:
- verifies the triggerIds of specific async resources are as expected,
i.e. the creation of resources was triggered by the resource we expect
- includes a printGraph function to generate easily readable test
input for verify graph
- both functions prune TickObjects to create less brittle and easier
to understand tests
PR-URL: https://github.com/nodejs/node/pull/12892
Ref: https://github.com/nodejs/node/pull/11883
Ref: https://github.com/nodejs/node/pull/8531
Reviewed-By: Andreas Madsen <amwebdk@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Sam Roberts <vieuxtech@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Add support for abi stable module API (N-API) as "Experimental feature".
The goal of this API is to provide a stable Node API for native
module developers. N-API aims to provide ABI compatibility guarantees
across different Node versions and also across different
Node VMs - allowing N-API enabled native modules to just work
across different versions and flavors of Node.js without recompilation.
A more detailed introduction is provided in:
https://github.com/nodejs/node-eps/blob/master/005-ABI-Stable-Module-API.md
and https://github.com/nodejs/abi-stable-node/blob/doc/VM%20Summit.pdf.
The feature, during its experimental state, will be guarded by a runtime
flag "--napi-modules". Only when this flag is added to the command line
will N-API modules along with regular non N-API modules be supported.
The API is defined by the methods in "src/node_api.h" and
"src/node_api_types.h". This is the best
starting point to review the API surface. More documentation will follow.
In addition to the implementation of the API using V8, which is included
in this PR, the API has also been validated against chakracore and that
port is available in
https://github.com/nodejs/abi-stable-node/tree/api-prototype-chakracore-8.x.
The current plan is to provide N-API support in versions 8.X and 6.X
directly. For older versions, such as 4.X or pre N-API versions of 6.X,
we plan to create an external npm module to provide a migration path
that will allow modules targeting older Node.js versions to use the API,
albeit without getting the advantage of not having to recompile.
In addition, we also plan an external npm package with C++ sugar to
simplify the use of the API. The sugar will be in-line only and will
only use the exported N-API methods but is not part of the N-API
itself. The current version is in:
https://github.com/nodejs/node-api.
This PR is a result of work in the abi-stable-node repo:
https://github.com/nodejs/abi-stable-node/tree/doc,
with this PR being the cumulative work on the api-prototype-8.x
branch with the following contributors in alphabetical order:
Author: Arunesh Chandra <arunesh.chandra@microsoft.com>
Author: Gabriel Schulhof <gabriel.schulhof@intel.com>
Author: Hitesh Kanwathirtha <hiteshk@microsoft.com>
Author: Ian Halliday <ianhall@microsoft.com>
Author: Jason Ginchereau <jasongin@microsoft.com>
Author: Michael Dawson <michael_dawson@ca.ibm.com>
Author: Sampson Gao <sampsong@ca.ibm.com>
Author: Taylor Woll <taylor.woll@microsoft.com>
PR-URL: https://github.com/nodejs/node/pull/11975
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Currently when node is build --without-ssl and the test are run,
there are a number of failing test due to tests expecting crypto
support to be available. This commit fixes fixes the failure and
instead skips the tests that expect crypto to be available.
PR-URL: https://github.com/nodejs/node/pull/11631
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Unset NODE_PATH environment variable when running tests.
On Ubuntu 16.04, some users experience test failures due to internal
libraries being installed in /usr/lib/nodejs/internal and NODE_PATH
including /usr/lib/nodejs. Tests that expect internal libraries to be
off limits without the --expose-internals flag will fail in this
situation. Currently, those tests are test/parallel/test-repl.js and
test/parallel/test-internal-modules.js.
This situation seems to (probably) be caused by some
not-entirely-uncommon package that gets installed.
Regardless, tests should ignore the user's NODE_PATH. (NODE_PATH is
tested in test/parallel/test-module-globalpaths-nodepath.js and
test/parallel/test-require-dot.js.)
PR-URL: https://github.com/nodejs/node/pull/11612
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Bryan English <bryan@bryanenglish.com>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
Refs: https://twitter.com/trott/status/835729396900061184
https://github.com/nodejs/node/pull/11086 had introduced a regression
that broke command line options processing for tools/test.py.
Basically, it made tools/test.py discard the command line argument that
would be passed after `--abort-on-timeout`. For instance, when running:
```
$ python tools/test.py --abort-on-timeout path/to/some-test
```
all tests would be run because the last command line argument
(`/path/to/some-test`) would be discarded.
This change fixes this regression.
Refs: https://github.com/nodejs/node/pull/11086
PR-URL: https://github.com/nodejs/node/pull/11153
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Currently, when a process times out, it is terminated by sending it the
SIGTERM signal. Sending SIGBART instead allows the operating system to
generate a core file that can be investigated later using post-mortem
debuggers such as llnode or mdb_v8.
This can be very useful when investigating flaky tests that time out,
since in that case the failure is difficult to reproduce, and being able
to look at a core file makes a big difference.
With these changes, passing the --abort-on-timeout command line option
to tools/test.py now sends SIGABRT to processes timing out on all
platforms but Windows.
PR-URL: https://github.com/nodejs/node/pull/11086
Ref: https://github.com/nodejs/node/issues/11026
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
This pr adds the ability to run `tools/test.py --path`. This means that
instead of defaulting to out/Release we can use the node version from
the path.
PR-URL: https://github.com/nodejs/node/pull/9674
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
The repeat option in test.py did not work as expected if `-j` was set to
more than one. Repeated tests running at the same time could share temp
directories and cause test failures. This was observed with:
tools/test.py -J --repeat=10 parallel/test-fs-watch-recursive
By using copy.deepCopy(), the repeated tests are separate objects and
not references to the same objects. Setting `thread_id` on one of them
will now not change the `thread_id` on all of them. And `thread_id` is
how the temp directory (and common.PORT as well) are determined.
Refs: https://github.com/nodejs/node/pull/9228
PR-URL: https://github.com/nodejs/node/pull/9249
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
Commit 084b2ec ("test: include component in tap output") introduced
an in hindsight glaringly obvious but fortunately not very critical
Windows-specific bug by failing to take the path separator into account.
This commit rectifies that, the prefix is now correctly stripped.
PR-URL: https://github.com/nodejs/node/pull/6915
Refs: https://github.com/nodejs/node/pull/6653
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Joyee Cheung