Projects that seek to implement Node.js compatible APIs end up
needed to reproduce various bits of functionality internally in
order to faithfully replicate the Node.js behaviors. This is
particularly true for things like byte manipulation, base64 and
hex encoding, and other low-level operations. This change
proposes moving much of this low-level byte manipulation code
out of nodejs/src and into a new `nbytes` library. Initially this
new library will exist in the `deps` directory but the intent is
to spin out a new separate repository to be its home in the future.
Doing so will allow other projects to use the nbytes library with
exactly the same implementation as Node.js.
This commit moves only the byte swapping and legacy base64 handling
code. Additional commits will move additional byte manipulation
logic into the library.
PR-URL: https://github.com/nodejs/node/pull/53507
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This parameter was always being set to `nullptr` by its callers, either
explicitly or implicitly via the default argument. It was also buggy, as
in some cases it wouldn't be written to, potentially leaking stack
memory (see the early returns in `StringBytes::WriteUCS2`). Remove it
entirely.
PR-URL: https://github.com/nodejs/node/pull/44092
Reviewed-By: Feng Yu <F3n67u@outlook.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Currently, the following warnings are emitted during compilation:
In file included from ../src/string_bytes.h:29,
from ../src/string_bytes.cc:22:
../deps/v8/include/v8.h:
In instantiation of ‘void v8::PersistentBase<T>::SetWeak(
P*,typename v8::WeakCallbackInfo<P>::Callback, v8::WeakCallbackType)
[with P = node::BaseObject;
T = v8::Object;
typename v8::WeakCallbackInfo<P>::Callback =
void (*)(const v8::WeakCallbackInfo<node::BaseObject>&)]’:
../src/base_object-inl.h:135:42: required from here
../deps/v8/include/v8.h:10732:16: warning:
cast between incompatible function types from
‘v8::WeakCallbackInfo<node::BaseObject>::Callback’ {aka
‘void (*)(const v8::WeakCallbackInfo<node::BaseObject>&)’} to
‘Callback’ {aka ‘void (*)(const v8::WeakCallbackInfo<void>&)’}
[-Wcast-function-type]
10732 | reinterpret_cast<Callback>(callback), type);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
And the same from test/cctest/test_aliased_buffer.cc
We have an open pull request against v8 for this:
https://chromium-review.googlesource.com/c/v8/v8/+/2080361
PR-URL: https://github.com/nodejs/node/pull/32679
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Two things in one on this commit:
(a) For the QUIC implementation, we need to separate out various bits
from node_crypto.cc to allow them to be reused. That's where this
commit starts.
(b) Quite a bit of the node_crypto.cc code was just messy in terms of
it's organization and lack of error handling and use of Local vs.
MaybeLocal. This cleans that up a bit and hopefully makes certain
parts a bit more manageable also.
Signed-off-by: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/32016
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Sam Roberts <vieuxtech@gmail.com>
Node codebase has evolved a lot in the more than 10 years of its
existence. As more features (and code) have been added, changed,
removed, it's sometimes hard to keep track of what gets used and what
not.
This commits attempts to clean some of those potentially left-over
headers using suggestions from include-what-you-use
Refs: https://github.com/nodejs/node/issues/27531
PR-URL: https://github.com/nodejs/node/pull/30328
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Using a `uint16_t` sequence for UTF-16 processing would typically
imply that the sequence already contains the correct 16-bit
code units.
However, our API does not account for that. The previous comments
were somewhat misleading, since endianness is typically applied to
sequences of bytes, which is not something that this API works with.
PR-URL: https://github.com/nodejs/node/pull/22623
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
- Return `MaybeLocal`s from `StringBytes::Encode`
- Add an `error` out parameter to pass JS exceptions to the callers
(instead of directly throwing)
- Simplify some of the string generation methods in `string_bytes.cc`
by unifying the `EXTERN_APEX` logic
- Reduce usage of deprecated V8 APIs.
- Remove error handling logic from JS, the `buffer.*Slice()` methods
now throw errors themselves.
- Left TODO comments for future semver-major error message
improvements.
This paves the way for better error messages coming out of the
StringBytes methods.
Ref: https://github.com/nodejs/node/issues/3175
PR-URL: https://github.com/nodejs/node/pull/12765
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Unify the common code of `Utf8Value`, `TwoByteValue`, `BufferValue`
and `StringBytes::InlineDecoder` into one class. Always make the
result zero-terminated for the first three.
This fixes two problems in passing:
* When the conversion of the input value to String fails,
make the buffer zero-terminated anyway. Previously, this would
have resulted in possibly reading uninitialized data in multiple
places in the code. An instance of that problem can be reproduced
by running e.g.
`valgrind node -e 'net.isIP({ toString() { throw Error() } })'`.
* Previously, `BufferValue` copied one byte too much from the source,
possibly resulting in an out-of-bounds memory access.
This can be reproduced by running e.g.
`valgrind node -e \
'fs.openSync(Buffer.from("node".repeat(8192)), "r")'`.
Further minor changes:
* This lifts the `out()` method of `StringBytes::InlineDecoder`
to the common class so that it can be used when using the
overloaded `operator*` does not seem appropiate.
* Hopefully clearer variable names.
* Add checks to make sure the length of the data does not exceed
the allocated storage size, including the possible null terminator.
PR-URL: https://github.com/nodejs/node/pull/6357
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
This makes several changes:
1. Allow path/filename to be passed in as a Buffer on fs methods
2. Add `options.encoding` to fs.readdir, fs.readdirSync, fs.readlink,
fs.readlinkSync and fs.watch.
3. Documentation updates
For 1... it's now possible to do:
```js
fs.open(Buffer('/fs/foo/bar'), 'w+', (err, fd) => { });
```
For 2...
```js
fs.readdir('/fs/foo/bar', {encoding:'hex'}, (err,list) => { });
fs.readdir('/fs/foo/bar', {encoding:'buffer'}, (err, list) => { });
```
encoding can also be passed as a string
```js
fs.readdir('/fs/foo/bar', 'hex', (err,list) => { });
```
The default encoding is set to UTF8 so this addresses the
discrepency that existed previously between fs.readdir and
fs.watch handling filenames differently.
Fixes: https://github.com/nodejs/node/issues/2088
Refs: https://github.com/nodejs/node/issues/3519
PR-URL: https://github.com/nodejs/node/pull/5616
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Introduced in joyent/node v0.10 as a backwards compatibility measure.
It's an ugly hack and allowing invalid UTF-8 is not a good idea in the
first place, remove it.
PR-URL: https://github.com/iojs/io.js/pull/1042
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
The copyright and license notice is already in the LICENSE file. There
is no justifiable reason to also require that it be included in every
file, since the individual files are not individually distributed except
as part of the entire package.
Move the big endian to little endian conversion logic for UCS2 input
from src/string_bytes.cc to src/node_buffer.cc; StringSlice() is the
only function that actually needs it and with this commit, a second
copy is avoided on big endian architectures.
Introduce two-byte overloads of node::Encode() and StringBytes::Encode()
that ensure that the input is suitably aligned.
Revisits commit 535fec8 from yesterday.
Now that we are building with C++11 features enabled, replace use
of NULL with nullptr.
The benefit of using nullptr is that it can never be confused for
an integral type because it does not support implicit conversions
to integral types except boolean - unlike NULL, which is defined
as a literal `0`.
Previously v8's WriteUtf8 function would produce invalid utf-8 output
when encountering unmatched surrogate code units [1]. The new
REPLACE_INVALID_UTF8 option fixes that by replacing invalid code points
with the unicode replacement character.
[1]: JS Strings are defined as arrays of 16 bit unsigned integers. There
is no unicode enforcement, so one can easily end up with invalid unicode
code unit sequences inside a string.
Performs a quick, non-exhaustive check on the input string to see if
it's compatible with the specified string encoding.
Curently it only checks that hex strings have a length that is a
multiple of two.
It imports the definition into every source file that includes
string_bytes.h, as evidenced by the build suddenly breaking left
and right because of missing Handle/Local/String/Value imports.
Michaël Zasso