Extend `requireNodeAddon(path)` to support more use-cases

[mani3xis]

Context

Currently, the requireNodeAddon() function requires only 1 string argument, which was expected to be a relative path to the compiled Node add-on. This worked great for the first few simple PoCs and demos, but it start showing its age and limitations already. In the meantime @kraenhansen in #12 added support for hashing the module paths to solve the issues of name clashes when modules are being packaged, which was later replaced by #32.

Problem statement

1. The current solution does not support addons, which are imported by specifying just the package name, whose package.json specifies the main field.
2. Leaking abstraction (exposes implementation details)

Proposed solution

Adding additional parameters to requireNodeAddon() that would enable the platform's add-on loader to resolve the path and try a few of them. My current suggestion is to have requireNodeAddon(requiredPath: string, targetPackageName: string, fromSubpath: string) where:

• requiredPath is (in most cases) the original relative path passed to the require() call. If this path was not relative (starting with . or ..) then it should be the package name which will be resolved by looking up the main field, making this argument a relative subpath again.
• packageName should be the name of the package that this add-on is expected to be in. Please note that such package name might be scoped, therefore - for simplicity - I've decided to keep it separated from the subpath.
• fromSubpath is the relative path (subpath) of the file that issued the call to the require() function. This parameter is needed to resolve the package-relative path where the add-on is supposed to be.

Example use case 1

Somebody in ./src/index.js within package @callstack/foo wrote:

const module = require('./build/Release/foo.node');

Our Babel plugin should replace it with following:

const module = requireNodeAddon('./build/Release/foo.node', '@callstack/foo', './src/index.js');

In this case, the native loader would be able to resolve the relative path by combining ./build/Release/foo.node on top of ./src/index.js, resulting in ./src/build/Release/foo.node - which should be added to the search paths.

Note

If the path would include the package name (especially when it's scoped), like @callstack/foo/src/build/Release/foo.node, then separating the package name from the subpath would be a little bit more convoluted (we would need to parse it). Passing those as separate arguments gives us those "for free".

Example use-case 2

Somebody in the package @callstack/foo set the main field in their package.json to ./foo.node. A different user in their package (say whatever) writes:

const { foofighter } = require('@callstack/foo');

then our Babel plugin should output:

const { foofighter } = requireNodeAddon('./foo.node', '@callstack/foo', './package.json');

Following the steps from the "algorithm" above, the relative path would be resolved to ./foo.node (it used the path from main and ./package.json gives ./), which will be expected to be in the @callstack/foo package. The exact value of the last argument is not that important here (can be ./ or . as well).

Example use-case 3

Somebody in ./src/wrapper.js that's part of package foo wrote:

const fs = require('node:fs');

Our Babel plugin is expected to output this instead:

const fs = requireNodeAddon('node:fs', 'foo', './src/wrapper.js');

More flexible loader

One of the main goals of this proposal would be to "hide" the resolution algorithm in native code. The JavaScript side should not care whether the names are hashed or changed in any other way. The closer the original Node.js code, the better. Those 3 separate arguments: requiredPath, targetPackageName and fromSubpath gives more flexibility to the native loader. If all the addons are flattened into a single directory (like for iOS), the JS does not care -- the native loader will be able to implement this logic and try different search paths and variants.

Lastly, it shouldn't be hard to adapt this approach to support ES modules that use import and export.

[mani3xis]

FYI: As I've extracted the require cache, resolver and platform-specific loading logic from the Turbo Module, in order to properly cache addons which were required from different directories (but still end up pointing at the same addon) I had to implement various path utility functions in C++. I've tested those (as we are dealing with user-defined input) for relative paths (some absolute paths should work, but I know not all of them). For everybody interested, I've temporarily uploaded the code to this gist.

[kraenhansen]

I had to implement various path utility functions in C++

I'm curious if we can't rely on https://en.cppreference.com/w/cpp/filesystem/path for some of that? (am I missing something obvious?)

[mani3xis]

That was my initial first idea, but the module paths are not the same, and have their own quirks. First of, the path delimiter should always be / (on Windows it's \\). Next, it should support URLs (Node.js supports data, http and file protocols, and they have their own prefix node). Moreover, you can have imported paths that start with # (theoretically they should be resolved by Babel). Finally, "bare specifiers" with scoped package names might not work, I'm afraid that std::filesystem::path might not accept @callstackincubator/react-native-node-api-modules/require, but I'd need to verify that.

[kraenhansen]

The fromSubpath is the relative path

of the file that issued the call to the require() function

But what is it relative to? The package root expected to contain the addon? What if the require call is in another package than the addon? Will that include ../node_modules/the-other/?

[kraenhansen]

Why would use-case 3 not have "node:fs" as it's second argument?

[kraenhansen]

I'm not sure I really understand the use-case of passing the fromSubpath ...
Couldn't we collapse these arguments into two? The package name and the relative to the addon within the package? I.e. what we currently refer to as the ModuleContext here:

https://github.com/callstackincubator/react-native-node-api-modules/blob/d789e8299d81eace4c7b2423c07332f4f388d5dd/packages/react-native-node-api-modules/src/node/path-utils.ts#L73-L104