How to scientifically develop a Node addon

In my last article about NAPI, I introduced how I converted a C library into a node addon as a novice. This article mainly conducts systematic study and practice on node addon.

What are Addons

Simply put, node.js is written in C++, so you can increase the capabilities of node through dynamic libraries (dll dylib). (Students who do not understand dynamic libraries and static libraries can go back and learn basic C knowledge). A node addon is a file ending in .node that is eventually imported via require.

Addons?are dynamically-linked shared objects written in C++ . The?require() function can load addons as ordinary Node.js modules. Addons provide an interface between JavaScript and C/C++ libraries.

If you use a binary file viewer to view the file ending with .node, you will find its Magic Number The Magic Number of the official dynamic library.

Addons principle

I don’t understand. skip. One thing that might need attention is how a module is exported.

How to write Addons

Direct native

Directly use the provided native modules to develop header files (v8, uv, etc.). However, in different node versions, the V8 version will be upgraded, resulting in a piece of code only taking effect for one node version.

NAN (Native Abstractions for Node.js)

The macro of NAN will judge the Node.js version at the current compilation time, and develop different results according to different versions of Node.js. In essence, it is still necessary to compile each version of node, but there is no need to rewrite the code, because the differences are smoothed out by NAN.

But this problem is still very big, mainly because it has to be compiled under different node versions, such as this. Dozens of versions were pre-ordered.

That is to say, because the header files of each version basically will be different. As a result, basically each node version (and Electron version) has to rebuild a new .node file for use on each platform (windows mac).

Node-API

First, let’s introduce the concept of ABI (application binary interface). Friends with good English can directly read here.

To put it simply, ABI is a set of conventions. You need to organize your data structure according to this set of specifications when building, so that users can use this ABI to directly use you without worrying about problems or recompiling.

Node-related header files include node.h v8.h, etc., which are maintained by different organizations. Therefore, the ABI is unstable. In order to solve this problem, the node team encapsulated an additional node_api.h on top of it. Therefore, we do not need to precompile a large number of addons to solve the problems used in different environments.

Node-API versions are rare and guaranteed to be compatible, so the addon you built based on NAPI 4 must be able to support NAPI 4 and later nodes (including Electron) to use.

That is to say, basically we only need to pre-build the addon according to platform-architecture, such as darwin-x64 darwin-arm64 etc. can be used in Common under the corresponding architecture.

An uncertain knowledge

The node official website can query the NODE_MODULE_VERSION corresponding to each nodejs. (The missing version number is basically for Electron)

NODE_MODULE_VERSION? refers to the ABI (application binary interface) version number of Node.js, which is used to determine the version of the C++ library compiled for Node.js to determine whether it can be loaded directly without recompilation . It was stored as a one-digit hexadecimal value in earlier versions, but is now represented as an integer.

If a non-napi package is used during the development process, there is a high probability (especially for Electron development) that NODE_MODULE_VERSION does not match an error.

If I understand correctly, using a non-napi build will result in a build every NODE_MODULE_VERSION. This kind of error will not occur when the N-API is built. Because the N-API ABI corresponds to the node_api.h related header files, and N-API is guaranteed to be backward compatible (new nodejs can use the old N-API version) and there are only 8 versions so far. And NODE_MODULE_VERSION refers to the ABI of v8.h node.h and other related header files.

Addons technology stack selection

Because I am the front end, I have to use js only if I want to develop addon. In theory, addons can be developed only if the language can generate dynamic libraries. It mainly depends on the corresponding ecological construction. The addon will require two jobs:

  1. It is unreasonable to build locally on the package user side, and there is a high probability that the required environment is missing. So consider precompilation and find a way to let users get the precompiled results;
  2. Let the user obtain the corresponding precompiled package. For example, you can download the corresponding precompiled product according to the user platform during the install phase. (detailed later)

C++

Can be built using node-gyp or CMAKE (cmake.js).

At the same time, there are prebuild prebuild-install prebuildify node-pre-gyp and many other tools to assist you in precompiling and uploading.

But the C++ environment configuration, learning process, etc. are extremely painful (personal experience). Since the history of C ++ is too long, you will need to learn various additional knowledge (for example, now that ES6 is already comprehensive, do you dare to say that you don’t need to know ES5?).

Rust (highly recommended)

My personal experience is that the language is not a problem, but the configuration environment is the most painful.

Rust is very friendly in its infancy as a young enough language. The napi-rs development experience is very friendly, and no additional build-related tools are required, and the project template is also built for you.

In addition, napi-rs 3.0 is in the rfc, and it is expected to support the compilation of addon into wasm. This prospect is still good.

If you don’t want to use napi-rs, you can use neon. In comparison, neon is more like scaffolding, and it tries to integrate v8-related APIs. Napi-rs is lighter. (If it is open source, I feel that napi-rs will be better, and you can integrate Github Actions)

Other

  • golang napi-go (few stars).
  • C# napi-cs (few stars) ditto.

All that said, unless you have an existing and well-defined C/C++ library that doesn’t have a Rust counterpart, I recommend using Rust.

Addons Development

Rust

Environment preparation

The Rust environment can be installed as you like. Pull a template. Core is the napi-rs dependency.

Compile

npm run build

C++

Environment preparation

A C/C++ related environment needs to be installed. The project template can refer to the project in my previous article. The main dependency is node-addon-api , there are sufficient documents below to teach you how to pass parameters, callback, Promise, AsyncTask, etc.

Compile

Use cmake.js.

Best Practices

There are only two scenarios that I think are suitable for using addons:

  1. Existing C/C++/Rust packages, hoping to use them directly in node
  2. Need to complete an input and output simple. But the calculation process is a complex task.

I believe that many students use addon for performance improvement. It should be noted that the communication cost between js and addon is very high! For example the following scenario:

  • Any operation related to JS Object: passing in Object, modifying Object, returning Object, etc.;
  • Frequently operate JS callbacks;
  • I guess the best reference type;

When designing the addon’s API, we must always pay attention to switching between the two sides as little as possible. There are a few suggestions:

  • Use async. The code is essentially multi-threaded, without blocking the node main thread;
  • If there is a file operation, you can consider processing it directly in the addon and asynchronously;
  • If the computational complexity is not enough, it is better not to implement it with addon;

Possible optimization points to consider:

  • Using callback may not consume the same performance as promise, and callback can be converted into promise through new Promise(resolve => addonFn(resolve)) (napi-rs seems to be faster when using callback);

About asynchrony

Asynchronous, or multi-threaded, can greatly increase the performance of addons. Since the addon can directly use the node library, generally at least there is a way to achieve asynchrony through libuv.

For example, in Rust, two asynchronous methods are provided:

  • async fn uses tokio
  • async-task uses libuv

Addons release

C++

  • prebuild/prebuild-install: Execute downloading pre-built packages and use node-gyp to build when the package cannot be downloaded / there is no corresponding package
  • prebuidify/node-gyp-build: It is recommended to switch to this for prebuild, the principle should be the same as the following optionalDependecies

optionalDependencies

Use npm to judge its own environment and automatically download the package corresponding to the required architecture.

root directory package.json

{<!-- -->
  ...,
  "optionalDependencies": {<!-- -->
    "@napi-rs/canvas-win32-x64-msvc": "0.1.36",
    "@napi-rs/canvas-darwin-x64": "0.1.36",
    "@napi-rs/canvas-linux-x64-gnu": "0.1.36",
    "@napi-rs/canvas-linux-arm-gnueabihf": "0.1.36",
    "@napi-rs/canvas-linux-x64-musl": "0.1.36",
    "@napi-rs/canvas-linux-arm64-gnu": "0.1.36",
    "@napi-rs/canvas-linux-arm64-musl": "0.1.36",
    "@napi-rs/canvas-darwin-arm64": "0.1.36",
    "@napi-rs/canvas-android-arm64": "0.1.36"
  }
}

Pre-order build package package.json

{<!-- -->
  "name": "@napi-rs/canvas-darwin-x64",
  "version": "0.1.36",
  "os": [
    "darwin"
  ],
  "cpu": [
    "x64"
  ],
  ...
}

Due to the configuration of optionalDependencies npm will only download pre-built packages that match the current environment, while ignoring other packages.

But this will cause Electron to package the wrong pre-built package when building software for non-native platforms. For example, when building a Windows installation package on a Mac, the package corresponding to the Mac platform is still downloaded at this time.

In Rust, if you don’t want to manually create so many subpackages above, you can use @napi-rs/cli and refer to @napi-rs/canvas.

Summary

This article is an attempt to summarize from the perspective of system understanding, learning and practice after I have studied and practiced node addon. In the future, I should mainly use Rust for addon development. There may be many mistakes above, welcome to point out.

References

  • From Violence to NAN to N-API–The Change of Node.js Native Module Development Mode
  • C++ addons
  • C++ addons with Node-api