feat: virtual modules

[tjzel]

Summary

Changelog: [Feature] Virtual Modules handling - modules that are in-memory only.

PR based on rejected

• feat: future modules #1594

This PR adds an early implementation of virtual modules handling in Metro, modules which exist only in memory and not on the disk.

My motivation for implementing virtual modules is that react-native-worklets would like to emit extra, encapsulated modules, based on some code, during the Babel transformation of the user's source code.

Basic idea

A Babel plugin during transpilation of (physical or virtual) module can register virtual modules via APIs provided by Metro. Registration consists of providing the following data:

• An identifier of the module. This could be a path (that either exists or not) that uniquely identifies the virtual module.
• The source code of the module. The source code has to undergo the Metro transformation in the future, it's not ready to be bundled.
• The source maps of the module. I have yet to learn more on this topic - preferably the source maps would be inlined with the source code, but I don't know if it's possible with how Metro handles sourceURLs etc.

The registering module can either depend on the generated virtual module or not.
A module cannot depend on a virtual module it didn't generate - it might not be possible to synchronize the registrations in time. In the future this could be loosened.

Ideally the virtual module would land in the Metro's virtual FS and treated as a regular file for the purpose of getting its contents, calculating SHA etc. However I don't know yet how feasible that is since there's a great deal of asynchronicity between Metro workers and its core components, like the FS.

Implementation

Because of the great amount of decentralization that is present in the Metro bundling system adding this feature seems to somehow add a lot of redundancy. Here are the simplified key points of the implementation:

1. A Babel plugin registers virtual modules in packages/metro-transform-worker/src/index.js with transformResult.metadata?.metro?.virtualModules property.
2. The map of virtual modules gets serialized in ‎packages/metro/src/DeltaBundler/Worker.flow.js‎ and passed to the DeltaBundler.
3. Delta bundler merges newly registered virtual modules with currently known ones and marks modules as virtual accordingly ‎packages/metro/src/DeltaBundler.js‎.
4. (currently a stub implementation) Node haste has special handling of virtual modules when creating a SHA ‎packages/metro/src/node-haste/DependencyGraph.js‎
5. (currently a stub implementation) Contents of the module are obtained from the registered source code ‎packages/metro/src/node-haste/DependencyGraph.js‎

Test plan

TBA

I got it working properly in Reanimated monorepo.

[jbroma]

Hey @tjzel,

do you have an example on how this is used within Reanimated to register virtual modules? In other words, how does the Metro public API looks like for VMs.

For context, In Module Federation implementation for Metro we have hacked our own virtual modules implementation but it's very brittle long-term and would love to move to this stable solution 🙌

[tjzel]

Hey @jbroma, sorry I missed your message.

The API to use it isn't pretty unfortunately since it's handled during Babel transpilation. You can see it being used on my draft branch:

https://github.com/software-mansion/react-native-reanimated/blob/%40tjzel/worklets/metro-pr-patches/packages/react-native-worklets/plugin/src/generate.ts#L95C1-L115C6

Long story short, we generate the full code of the virtual module and register it in the transpilation process metadata.

On the other hand, with current Metro APIs I believe it's quite a challenge to create a convenient API for such a feature.