Skip to content

WebXR Mesh Detection#5791

Merged
mvaligursky merged 9 commits intoplaycanvas:mainfrom
Maksims:webxr-mesh-detection
Dec 4, 2023
Merged

WebXR Mesh Detection#5791
mvaligursky merged 9 commits intoplaycanvas:mainfrom
Maksims:webxr-mesh-detection

Conversation

@Maksims
Copy link
Collaborator

@Maksims Maksims commented Nov 5, 2023

Integration of WebXR Mesh Detection API.

Tested on Quest 3. This data on Quest 3 comes from pre-scanned information and added furniture. You can trigger a request for manual room capture, so users can provide refined information.
Includes an example of building geometry and labeling them.
This API design based on planeDetection API.

Test link: https://examples.playcanvas.meta.space/#/xr/ar-mesh-detection

This API can enable many cool AR applications:

  1. Efficient depth based occlusion of virtual geometry.
  2. Physics interactions of virtual objects with mesh that approximates real-world.
  3. Using semantic labels it is possible to identify where screens are, table surfaces, and more.
  4. And much more.

New APIs:

// pc.XrManager
xr.meshDetection // XrMeshDetection interface that provides an access to XrMeshe's
xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
    meshDetection: true // new option to request mesh detection features
});

// pc.XrMeshDetection
// this class provides an access to meshes.
// XrMesh instances can be added/removed mid-session by underlying system, due to real-time scanning, tracking state changes, manual scanning, etc.
meshDetection.supported // true if mesh detection is supported
meshDetection.available // true if mesh detection is available. This information is only available when session has started.
meshDetection.meshes = [ ]; // an array of XrMesh instances
meshDetection.on('available', () => { }); // fired when mesh detection becomes available
meshDetection.on('unavailable', () => { }); // fired when mesh detection becomes unavailable
meshDetection.on('add', (xrMesh) => { }); // fired when a new XrMesh have been added
meshDetection.on('remove', (xrMesh) => { }); // fired when XrMesh have been removed

// pc.XrMesh
// this class represents a specific XrMesh, its vertices data and attributes
xrMesh.label // string - a semantic label of this mesh
// can be any of: https://github.com/immersive-web/semantic-labels/blob/master/labels.json
// but not limited to this list. e.g. Quest 3 also uses "global mesh".
xrMesh.vertices // Float32Array - local positions of vertex data. This data can change during lifetime of XrMesh, when "change" event fired.
xrMesh.indices // Uint32Array - indices for vertices to construct triangles. This data can change during lifetime of XrMesh, when "change" event fired.
xrMesh.getPosition() // global position of XrMesh (Vec3), this data can change without firing "change" event.
xrMesh.getRotation() // global rotation of XrMesh (Quat), this data can change without firing "change" event.
xrMesh.on('change', () => { }); // fired when any of its attributes: vertices, indices, label have been changed.
xrMesh.on('remove', () => { }); // fired when XrMesh have been removed

Fixes #5699

a95e570cc5a5ba934b551e679016a956.mp4

I confirm I have read the contributing guidelines and signed the Contributor License Agreement.

@Maksims Maksims marked this pull request as ready for review November 5, 2023 17:27
@mvaligursky
Copy link
Contributor

Great PR and the example! Thanks so much @Maksims .
I left a few small comments, but this is pretty much ready to be merged.

@Maksims
Copy link
Collaborator Author

Maksims commented Dec 4, 2023

Made a quick update based on comments, and removed ID as it is nowadays redundant, as Map is widely used for indexes.

@mvaligursky mvaligursky added enhancement Request for a new feature area: xr XR related issue labels Dec 4, 2023
@mvaligursky mvaligursky merged commit 706d7bc into playcanvas:main Dec 4, 2023
@Maksims Maksims deleted the webxr-mesh-detection branch December 4, 2023 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: xr XR related issue enhancement Request for a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebXR Mesh Detection

2 participants