EXT_feature_metadata deep dive

[pjcozzi]

@ptrgags @lilleyse this pull request is not meant to be merged; it is just for feedback from the spec deep dive.

[pjcozzi]

@ptrgags @lilleyse this overall looks pretty solid to me, most of my comments are on the writing and the naming. However, I only skimmed the JSON reference so let me know if there is a specific part of the schema that it would be helpful to deep dive on.

I don't think that this needs to be split into more than one extension but let's discuss if/when Feature Textures should be used.

Other discussion topics:

• Name this extension from the context of 3D Tiles or glTF? E.g. "batched"
• Mix and match: features in different buffers; vertices with multiple feature ids; using both feature tables and feature textures. Help the reader build the mental model. Probably need more diagrams
• Writing
  • Mixing the why and the what/how: batching in the intro, multiple feature IDs
  • With core concepts, start broad then go narrow/specific. Try not to implicitly introduce overall concepts and structure implicitly while introducing the details that implement them.
  • JSON examples needs prose introducing them
  • Needs an appendix / sample models to show complete connections between schema, feature IDs, feature table, etc.
• Feature Texture -> “Feature Metadata Texture” ?
• Use case for Feature Textures - high frequency?

---

@abwood @emackey this glTF extension is basically the next generation Batch Table for 3D Tiles so that now 3D Tiles will contain .gltf/.glb + this extension instead of .b3dm and .i3dm, and that it will have more flexible and semantically-rich ways to store metadata. Any chance you have a few cycles for a shallow or deep review??? 🙏

[pjcozzi]

Let's discuss if "feature metadata" is the right name - and in what context should the name be "right", e.g.,

• This is a glTF context so should the name be more easily understand to that community, or
• This was built for 3D Tiles so should the name be more easily understand to that community?

If the former, then this extension might be something like "batched_metadata", "batched_objects", "batched_meshes", etc.; however we do want the names within the extension, e.g., "feature" to line up with the rest of the 3D Tiles ecosystem I think.

[donmccurdy]

Thoughts, in no particular order:

1. glTF extension naming convention is <PREFIX>_<scope>_<feature> where "scope" is preferably a noun. For a name like batched_objects or batched_meshes I would expect this extension to define the batching mechanism, but arguably the batching mechanism is already there, and this extension provides the metadata/properties that make batches interactive.
2. In my mind "feature" is a geospatial term, but the concept (i.e. a part within a primitive) does not exist in glTF today so I'm not aware of any other precedent.
3. The term "metadata" is a little overloaded, e.g KHR_xmp_json_ld writes, "metadata has no normative effect on the glTF asset appearance and rendering". Arguably "properties" (e.g. GeoJSON properties) would be another option here.

Still thinking this one over, but leaning along lines of...

EXT_[feature | batch]_[metadata | properties]

[donmccurdy]

Renamed to EXT_mesh_features in #20 – hopefully the right balance here, with "features" from 3D Tiles ecosystem and similar "mesh" prefix to EXT_mesh_gpu_instancing. Comments still welcome though!

[pjcozzi]

Just a heads up that I was a bit confused as to why this is in master, CC #3. Even though this is a fork of the main glTF repo, I would suggest to keep master in sync with the main repo or - if there is a good reason - to have another branch in this repo that mirrors master from the main repo.

For this pull request, I just picked a branch that was out of date so the diff for EXT_feature_metadata would be available.

[lilleyse]

Created a new branch for this extension and future Cesium extensions and retargeted this PR: https://github.com/CesiumGS/glTF/tree/3d-tiles-next

[pjcozzi]

Does this add new functionality to the GPU instancing extension? Or is the feature metadata extension dependent on GPU instancing?

I believe it is the later; this extension gets the new functionality; it does not modify the GPU instancing extension.

This is just wording, but it is important to get right so it doesn't confuse the reader or the ecosystem.

[donmccurdy]

Updated in #21:

Optionally, this extension may be used in conjunction with EXT_mesh_gpu_instancing. When used together, certain GPU instance attributes defined by EXT_mesh_gpu_instancing are used as instance feature IDs.

[pjcozzi]

Start with why

Before describing what this extension is, e.g., what a feature is.

Start with the motivation for the extension in the first place and a few use cases to make that concrete.

[donmccurdy]

Addressed in #21, I think.

[pjcozzi]

A good supplement to this table that will help the reader build their mental model would be a diagram that shows the breath of granularity from per-texel to per-vertex to per-triangle to per-node, etc.

That would be helpful early in the spec.

[pjcozzi]

This is the first time that multiple feature tables are introduced. That concept should be intrduced separately and motivated by the potential need to have multiple classes in the same glTF.

The text here should also be expanded to note the use of both explicit and implicit feature IDs if I understand it correctly.

If the format of a table proves difficult to give each of these examples enough context, just switch to making each example / row in the table its own subsection.

[pjcozzi]

Vocab. Is "group" here aligned with group in the other 3D Tiles Next specs?

[pjcozzi]

Will calling this composite confuse readers familiar with composite tiles in 3D Tiles?

[pjcozzi]

in a model

Throughout, please avoid "glTF = model"

From https://www.khronos.org/gltf/:

glTF™ (GL Transmission Format) is a royalty-free specification for the efficient transmission and loading of 3D scenes and models by engines and applications.

[brookesc]

March 8, 2021

Peter, Patrick, Sean

General Notes

Feature Textures

• High variation example -- if you scan temp, you get a different value per pixel
  • Is it really a different value?
  • Error ellipsoid -- Voxel, 4 error ellipsoids/tile
  • May need to separate what could happen vs real use-case
• Vegetation index
• Per pixel want to do volume/height under the pixel
  • Could be high frequency /variance, could be valuable for end user
  • Storage - at each pixel can build a cone, store an angel in each pixel, use that to accelerate
  • Keep it, use cone tracing, per pixel angle as an example, get brass tacks on vegetation index to see if ‘it’s a real thing’
• Naming
  • Feature ID texture, feature cable, feature texture -- should be “feature metadata texture” etc.?

Big Picture

• Need to help with understanding the scope and bounds for other readers -- for example, how things in this spec can or cannot be mixed and matched?
  • Vertices can have multiple feature IDs
  • Be explicit and up front with explanations
  • Should be able to draw all of the concepts once they’re done and connect all the lines
• Mixing why and what/how
  • Lead with the why
  • Start broad for context, go narrow
• Who is the reader/audience, then decide what they know based on that
  • Type aware metadata with mechanics that allow batching; “ready to render”; “platform for adding semantically rich data” -- ensure this is explained
  • Explain core value
• Least amount you can do to get per pixel granularity?
  • How important to have type awareness of those values to build semantically aware extension? Very important
  • That batch table ++ compared to current feature metadata extension - how much more complicated is the latter? 2-3x more -- supports instancing, multiple feature tables, assigning vertices to multiple different feature tables, has stats (good for styling), metadata textures
  • When implementing, ‘how hard is this and how much value does it bring in terms of functionality?’ Cut what is difficult and does not add high value
• Deep Dive: Implementation → TBD

---

CC @lilleyse @pjcozzi @ptrgags

[donmccurdy]

I'm having trouble figuring out how to leave comments at arbitrary locations in this PR, since the "Files changed" list doesn't include the extension itself... is this a good place to leave comments/questions, and if so does it need a rebase to allow that? Would it be more, or less, confusing if I opened a new PR for comments/questions?

[lilleyse]

Unfortunately syncing master with KhronosGroup/master and rebasing 3d-tiles-next on top of master caused this PR to be autoclosed. The discussion can continue in KhronosGroup#2019 #9