Multiple Components Inside Uncompressed Item

[dukesook]

I'm interested in encoding a single 'unci' item with many monochrome components. Is there an API function for this? Is it experimental? Or will this be part of v1.22.0?

[farindk]

The current implementation does not support multiple components with the same type. Even if using the experimental heif_image_add_channel(), this case was not considered yet. We will first have to change the function signature for that. E.g. return a component ID (or index) that lets us differentiate this.

Another option would be to have an API that very closely resembles the ISO/IEC 23001-17 capabilities. That would allow us to just pass through whole the image data and libheif will use exactly the alignment/padding specs of the input.

I am also not aware of any standardized way to specify what kind of data is contained in a component. If there are several monochrome components, how can we list what is contained in each of them?

[dukesook]

I am also not aware of any standardized way to specify what kind of data is contained in a component. If there are several monochrome components, how can we list what is contained in each of them?

This is a good question. For GIMI, each component receives a unique content ID and metadata is attached via RDF. However, I'm not aware of a more standardized "MPEG" way to do this.

[farindk]

For GIMI, each component receives a unique content ID and metadata is attached via RDF.

How is it possible to assign a content ID to each component of a single image? I found it.

Would it be a viable alternative to add several image items with one monochrome component in each?
That can be easily done with the current libheif.

[dukesook]

To do this in GIMI, instead of the traditional ItemContentID, you would associate the item with an ItemComponentContentID property (see section 6.7.5):

aligned(8) class ItemComponentContentIDProperty
extends ItemProperty('uuid',0x9db9dd6e373c5a4e811021fc83a911fd){
  unsigned int(32) number_of_components;
  for(i=0;i< number_of_components;i++)
    utf8string ItemComponentContentID;
}

Yes, adding each component as it's own item has been easy and useful. But placing many components into a single item is a crucial feature for handling hyperspectral imagery.

Unfortunately, I'm not sure of a clean way to implement this in libheif without major refactoring since HeifPixelImage maps each channel to a single ImagePlane.

[dukesook]

Another option would be to have an API that very closely resembles the ISO/IEC 23001-17 capabilities.

That might be nice, but another use case is encoding many components using JPEG 2000 so a codec-agnostic solution would be ideal.

How does the following sound?

• Change the m_planes variable type from map<heif_channel, ImagePlane> to vector<ImagePlane>
• Give the struct ImagePlane a heif_channel variable.
• Refactor HeifPixelImage methods like get_channel() & get_plane() to simply return the first match found.
• Have the experimental heif_image_add_channel() return the component index.

[farindk]

How does the following sound?

Yes, that is the way to go. Probably, we also need a flag like is_visual_image() to indicate whether a color-conversion makes sense. I'm thinking of a hyperspectral image with many monochrome channels. If we simply return the first monochrome channel found, the color conversion will convert that channel happily to RGB which is unexpected and leaves the image in a state in which it is monochrome and RGB at the same time. These kind of cases are what I am worried about. We just have to be careful.

[dukesook]

I started draft PR #1660 which has vector<ImagePlane> m_planes which is somewhat functional, but required many changes.

Here's another idea that I just thought of, do you think we could add a new hyperspectral colorspace? I believe it would:

• Help the color converter distinguish between the channels
• Work with std::map<heif_channel, ImagePlane> and therefore reduce the amount of refactoring

However, I not sure if a single ImagePlane object could or even should store multiple channels of imagery.

[farindk]

How would we detect that the input image is hyperspectral? Just because it has a number of monochrome channels does not mean that it is hyperspectral.

What would the color conversion do with a hyperspectral image? Or is it only for signaling that it is not a luminance-only image and should not be converted?

[dukesook]

The caller could specify by passing in heif_colorspace_hyperspectral when calling heif_image_create().

Yes, if appropriate, the color conversion could throw an error and avoid the "monochrome and RGB at the same time" issue the you previously mentioned.

[farindk]

The caller could specify by passing in heif_colorspace_hyperspectral when calling heif_image_create().

Yes, that's when we are encoding an image. But what about decoding? There is currently no way to store this information in the file, I think.

[dukesook]

Okay, I see. Perhaps the vector<ImagePlane> m_planes is still the best option.