How to describe interleaved planes?

[emersion]

Section 3.2 explains how formats such as Y'CbCr 4:2:0 are described with two Y' planes under the specification (when other conventional APIs would only use one Y' plane). Table 96 contains an example descriptor for this format.

However, I don't see any way to make the difference between two consecutive planes and two interleaved planes. Section 3.2 uses per-plane byte offsets and strides to denote the difference, but I don't see these reflected anywhere in the descriptor in table 96?

Is the expectation that there is no way to make the difference in general? Or, as the following note explains, is the expectation that descriptors may reflect interleaved planes via the fourth sample coordinate, but this isn't mandatory?

One suggested convention for interlaced data is that the field of a sample be encoded in the fourth sample coordinate (the first field as samplePosition3 = 0, the second field as samplePosition3 = 1, etc.)

Or, am I missing something else?

[fluppeteer]

The expectation was that the start address and line stride of a plane are not part of the "format" - as such you can interleave (linear) Y planes by doubling the Y stride and describing one plane as having a base address half a stride offset relative to the other. Arguably all the formats only make sense in the context of a suitable way of extracting their constituent bits from memory. The interleaved plane approach is a convention that shows a way of describing vertical downsampling; in one sense it's a hack, but in another it's a very general one, and more extensible to variations in layout than most. If the memory layout is nonlinear (e.g. Yi,j;Yi+1,j;Yi,j+1;Yi+1,j+1;Ui/2,j/2;Vi/2,j/2 stored in consecutive bits) it's a non-issue - you could argue that interleaving linear Y in the first place is a hack!

The spec could be better at giving examples of how to handle this. It's on the list...

Did that help at all? If it can be made clearer here, we can try to make it clearer in the spec.

[emersion]

Maybe I should explain my use-case a bit to reflect why I want to know whether planes are interleaved. I'm working on pixfmtdb which describes the memory layout of various pixel formats. The memory layouts are automatically generated from KDFS descriptors.

For Y'CbCr 4:2:0 I'd like to print something like this:

Plane 0:
0                        1
Y1₇Y1₆Y1₅Y1₄Y1₃Y1₂Y1₁Y1₀ Y2₇Y2₆Y2₅Y2₄Y2₃Y2₂Y2₁Y2₀
Y3₇Y3₆Y3₅Y3₄Y3₃Y3₂Y3₁Y3₀ Y4₇Y4₆Y4₅Y4₄Y4₃Y4₂Y4₁Y4₀

Plane 1:
0
U₇U₆U₅U₄U₃U₂U₁U₀

Plane 2:
0
V₇V₆V₅V₄V₃V₂V₁V₀

I want the first two planes in the KDFS to be detected as interleaved and printed like this, because that's what everybody expects. It would be confusing to print 4 planes for formats documented to have 3 planes.

I was originally hopeful because of this note in the spec:

There is no expectation that an API must actually use this representation in accessing the data: it is simple for an API with explicit support for the special case of integer chroma downsampling to detect interleaved planes and to deduce that they should be treated as a single plane of double vertical resolution.

I thought there would be an easy way to detect the special case.

So I guess there is no way to figure this out then?

[fluppeteer]

Thanks for the explanation. :-)

So I guess there is no way to figure this out then?

Not just from the pixel format - it's a bit of a cheat to punt this, but trying to encode plane offsets is a bit of a can of worms, and the obvious ways to do it (including trying to encode chroma downsampling) don't extend nicely to things like Bayer and X-Trans in the same way.

However, presumably something alongside the format needs to know plane base addresses and sizes/strides. If there's a "portable" description of the start address, size and stride of each plane without reading too much into the representation, it should be possible to see that the plane locations overlap - if plane 1 starts at a half-stride offset from plane 0 and the plane sizes are half the stride, it's fairly clear that they interleave, so if whatever is using the result needs to know that it's a downsampled luma/chroma image then it ought to be a fairly simple special case. The assumption is that whatever needs to know is probably going to be specialised to a subset of the thing that could be described anyway, so this unpacking shouldn't; be too complicated. Equally, if you know you have a downsampled format and you're trying to describe it, it shouldn't be too hard to double the plane offset values.

All this does assume that you're treating the memory layout as linear, but unless you have relatively small address swizzles (rather than, say, Morton order the whole way up) that's probably best handled outside the remit of the DFD.

I should help ensure that there are better, or more complete, examples in the KDFS - although it's good to see people trying to use it in anger. If something is hard to express, so long as fixing it won't reduce expressivity in an area we need, we can iterate. (For example, I'd already considered using a bit field for describing channels - which would reduce the number of channel available, but make some of the format descriptions more concise and easier to decode. We've not yet gone that way, and it would be a compatibility break, but these things become clearer with use.)

[fluppeteer]

Some more explanatory wording has been added to the introduction in chapter 2 of version 1.4.0 of the spec. @emersion - please let us know if this is sufficient, or if something more is needed (in 1.4.1).