RGB9E5 representation / custom float formats

[mlvdm]

I am trying to understand how to represent the RGB9E5 format as shown for:

• GL : https://www.khronos.org/registry/OpenGL/extensions/EXT/EXT_texture_shared_exponent.txt
• VK : https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#textures-sexp-RGB

Note that the conversion to a value is given in both of the above as value = mantissa x 2^(exponent - bias - bits_in_mantissa)
For this format, bias = 15 and bits_in_mantissa = 9, and there is no implicit 1 or sign bit.

The data format spec (section 10.4, non-standard float formats) explains how to encode custom formats, and it gives an example of RGB9E5 as well (table 98), but I think I'm missing how that follows.

sampleUpper is supposed to be the mantissa value that represents 1 for exponent (after bias) 0.
sampleLower is supposed to be the mantissa value that represents 0 for exponent (after bias) 0.
For RGB9E5, substituting 0 for exponent - bias, we are left with value = mantissa x 2^(0 - 9).
To represent the value 1, mantissa should then be 512 because 1 = 512 x 2^-9.
To represent the value 0, mantissa should then be 0 because 0 = 0 x 2^-9.

However, in Table 98 (the example for RGB9E5) sampleUpper is set to 256, not 512.
(As expected though, sampleLower = 0, bitLength = 9 and bias (sampleUpper for the exponent sample) = 15).

I wonder if Table 98 contains a mistake, or if I am missing something somewhere.

PS: if 512 is indeed the correct value, then that runs afoul of the "implicit one detection", because 512 is one larger than the representable mantissa value. Moreover, to actually represent the value 1 in RGB9E5, I believe you need to use exponent-after-bias 1 (or higher) or you run out of mantissa bits.

PS2: And by extension, there is no single representation of 1 in RGB9E5 either, because I can arbitrarily bump the exponent by one and shift the mantissa by one, and it's still a value of 1. Is there a specific "representation of 1" that should be used?

[fluppeteer]

Oh dear. It was late and I was tired when feeling smug that I'd worked out how to represent this.

I suspect I wanted 256 in the example - I suspect my formula is wrong for the explicit 1 case, and should have an offset. Also it definitely shouldn't say "sampleUower". I'll engage brain and sort out the mess.

I would expect anything capable of decoding an explicit exponent format to handle the equivalent representations, but it wouldn't be a bad idea to encourage people to use a canonical/normalised encoding so that there's a fixed bit representation for a value. Vulkan defines a standard encoding for RGB9E5, so I would propose aligning to that. I think (although I'll check when I'm not multi-tasking) that means 1.0 should have the 1 in the most-significant bit of the mantissa (which minimises the difference between 1 and 1.00001), but I'm open to discussion if there a benefit to minimising the size of the mantissa.

I'll put something together. Thank you for catching it.

[mlvdm]

There will only ever be zero or one representations of the value 1 if you fix the exponent (ie: today you say "exponent after bias = 0", which fixes the ambiguity). The issue I think lies with the implicit one rule which doesn't work for this case.

(There could well have been an alternate dimension RGB9E5 format which was defined as mantissa x 2^(exponent - 15 - 8), which seems to be the format shown in Table 98, or a RGB9E5 format which was defined as mantissa x 2^(exponent - 15 - 10), which suffers the same implicit one issue as the original)

Maybe the solution is to encode the absence of the implicit one not in the sampleLower or sampleUpper fields. Maybe one of the FSEL bits on the mantissa sample could be repurposed? Maybe something like "if F=1 on the mantissa sample, there is no implicit one bit, otherwise (the previous rule for implicitOnePresent = sampleUpper > maxMantissaValue)", that would be more or less backwards compatible except for formats that needs to store sampleUpper that would otherwise trigger the old rule, which is formats that are currently not encodable anyway).

(There is other edge cases, which seem pretty unlikely, where if the exponent bias is somehow not near the "typical" values, you'd either run out of bits (you'd need to store value > UINT32_MAX) or there is no representable 1 (a sampleUpper of 1 would already represent a value larger than 1). Although maybe it's not super farfetched, the old 80-bit long double would need to encode UINT64_MAX+1 in sampleUpper I believe).

Anyway, I'll leave it up to the authors, maybe there's a better option somewhere, and I'm not sure what kind of flexibility you are looking for.

[fluppeteer]

Sorry for the delay in chasing this.

The encoding of custom float formats changed while the KDFS was under development, and the RGB9E5 description apparently didn't get fixed.

The bias is the value of exponent which will place a binary point above the top encoded bit in the mantissa. (At least, this definition of bias matches Khronos's use in RGB9E5.) Formats with an explicit 1 therefore encode 1.0 with an exponent set to the bias + 1 (whereas formats with an implicit 1 encode 1.0 with an exponent = bias); this unfortunately removes the convenience of the sampleLower combination looking like 1.0 encoded in the format, but it's probably more confusing (and possibly limiting) to place "bias-1" in sampleLower.

The correct encoding for RGB9E5, I think, is to have the mantissa sampleLower = 0 and sampleUpper = 256 (which indicates that this is an explicit-1 format, because 256 falls within bitLength for the sample), and for the exponent to be encoded with sampleLower = 15 (the bias used in the Khronos formulae) and sampleUpper = 31 (because no values are reserved for infinity or NaN).

I think the description of how this works was mostly correct, except the term you flagged with "exponent after bias = 0". I've done some rewriting of that section, including an example mapping to RGB9E5, so I'll try to get that folded into a public spec soon.

@mlvdm - does this sound plausible to you?

[fluppeteer]

You're right that you might need an extra sample for the mantissa to describe formats that go over a word length. You do anyway for (say) a 64-bit double, of course, but I appreciate the inconvenience.

Having had a quick look at the 80-bit format just to check, it's... weird. There are only 63 bits of mantissa used for standard denormalised numbers, and bit 63 is used for special cases. I think it looks like most code could ignore it and assume it's 1 (which would do the wrong thing for 0 and denormalised numbers, but hardware would interpret them correctly), but it might require special handling anyway. I'd assumed it was a standard IEEE float format with a 64-bit mantissa; live and learn!

[mlvdm]

This sounds reasonable to me, so when decoding a DFD if I see sampleLower of the mantissa be < 2^bitLength, I can assume there is no implicit one. In this case, 256 < 2^9, so that is OK.

(But IIUC sampleLower effectively just encodes 1 bit of information, since setting it to 255 or 42 would have the same meaning, and setting it to 1000 would mean "implicit one" just as much as 512 would, as sampleLower doesn't represent any particular value anymore?)

[fluppeteer]

Thanks, @mlvdm.

I had to think about this (the curse of coming up with the cunning plan some time ago). I think all the samples are still useful, although I'm willing to be persuaded that I've confused myself:

• The exponent sampleLower indicates the bias used to encode 1.0 (handled slightly differently for implicit and explicit 1 cases).
• The exponent sampleUpper indicates the upper exponent range and therefore whether encodings are reserved for infinities and NaNs (this probably only encodes one useful bit).
• The mantissa sampleUpper shows how 1.0 is encoded, including whether an implicit or explicit 1 is used.
• The mantissa sampleLower does the same for -1 or 0 (more usefully for explicit-1 formats).

If your "1.0" value is actually encoded at 0.75 (by symmetry with Y'CbCr formats that have headroom and footroom), the mantissa sampleUpper would show the encoding for 0.75 (either 1.10000b or 0.110000b) and the exponent bias would have been chosen for the correct number to be encoded. This does mean the bias may not correspond to a conventional binary encoding of 1.0.

There is a potential issue that if you have a signed format which encodes a range between -0.25 and +0.75 (for example) with an implicit 1, this description can't handle it - you'd need a different exponent for the two cases. Of course, this applies only if the encodings are also not a "standard" float format which can be described directly. So it's not perfect, but I wouldn't think such a representation would be all that common; I really hope that float format colour spaces which have an offset black and white point (which absolutely aren't rare) tend fo use relatively standard float formats.

If there's an incompatible break in the KDFS at some point, I'll try to revisit this. I have a vague desire, having used it more in anger, to rearrange the way channels are described, and that may afford an opportunity - but I don't want to over-complicate changes in the meantime.

[fluppeteer]

You know those times when you post something publicly and go for a walk and come back completely disagreeing with yourself?

My "the exponent sampleUpper is only encoding one bit of information" argument makes me feel it's wasteful not to have more direct support for custom float encodings for which the sampleLower and sampleUpper would have different exponents. I really hope there aren't encodings out there that use more than the IEEE "exponent = max" approach for infinities and NaNs. Other than the Intel 80-bit format, of course, but I think you can effectively ignore bit 63 (which we fortunately have a way to do) and everything might just work, with apologies to anyone using an 8087 or 80287, and my brain cells aren't up to decimal packed floats at the moment.

So it took me a while, but I think I agree with you, @mlvdm, although for a slightly different purpose: if there's an exponent sample associated with a channel and a location, if the corresponding mantissa has a float bit, this should indicate that the exponent supports NaN and infinities; otherwise the full exponent bit range is available. The sampleLower for the exponent then encodes the exponent for the sampleLower mantissa, and the sampleUpper for the exponent encodes the exponent for the sampleUpper mantissa (and we hope nothing too complicated is going on for shared exponent formats).

That leaves one question: sampleLower and sampleUpper are now more directly encoding -1.0/0/whatever and 1.0/whatever. For formats without an implicit 1, the standard way of describing the bias (which I went to such trouble to try to word) isn't what you put in the exponent field when describing sampleUpper. With a break from the Khronos internal terminology of "bias" (that I can still describe internally as being offset by 1), should I use the exponents to encode sampleUpper and sampleLower, or encode the bias and have an offset for non-leading-1 formats? I'm inclined to say that just having the values encoded is less confusing, and I should fix the formulae.

What do you think?

[mlvdm]

I am not sure, I did not previously consider the headroom case.
Given that, a DFD parser cannot assume that the encoded floating point value 1.0 would map the the logical "white" 1.0.
So it will have to come up with the complete model of the floating point encoding in order to interpret the white value.

To do that, we are given the exponent sample: 5 bits in length, sampleLower=15 (the bias) and sampleUpper=31 (no infs/nans). The FSEL bits tell me this is a sign/magnitude format, so the interpretation is value = mantissa x 2^(exponent-bias). So far so good.

Then, I find the mantissa sample which I recognize by FSEL=0 and the same channel, it has 9 bits in length and all I have is sampleUpper for the white point and sampleLower for the black point.

I might find 256 as in the example, and we can map that to 1.0 because we know that 256 < 512 implies there is no implicit one, and by definition of explicit one we know to add 1 to the bias. And sure, 256x2^(-15+1)=1.0

Let's now consider that I want white to be 2.0. I cannot change the bias in the exponent sample (as I need to know that the encoding scheme uses the bias of 15 to decode). But I equally cannot put 512 in the mantissa sample, as that would now look like an implicit one is present, and the white value is interpreted to be 1.0, not 2.0 (as the float16 example).

So I think at this point, the current scheme kind of falls over. You suggested that maybe we should store the actual exponent value for the white and black points in the exponent sample. But I'm not sure that you can leave out the 15 "base" value, as it's needed for the encoding scheme, and I don't think you can infer it in another way since there is no anchor on the mantissa fields that specifies the encoded value has a known float value.

Maybe instead the mantissa sample should store the white and black points including the mantissa and sign + exponent fields (ie: store the entire encoded value). Because I can know the encoding scheme from the sign + exponent samples for the same channel, I can reconstruct the meaning (although we would need to establish for example that these mantissa fields are always stored in "sign field" .. "exponent field" .. "mantissa field" sequence with lengths from their associated samples so I can split the stored value correctly).

PS: I didn't consider anything except Mx2^(E-B) forms with possibility fo implied/explicit ones, there could be other cases not considered.

[fluppeteer]

Yes, you can't simultaneously know the value of 1.0 and an arbitrary white amplitude (without more degrees of freedom in the encoding). I wonder if you need to, though. You don't necessarily need to define white as 2.0 and simultaneously have a way to distinguish the encoding of 2.0 from 1.0, I think - so long as you can map the bit pattern (which happens to be 2.0 in "normal" speak) to 1.0.

I'm not averse to the whole value being encoded in the mantissa sample, though. I'm slightly wary though that doing this still only supports encoding "1.0" as a power of two rather than an arbitrary number, as you say. This might handle a lot of common cases (including the typical black and white point case I was worried about), but it's still incomplete in a sense. It does mean we can go back to distinguishing the existence of special values by the exponents, though (it might still be useful to find other uses for the exponent sample sample_lower if only one bit is relevant), and use the F bit as you suggested to pick out the implicit 1 on the mantissa. That might still be an improvement. The sign bit is normally a separate part of the mantissa, and working out how that ties in to the sample_lower/sample_upper might be worth considering.

I'll have a think while I sleep, but thank you for the brainstorm. Any other thoughts are welcome - it's always a good thing to have more flexibility.

[fluppeteer]

Follow-up: there's a bit of a question when it comes to encoding "the whole number" in the mantissa sampleLower/sampleUpper fields specifically for something like RGB9E5 - we'd need a defined ordering in the sample bit layout. In the case of RGB9E5 you'd probably need to mandate 9 bits of mantissa in the bottom 9 bits of sampleLower/sampleUpper and have the exponent immediately above it (and if there was one, a sign bit immediately above that). Which is fine, it just doesn't look particularly like one of the samples, so a flexible decoder needs to decode things twice to make sense of it. Having the exponent and mantissa stored separately is logically simpler, but I'm not sure whether it's practically simpler.

That's not the end of the world, just a consideration in balancing ease of use against functionality. I guess the big question is whether retaining a separate (and power of two only) meaning of "1.0" is meaningful given the separate upper/lower range definition. I'm not sure either way, and maybe that's an argument for going with the version that's more flexible.

I'm going to drag @lexaknyazev and @MarkCallow in for opinions. Whatever we choose here is a quick fix, but we need to decide before there's too much content in the wild using an obsolete 9995 descriptor, so it would be nice to do it once and for all.

[fluppeteer]

It appears that we have a more urgent deadline than I had thought.

@mlvdm, absent a strong feeling either way, I'm happy to follow your suggestion and propose that the whole value (including exponent and sign) be encoded in sampleLower and sampleUpper for the mantissa values. That is:

• The exponent sampleLower should encode the bias.
• The exponent sampleUpper should use one bit to encode whether special values are allowed in the exponent; other bits are reserved.
• The mantissa sampleLower and sampleUpper should encode the lower and upper range (as with fixed-point formats), with the specified number of mantissa bits immediately followed by the exponent used to represent the value (in higher bits of sampleLower/sampleUpper), followed by a sign bit if present. This means that an explicit representation of an IEEE754 32-bit float happens to store float values in sampleLower and sampleUpper. Additional samples will be required if sampleLower and sampleUpper cannot hold the specified range (e.g. for 64-bit doubles).
• If the format uses an implicit 1 in the mantissa, the float (F) qualifier should be set on the mantissa sample; this can be distinguished from a true float by the presence of an exponent for the same channel and sample location.

Is this what you would like? Does it affect anything you're doing? It's a breaking change, but I'm not aware of anything using the non-standard float format except KTX2, which is only about to start using it.

@lexaknyazev, @MarkCallow - input please? The most obvious alternative is the approach which stores the sampleLower and sampleUpper values across the mantissa and exponent samples, but this loses a separate definition of "1.0" in the custom float format (whether this is valid anyway will depend on whether the format is expected to have a power-of-two representation). I'm not sure whether this matters and whether it is more useful flexibility or confusing, but it might make some black/white-point representations a little easier to follow.