RFC: Extend buffer with a metatable

[vegorov-rbx]

Rendered

[jackdotink]

This feels like an exceptional number of special cases compared to how the same features work in the rest of the language. Are the potential optimizations worth that cost?

What are some potential usecases? The motivation explains what the feature does without explaining why that would be valuable.

It's outside the scope of the RFC, but it's worth considering how this might play with buffer slices, should they ever be added.

There have been rumors of a potential compiler and VM mode to disable function environment modification. Could optimizations not work with that rather than creating another special case?

[vegorov-rbx]

This feels like an exceptional number of special cases compared to how the same features work in the rest of the language. Are the potential optimizations worth that cost?

There is only one special case here and that's for the treatment of environment.
In some ways it's no different from inlined function not respecting environment changes because it's now a part of the caller.

Everything else works within the existing features of the language, implementation followed the common semantics of metatables.

[jackdotink]

There is also metatable freezing, snd immutability. Which isn't bad, just an exception from the rest of the language.

[vegorov-rbx]

That uses the table freezing methods and is implemented at the buffer library level.
Nothing exceptional there.

edit: and it is a well-established practice for a similar library feature in a different implementation, so it will actually be familiar.

[gaymeowing]

I think this would be better as being separate from the buffer library, and instead as table.record, table.setbuffermetatable or something else?
Although I have often wished for something like this, as this would be great for serialization libraries to not have to decode buffers into several tables thus increasing memory consumption. With libraries being able to just give the buffer a metatable, allowing for nicer syntax than writing/reading using the buffer library directly.

[vegorov-rbx]

Tables already support metatables and the functionality is unrelated to tables.

[vegorov-rbx]

It's outside the scope of the RFC, but it's worth considering how this might play with buffer slices, should they ever be added.

Added a section expanding on this potential future.

[gaymeowing]

Would adding a way to make buffers read only via a buffer.freeze method be considered for future work? So classes implemented as buffers like Vector4 could behave almost the same as regular vectors in the sense that they're fully immutable.

[vegorov-rbx]

Would adding a way to make buffers read only via a buffer.freeze method be considered for future work? So classes implemented as buffers like Vector4 could behave almost the same as regular vectors in the sense that they're fully immutable.

No, read only buffers were rejected before because that impacts performance and code size of every buffer write.

[TenebrisNoctua]

I don't think I'm really sold on this, what is there in this proposal that absolutely cannot be achieved with an external library that wraps around buffers? Is there a significant benefit that comes from this?

[deviaze]

I don't see why this is a necessary addition to buffers, were people asking for methodcalls on buffer objects? I don't see that as a common complaint. I feel that increasing the footprint of buffer instances defeats the point of having a smol byte buffer/Vec equivalent in Luau.

[vegorov-rbx]

Yes, of course people were asking for this feature :)

The increase is very small and it doesn't defeat the point of buffers even for small objects, we are willing to pay this cost.

[vegorov-rbx]

Made some updates to the RFC.

• Added that buffers have already supported a global host-provided metatable and that metamethod changes only apply to __eq which wasn't called before
• Explicitly called out that __add/__len/etc are available.
• Fixed the size calculation that I got wrong for 0-8 byte buffers, increase is 8 bytes and not 16 in the current allocator.
• As a consequence of the new field, data is correctly 16-byte aligned (that was important for certain userdata).

Don't have it in the RFC, but wanted to say that original buffer RFC called out not having a metatable with an intention of 'Luau not setting up the metatable to not interfere with customizations' and not as a block for buffers to never have metatables.

[alexmccord]

This doesn't seem to mention anything about the performance characteristics of buffer vs table (even if deserialized from buffers) wrt algorithms that does heavy reads/writes? If I have an algorithm that is very heavy on reads/writes, it has to undergo some overhead to turn a few bytes of a buffer into a TValue (or vice versa) whereas tables skips this ser/de.

[alexmccord]

If each buffer needs a pointer to their own metatable, which is 1 word, and you need to know which buffer is host or user defined, then you need that 1 additional bit (and 7 bits padding) to store somewhere. And what happens if you need lua_newbufferdtor a la lua_newuserdatadtor (this feels inevitable)? That would incur yet another 1 word. As I understand this, it doesn't look like you can fit all of this in 8 bytes?

[vegorov-rbx]

There is no need to separate host and user-defined buffers, implementation of the RFC doesn't have this extra bit.

As a side note, there are 8 bits available in the buffer padding today.

[dyslexicsteak]

Although this feature is theoretically perfectly on-brand, being a relatively small addition which generalises well, I still think the motivation is dubious.

The RFC lacks a lot of focus as it is; the first paragraph in the motivation brings up a use case enabled by buffers, with or without a metatable. The usage example presented is also unidiomatic as it is; buffers in Luau are usually packed, not padded and matched to a foreign ABI, which is a bad thing to encourage leaking into userspace. Coordinating a type on both sides, layout-wise, can also not occur without the host's prior knowledge, and since the host knows this statically, a userdatum works fine here; I presume this feature potentially has less friction than userdata, but that wasn't mentioned at all in the RFC and I don't find it to be a compelling reason to add this. It is also generally bad practice to overlay structs on top of blocks of memory without deserialising for various reasons. I'm not sure why this is being promoted.

The parallels drawn to LuaJIT highlight an essential difference: LuaJIT's FFI is anarchistic; it is a low-level C interop mechanism that exposes raw memory and layout directly and makes no attempt to integrate with Lua's object model or provide safety guarantees. Type descriptions are handled in the VM, and userspace is thrown into the deep end as to dealing with foreign values. In contrast, the proposed buffer metatable mechanism is a userspace-extensible abstraction meant to integrate with Luau's object and type system (more on that later). These are fundamentally different design goals, and borrowing the rhetoric of LuaJIT here conflates two very different design philosophies and does not meaningfully strengthen the motivation for this feature; there isn't a ctype shaped hole in Luau right now.

Additionally, I'm unsure why this RFC seeks to paint metatables on buffers as a sort of userspace userdata; control over the byte representation of types on the host side isn't the primary motivator for userdata; userdata are an abstraction for making host values available to scripts. Everything we can do with this feature is already possible with Luau; it's just giving buffers a customisation point for a more syntactically pleasant experience at a high cost in the exact situations where they are most valuable.

Tables are the idiomatic vehicle for user extensible behaviour, structured field access, and structural typing; users already get to place their tables on a spectrum of PODness, and the compiler and VM already invest in making common table patterns fast. Metamethod dispatch, especially with method call syntax, adds several layers of indirection and logical steps the VM must go through to reach the metamethod or table to look up a closure, which itself dispatches based on its arguments. Unlike with tables, with a buffer, after the closure performs its own logic (if any), it finally calls a buffer library access function, which must perform bounds checking and convert to or from TValue. The buffer RFC itself mentions the overhead of metamethod dispatch on userdata in its motivation, so which is it?

It is also far out of scope for this RFC, in my opinion, for a type system extension for intersecting tables with buffers or some such to describe the capabilities of the attached metatable, and frankly, a non-starter for using it with the presented ideas. This metatable-based buffer shape contract is too weak for the proposed use case of punning a buffer as a struct on the host side; even if the fields have the same name and are of the same type, the exact layout, width of values, and whether they even reside in the buffer itself or are produced solely by a metamethod are unknown. If the host side has to call into Luau code to perform an access, the benefits are already lost. The intersection idea, at least preliminarily, doesn't seem to generalise well.

[vegorov-rbx]

Tables are the idiomatic vehicle for user extensible behaviour, structured field access, and structural typing; users already get to place their tables on a spectrum of PODness, and the compiler and VM already invest in making common table patterns fast

The point of this extension is not to replace idiomatic work with tables, but to provide customisability common with other types for additional flexibility.

Metamethod dispatch, especially with method call syntax, adds several layers of indirection and logical steps the VM must go through to reach the metamethod or table to look up a closure, which itself dispatches based on its arguments. Unlike with tables, with a buffer, after the closure performs its own logic (if any), it finally calls a buffer library access function, which must perform bounds checking and convert to or from TValue.

If performance is of concern, the described handling of metatables follow from the work luajit did to show how it allows a performant access in presence of flexible customization.
Both native code generation and the interpreter (by means of opcode specialization) are able to perform inline caching of the target operation, removing those indirections.

Appreciate the feedback and I was able to incorporate a lot of it in the design.
At this point I don't see any blockers for this proposal, so it is expected to go forward.

[alexmccord]

At this point I don't see any blockers for this proposal, so it is expected to go forward.

You might as well not have written an RFC if this is the attitude.

[vegorov-rbx]

The RFC made the proposal more detailed and complete, improving it.
It is also a great chance to get issues that would have been an implementation blocker caught early.

[alexmccord]

Not a float4.

Suggested change

	-- A float4 in a buffer
	-- A vector4 in a buffer

[alexmccord]

With the new setmetatable suggestion, this would be

Suggested change

	`buffer.create(size: number, metatable: table?): buffer`
	`buffer.create<M>(size: number, metatable: M): setmetatable<buffer, M>`

But because currently Luau reports an error when you supply too few arguments even if the tail end of the parameters are generics that can range over nils, you have to turn it into an overloaded function type:

Suggested change

	`buffer.create(size: number, metatable: table?): buffer`
	`buffer.create(size: number): buffer`
	`buffer.create<M>(size: number, metatable: M): setmetatable<buffer, M>`

This is still inconsistent with the current typing algebra for setmetatable though: the function setmetatable cannot be called with buffer but the type function setmetatable can? The behavior of every type functions have been modeled after their cousin function in the value namespace, making this new equation an odd duck. I really want to avoid any divergences in this aspect.

Then there's also the question of what setmetatable<buffer, nil> would even do? If we base it on how setmetatable<{}, nil> behaves (e.g. setmetatable({}, nil)), then by identity with the substitution of M=nil, this would presumably reduce to buffer. Surely we would like for you to get a type error where the type function setmetatable cannot be called on a buffer that have already been created without one, right? As it currently stands with my assumption, this is unsound since setmetatable<setmetatable<buffer, nil>, {...some metatable...}> will gladly report zero errors since the inner setmetatable just reduces to buffer without any provenance showing that setmetatable have already been called on it.

It might seem absurd to care if someone wrote that directly (and it is), but the type system can (and will) derive M=nil (or some variation thereof where M is a union with nil) regardless of the syntax itself, and type functions are distributive over every type arguments: F<A | B, C | D> = F<A, C> | F<B, C> | F<A, D> | F<B, D>. All you need to do is replace C | D with {...some metatable...} | nil, then by distributivity you get setmetatable<buffer, {...some metatable...}> | setmetatable<buffer, nil>, which reduces to { buffer, @metatable {...some metatable...} } | buffer. Part of the reason for having a type system is to detect logic errors, and setmetatable being called more than once on a buffer categorically is a logic error, whether written by user or through type inference.

If you make setmetatable<buffer, nil> not reduce to buffer to avoid being unsound, this makes it different from the first overload (namely buffer.create(size: number): buffer) which would mean the first overload must return setmetatable<buffer, nil>, not buffer, so this should really be:

Suggested change

	`buffer.create(size: number, metatable: table?): buffer`
	`buffer.create(size: number): setmetatable<buffer, nil>`
	`buffer.create<M>(size: number, metatable: M): setmetatable<buffer, M>`

Which then comes back to my first problem I mentioned where we now have one type function that diverged behaviorally from their value cousin... This could be alleviated by allowing setmetatable(buffer.create(16), my_buffer_api) (and treat my_buffer_api as if it had a __metatable field, which prevents a second setmetatable call), which makes the typing algebra consistent with its cousin.

The third issue is how this relates to subtyping: setmetatable<buffer, M> is a subtype of buffer (not the other way around), which makes it too easy to pass any arbitrary buffer into a function that takes a buffer. This is fine, but that function could do anything to the buffer in question. Literally anything. It's the equivalent of char* in C where you could just... change one byte in the buffer and now your whole data structure is straight up garbage data. This is the single biggest concern I have with this proposal: it tries to pretend buffer are not always a opaque blob of data, but it is! This potentially introduces a vector that creates more buggy programs. This is why I am against this RFC in general, we shouldn't make it seem like the type system can give buffers a type. If we do, people will start replacing tables with buffers, and end up mixing all sorts of different buffer types. We don't even have a notion of a readonly buffer, or a standard way to do type refinements on buffers if you try to simulate Rust enums in it, whereas tables can already do this. I have always viewed buffer as ephemeral data that is usually owned by a function or some data structure that needs buffers for performance reasons only, not intended to be exposed to the public. (put another way: this reeks of stringly-typed APIs)

[alexmccord]

It's also harder to make illegal states unrepresentable:

local buf_mt = {
  __index = {
    x = function(b) return buffer.readf32(b, 0) end
  },
  __newindex = {
    x = function(b, v) buffer.writestring(b, 0, v) end
  },
}

local b = buffer.create(5, buf_mt)

local x: number = b.x
b.x = "five"

This makes shape types fundamentally awkward and could even be something absurd like { read x: number, write x: string }. Right now, in this function we infer t: { x: number }, and I think everyone wants this to report an error that you cannot assign string to x.

local function f(t)
    local x: number = t.x
    t.x = "five"
end

It also breaks typestates, which currently assumes that any read after t.x = "five" is of type string (or really, the fresh type for the string literal).