Transformer Proof-of-Concept

[ianjosephwilson]

Proof of concept for discussion, needs tests and more refactoring.

• Repacks TNode tree into a new Template after parsing to determine the structure.
• Renders Template directly to str without Node using iterative solution.
• Extension point concepts
  • Components can affect tdom's rendering of descendents by returning a dict[str, object] in a 2-tuple
    • ie. Components can ask tdom to set context vars for descendent components to access by returning (Template, {'context_values': ((CTX_VAR, 'value-to-set'),)})
  • "System" kwargs can be provided to participating components (beyond just children).
    • ie. def Component(sys_context, children) -> Template

Some of this we might be able to back-port into tdom's Node renderer.

[pauleveritt]

@ianjosephwilson That certainly would work for test-writing but it would eliminate other uses of Node. I previously mentioned middleware and tooling.

But I'm also interested in interoperability. For example, I have a decorator that lets an htpy "component" be used in tdom. They both have similar concepts of a node structure. It would be great to converge on a common standard, to prevent tdom from being yet-another-template-language.

[davepeck]

@ianjosephwilson

Wow, lots to think through here. Thank you.

In lieu of an organized response, here's a grab bag of thoughts:

1. In the long-term, if it becomes successful, I assume tdom will want a true compilation step for performance optimization. This is an excellent draft stab at such a step. The question on my mind is: at what point in this project's life is all the extra machinery here (or something like here) desirable? I'm happy today if tdom is competitive with Django's comically slow templates; render time is not a dominant cost in a typical web app. If compilation machinery slows down iteration on tdom's developer-facing experience, I'd probably pass for now. If it doesn't, it seems like a win.

2. It feels like Node can sit on the outside of this entire pipeline, useful for code that wants it. html(t"...") can return a string; node(t"...") can return a Node. For the typical web app, node() seems extraneous: you want to get to a rendered str in as few hops as possible. (I say outside of the "entire" pipeline; I'm sort of hand-wavingly assuming there's a way for node(...) to avoid re-parsing things.)

3. The more I dwell on it, using Template for component children feels good to me. In React-land, an older common pattern was to use React.Children.map() — for instance, in a layout component — to wrap or otherwise alter each child. Use of Template precludes this pattern since children is, effectively, opaque. I think I'm okay with this?

4. As a result, I suppose I have to get okay with using Template instances to pack roughly anything that has strings alternating with values. I don't like it, though: type information is lost, and we only really care about Interpolation.value; it still feels like a smell to use a type that is mostly intended to arise from literal syntax in this way. Ah well?

5. CONSIDER: what about interpolation format specs, conversions, etc.?: yes, and I still don't have a good instinct here.

6. The context stuff is interesting but feels entirely separable from this PR.

Ian, curious how you're feeling about the direction overall.

[pauleveritt]

@ianjosephwilson and I had a long chat this afternoon. I really enjoyed it and I now understand much better his thinking and his direction.

2. Ian and I kind of converged on what you describe. There's the step to assemble all the pieces, where components call components, etc. This should all stay in Template. On the way out the door, as we make a string, we might go to Node if you have middleware. Or a test might go to Node.

2a. First fly in the ointment: this means the TNode side is ignorant of the DSL. It doesn't know HTML. So there are some features we'll lose from current tdom -- boolean attribute collapsing, self-closing tags, perhaps aria and data. We briefly discussed the idea of "dialects" -- simple understandings of the DSL, available in template-land, without making a Node..

2b. It means htpy and tdom can't be mixed together, except as strings.

2c. Some of my needs require component information (e.g. paths relative to component disk location.) It so happens this relates to tracing -- the need for good error messages.

4. I do worry that, by dropping DSLs entirely, we're losing something. tdom will know nothing about the semantics of the language being rendered. It won't know about trees and containment. I will say, 22 years ago when Guido was working on ZPT (our Zope template language, which was structural) he complained he just wanted a template system for strings, not nodes, so Guido will be happy if tdom drops the dom. 😄

[ianjosephwilson]

@ianjosephwilson and I had a long chat this afternoon. I really enjoyed it and I now understand much better his thinking and his direction.

@pauleveritt Whew! Thanks for meeting. It is going to take a while for my brain to digest all that information. It has given me a few ideas though already.

2. Ian and I kind of converged on what you describe. There's the step to assemble all the pieces, where components call components, etc. This should all stay in Template. On the way out the door, as we make a string, we might go to Node if you have middleware. Or a test might go to Node.

I really like the approach-ability and simplicity of the Template IN Template OUT concept. I realized we can keep using Node in a template by just abusing the lack of typing and the fact that things are passed into str() if needed:

def Comp(children: Template):
    return t'<div>{children}</div>'

node = Element('span', children=[Text('1')])
# Behold! A Template!
children_template = t'{node:safe}'
# type system says Template IN Template OUT... OK!
comp_template: Template = Comp(children=children_template)

This could maybe provide that escape hatch to go to Nodes early. I have to think more about that.

2a. First fly in the ointment: this means the TNode side is ignorant of the DSL. It doesn't know HTML. So there are some features we'll lose from current tdom -- boolean attribute collapsing, self-closing tags, perhaps aria and data. We briefly discussed the idea of "dialects" -- simple understandings of the DSL, available in template-land, without making a Node..

Some of these are supported and maybe I still didn't explain the difficulties with the hybrid static-vs-dynamic -- static: "we know" and dynamic: "we can't know until we render" very well. We touched on it but I need some better examples and maybe a "visualization" we could see.

2b. It means htpy and tdom can't be mixed together, except as strings.

I'm not 100% sure about this but I guess if its to walk the tree then yeah it won't work well. I think I need to push these examples along a bit and we could start experimenting with this.

2c. Some of my needs require component information (e.g. paths relative to component disk location.) It so happens this relates to tracing -- the need for good error messages.

This did panic me a bit because it seems like it could be a real show stopper issue but I think there is a caveat. TDOM should be able to retain some sort of information when it invokes a component and gets a template back. I'm not sure exactly how its going to track that and I have some "questionable" strategies to try but it seems like it should be possible to say this component made this template. The intention being to either tack on something we need later or to provide debugging information when a nested component in that template fails. I need to research this and probably look at some other systems.

4. I do worry that, by dropping DSLs entirely, we're losing something. tdom will know nothing about the semantics of the language being rendered. It won't know about trees and containment. I will say, 22 years ago when Guido was working on ZPT (our Zope template language, which was structural) he complained he just wanted a template system for strings, not nodes, so Guido will be happy if tdom drops the dom. 😄

We still have the TNodes which provide the DSL information but the transformed template is more limited. I will try to come up with some sort of visual / explanation to improve clarity here. I have a hard time explaining it as well as understanding it myself.

[ianjosephwilson]

@ianjosephwilson

Wow, lots to think through here. Thank you.

In lieu of an organized response, here's a grab bag of thoughts:

1. In the long-term, if it becomes successful, I assume tdom will want a true compilation step for performance optimization. This is an excellent draft stab at such a step. The question on my mind is: at what point in this project's life is all the extra machinery here (or something like here) desirable? I'm happy today if tdom is competitive with Django's comically slow templates; render time is not a dominant cost in a typical web app. If compilation machinery slows down iteration on tdom's developer-facing experience, I'd probably pass for now. If it doesn't, it seems like a win.

I agree things are a bit muddled here. I wanted to keep the performance in view while trying to get the feature-set under control and really iron out what is happening. In a way similar to the attribute resolution. I think doing them both at the same time here actually helped with that. I am going to try to untangle that a bit so we can look at the core functionality and then decide if this transformation is still helping.

2. It feels like Node can sit on the outside of this entire pipeline, useful for code that wants it. html(t"...") can return a string; node(t"...") can return a Node. For the typical web app, node() seems extraneous: you want to get to a rendered str in as few hops as possible. (I say outside of the "entire" pipeline; I'm sort of hand-wavingly assuming there's a way for node(...) to avoid re-parsing things.)

This is my feeling too and maybe some of the more "middleware" use-cases @pauleveritt wanted/needs could be resolve in other ways or with some more advanced functionality. The most common case would be to have Node as the final output though.

3. The more I dwell on it, using Template for component children feels good to me. In React-land, an older common pattern was to use React.Children.map() — for instance, in a layout component — to wrap or otherwise alter each child. Use of Template precludes this pattern since children is, effectively, opaque. I think I'm okay with this?

I did see that but it seems react has moved away from manipulating those children and more towards treating it as a "opaque" slot. I think that would be our common case but you could do all sorts of weird stuff as long as the end result is another Template. It seems like re-building the Template itself is fine but modifying what's in tdom, the cache, etc. would be "highly" discouraged for now.

4. As a result, I suppose I have to get okay with using Template instances to pack roughly anything that has strings alternating with values. I don't like it, though: type information is lost, and we only really care about Interpolation.value; it still feels like a smell to use a type that is mostly intended to arise from literal syntax in this way. Ah well?

As mentioned before some of that "cleanup" is bundled into this and needs to be called out probably in another PR of some sort. I tried to really constrain what could be in there.

5. CONSIDER: what about interpolation format specs, conversions, etc.?: yes, and I still don't have a good instinct here.
6. The context stuff is interesting but feels entirely separable from this PR.

I agree but I wanted to be sure that the system still would be compatible with this alternative design. Most notably this situation:

>>> def Inner(children):
...     print ('Inner')
...     return t'<span>{children}</span>'
...     
>>> def Outer(children):
...     print ('Outer')
...     return t'<div>{children}</div>'
...     
>>> res = html(t'<body><{Outer}><{Inner}>{"CONTENT"}</{Inner}></{Outer}></body>')
Inner
Outer
>>> res = render_service_factory().render_template(t'<body><{Outer}><{Inner}>{"CONTENT"}</{Inner}></{Outer}></body>')
Outer
Inner

Ian, curious how you're feeling about the direction overall.

I've been pretty convinced we should use Template in this way since November. I don't want to shut doors on use-cases though because we are still trying to figure everything out and I'm still trying to get the a full understanding of what else is going on.

• I want to document the interpolation rules better just like the attribute resolution, "content resolution" ? I guess?
• node() is a good starting point. I have to think about what the best way for that to work would be but regardless of the insides the final results should be equal when str-ified, ie. str(node(...)) == html(...)
• I need to cleanup and cut up this PR and then we can try to merge in the good parts as pieces.