Attribute resolution rules: replacement, merging, omitting, etc.

[ianjosephwilson]

What are the rules for how the attributes are resolved? I have been experimenting with different parsing/serializing ideas and I don't quite understand them.

Attribute Sources (Some? of them?)

• parsed as static (t"<input disabled>", t'<input value="1">')
• bound interpolation (t"<input disabled={disabled}>")
• bound template (t'<img alt="A picture of {img_alt}">')
• spread interpolation (t"<input {dict(disabled=True)}>")

Special cases/kinds (Some? of them?)

• style
• class
• aria
• data
• everything else

Merging

Should multiple input attributes be merged or replaced or allow omitting an attribute defined previously by another source?

If you have t'<div style="padding-left: 10px" {dict(style="padding-right: 20px")}></div>' I would think that you'd get something like <div style="padding-left: 10px; padding-right: 10px">.

Even more so for the case of classes:

def get_container_t(bg_red=True):
    attrs={'class': {'bg_red': bg_red}}
    return t'<div class="container" {attrs}></div>'

Finally in the case of boolean attributes I would expect to be able to "turn off" attributes with a later falsy value. This seems to work but only by accident because disabled=True also does not render the attribute.

NOT_SET = tuple()
def get_our_input_overrides(disabled=NOT_SET):
    attrs = {}
    if disabled is not NOT_SET:
        attrs['disabled'] = disabled
    return attrs
def get_input_t(attrs):
    return t'<input disabled {attrs}>'
assert get_input_t(get_our_input_overrides(disabled=False)) == '<input />'

Overriding issue

def override_with_spread(v, out):
    attrs = {'disabled': v}
    node = html(t"<input disabled {attrs} />")
    assert str(node) == out

def test_override_with_spread_true():
    override_with_spread(True, "<input disabled />")
def test_override_with_spread_false():
    override_with_spread(False, "<input />")
def test_override_with_spread_none():
    override_with_spread(None, "<input />")

def test_override_with_spread_empty_string():
    override_with_spread('', '<input disabled="" />')
def test_override_with_spread_string():
    override_with_spread('disabled', '<input disabled="disabled" />')

tdom/processor_test.py::test_override_with_spread_true FAILED                                                                                                                          [ 20%]
tdom/processor_test.py::test_override_with_spread_false PASSED                                                                                                                         [ 40%]
tdom/processor_test.py::test_override_with_spread_none PASSED                                                                                                                          [ 60%]
tdom/processor_test.py::test_override_with_spread_empty_string PASSED                                                                                                                  [ 80%]
tdom/processor_test.py::test_override_with_spread_string PASSED                                                                                                                        [100%]

========================================================================================== FAILURES ==========================================================================================
_______________________________________________________________________________ test_override_with_spread_true _______________________________________________________________________________

    def test_override_with_spread_true():
>       override_with_spread(True, "<input disabled />")

tdom/processor_test.py:1118: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

v = True, out = '<input disabled />'

    def override_with_spread(v, out):
        attrs = {'disabled': v}
        node = html(t"<input disabled {attrs} />")
>       assert str(node) == out
E       AssertionError: assert '<input />' == '<input disabled />'
E         
E         - <input disabled />
E         + <input />

tdom/processor_test.py:1115: AssertionError
================================================================================== short test summary info ===================================================================================
FAILED tdom/processor_test.py::test_override_with_spread_true - AssertionError: assert '<input />' == '<input disabled />'
======================================================================== 1 failed, 4 passed, 229 deselected in 0.18s =========================================================================

[davepeck]

Yes, we need to get very precise on the rules here.

The general idea is that whichever value of a given attribute comes rightmost should win, except for cumulative attributes like class, which I'd expect to merge via some sensible set of rules.

But I know that general idea needs to be fleshed out further; things aren't yet fully baked. Getting precise on the rules and updating the implementation and tests has been my highest priority TODO item, so if you have specific suggestions for the rules you'd expect, it would be very much appreciated!

Making your exploration even trickier, this appears to be a bug: html(t"<input disabled={True} />") and html(t"<input {dict(disabled=True)} />") should be equivalent even in the current build.

[ianjosephwilson]

I've taken a few stabs at this and its pretty easy to end up in the weeds. I think I almost broke my brain on it TBH. I guess this seems like an okay starting point. Let me know what you think. I will probably tweak things to make this work and then send a draft PR again and we could see which tests broke.

A Proposal:

• static_attrs -- parsed from the raw html: dict[str, str|None]
• named_attrs-- optional explicitly named attributes to be interpolated
• spread_attr -- an optional spread attribute that will contain something coercable to a dictionary
• dynamic_attrs = dict(chain(named_attrs.items(), spread_attr.items())) (just for discussion)
• attrs_str -- The html serialization of the attributes if rendering to html.

Reducing complexity

• Don't allow multiple spread attributes:
  t'<img {dict(alt=another_alt)} {dict(title=another_title)}>' --> ERROR
• named_attrs and spread_attr do not intersect:
  t'<img alt={alt} {dict(alt=another_alt)}>' --> ERROR
• In general dynamic attributes win over static attributes, regardless of position:
  html((t'<img {dict(alt="The field we avoided.")} alt="A field of weeds.">') == '<img alt="The field we avoided.">'
• In general aria and data are resolved last and clobber everything else:
  html(t'<img data-active="true" {dict(data=dict(active=False))} data-active={'maybe'}>') == '<img>'

Special cases

• Dynamic class attribute:
  • dynamic_attrs['class'] is None implies removal of attribute
    • 'class' not in attrs
  • dynamic_attrs['class'] is False ignored as before to support is_active and "active"
  • isinstance(dynamic_attrs['class'] , str) implies replacement
    • attrs['class'] = dynamic_attrs['class']
  • otherwise keeps most of the special class accumulation behavior, with static_attrs['class'] being split and merged in with dynamic_attrs['class'] based on the dynamic value
• Dynamic data attribute:
  • Data attrs with True are bare attributes, ie. <input disabled>
  • Data attrs with None or False are unset, ie. <input>
  • All other data attrs are just flattened and cast to str() in the expected way when generating attrs_str
• Dynamic aria attribute:
  • Aria attrs with True become "true", <input aria-selected="true">
  • Aria attrs with False become "false", <input aria-selected="false">
  • Aria attrs with None are unset, tt'<input aria-selected="true" aria=dict(selected=None)>' == <input>
  • All other aria attrs are just flattened and cast to str() when generating attrs_str.
• Dynamic style attribute:
  • None will unset the style attribute completely
  • str(), including "", will just replace whatever is in attrs
  • Otherwise style name and values will be rolled up as usual and replace whatever is in `attrs

Special special cases

• What do we do with static data and aria attributes? Just leave them in ? If we want to pass the dynamic blobs to a component will they colllide? Does data or aria ever get used as an actual attribute? <div data="madeup">.

Components

• We don't want to start serializing some things before going to components, we probably need to figure out what those are.
  • For example would we want to leave class as a list/tuple or would we stringify it before?
  • Would all data-* and aria-* attribute values be stringified? Do the data and aria blobs go on to the component?
  • If attributes are going to be removed/unset would they be omitted before calling a component or would the component receive whatever their unsetting value is ?

Improvements

• We could possibly pre-process the static data-*, aria-*, style and class attributes if we want to try to merge everything in a more complete way.

More cutting?

• If we wanted reduce the scope even more:
  • don't have special support for the data or aria concepts at all
  • class could be a more regimented structure or even just strictly None|str|dict[str,bool]

Actual changes

I think most of the examples would keep working this is all about just making the rules more strict and trying to pin down all the edge cases. The major changes would be:

• Enforce single spread attr
• Enforce distinct set of keys across all dynamic sources
• Tweak some of the special attribute handling cases to allow unsetting things.

[davepeck]

Thanks @ianjosephwilson -- it's really helpful to see your instincts here and compare notes!

Informally speaking, and ignoring special-case attributes, do you think it'd be fair to describe your instincts as drawn from python function invocation and unpacking? Specifically, I'm thinking of your instinct that attribute names should not be repeated, with a possible exception in the rules around "dynamic" attributes. Just to spell it out: if we have a function with the signature f(**kwargs), it's fine to call with f(**{"a": 1}, b=2, **{"c": 3}) but f(**{"a": 1}, c=2, **{"c": 3}) and related forms lead to a TypeError.

I suppose my instincts draw more from Python dictionary unpacking into dictionaries, that is syntax like {**d1, "k": v, **d2} where the rightmost values always win. Specifically, I've been thinking of every attribute source in tdom — static, named and spread — as just another dictionary that participates in a left-to-right merge. In the case of static and named, that dictionary just has one key. You're welcome to have as many spreads, or repeats of a given attribute, as desired.

I think this approach also matches the mental model a little better for people decamping from React/JSX. In javascript, {...obj1, ...obj2} has the same left-to-right behavior and JSX component invocations are the same aka if const props = {foo: 42}, then <Component foo={99} {...props} /> invokes the component with 42, while <Component {...props} foo={99} /> invokes it with 99. etc.

For the special individual attributes — class and style — I imagine defining a custom merge function for each that can be used in a call to reduce().

My instincts are much fuzzier about handling of the data-* and aria-* vs special-purpose generic data and aria attributes in some kind of merge. I left a pretty fuzzy set of tests in the code as a stake in the ground, but clearly not good enough today. I need to think more about your approach here. (There is other complexity related to aria-* attributes, and what boolean values should convert to in practice, etc.)

My instincts are also fuzzy when it comes to value removal/clearing, and I like your thoughts there as a place to start. Taking a cue from JSX, if you pass null or undefined to a built-in tag (like <div title={null} />) it won't appear in the resulting DOM at all. On the other hand, when invoking a component (like <Component k={null} />) that null is explicitly passed to the component invocation. (There may be edge cases here for special-case attributes in React JSX land too; I'm not sure.)

[ianjosephwilson]

Thanks @ianjosephwilson -- it's really helpful to see your instincts here and compare notes!

I think there are a lot of html builders out there in Python that just make 10000 function calls to build an html template. With t-strings, this neat parsing trick and caching I think this library could be decently fast (for Python). So given that this is a new API and no one depends on it and performance might be a critical factor then we might want to keep the API as simple as possible. On the other hand though if this was made pluggable then we could just turn on our favorite attribute accumulator and Templat' away.

Informally speaking, and ignoring special-case attributes, do you think it'd be fair to describe your instincts as drawn from python function invocation and unpacking? Specifically, I'm thinking of your instinct that attribute names should not be repeated, with a possible exception in the rules around "dynamic" attributes. Just to spell it out: if we have a function with the signature f(**kwargs), it's fine to call with f(**{"a": 1}, b=2, **{"c": 3}) but f(**{"a": 1}, c=2, **{"c": 3}) and related forms lead to a TypeError.

Yes this was the pattern I was thinking of. Sort of like this:

def element_attrs(static_attr_dict: dict[str, str|None], named_dynamic_attr0: object|None, named_dynamic_attr1: object|None, ..., **dynamic_spread_attr: dict[str, object|None]):
    pass

I suppose my instincts draw more from Python dictionary unpacking into dictionaries, that is syntax like {**d1, "k": v, **d2} where the rightmost values always win. Specifically, I've been thinking of every attribute source in tdom — static, named and spread — as just another dictionary that participates in a left-to-right merge. In the case of static and named, that dictionary just has one key. You're welcome to have as many spreads, or repeats of a given attribute, as desired.

This pattern/concept was my first instinct as well then I changed my mind to the python function call pattern. This feels like a "program-all-the-things!!" type trap but I don't know. I guess in the VDOM I used there were some static props pre-processed and then a blob of attrs you provided. The static stuff would be clobbered by the dynamic stuff except class. I worry about pre-baking in a bunch of functionality that we support but doesn't get used that often. On the other hand it is nice to have the resolution right there in the template itself even if it can be a confusing.

I think this approach also matches the mental model a little better for people decamping from React/JSX. In javascript, {...obj1, ...obj2} has the same left-to-right behavior and JSX component invocations are the same aka if const props = {foo: 42}, then <Component foo={99} {...props} /> invokes the component with 42, while <Component {...props} foo={99} /> invokes it with 99. etc.

For the special individual attributes — class and style — I imagine defining a custom merge function for each that can be used in a call to reduce().

My instincts are much fuzzier about handling of the data-* and aria-* vs special-purpose generic data and aria attributes in some kind of merge. I left a pretty fuzzy set of tests in the code as a stake in the ground, but clearly not good enough today. I need to think more about your approach here. (There is other complexity related to aria-* attributes, and what boolean values should convert to in practice, etc.)

My instincts are also fuzzy when it comes to value removal/clearing, and I like your thoughts there as a place to start. Taking a cue from JSX, if you pass null or undefined to a built-in tag (like <div title={null} />) it won't appear in the resulting DOM at all. On the other hand, when invoking a component (like <Component k={null} />) that null is explicitly passed to the component invocation. (There may be edge cases here for special-case attributes in React JSX land too; I'm not sure.)

Yeah I think there is an issue of "overriding" defaults easily but also making sure whatever is built is deterministic so we don't have template outputs changing all the time because the input is subtly different in ways people don't understand. Both for caching and if there was a ever "diffs" being made for partial re-rending. Also they might need an exact format (within reason). An example is that for boolean attributes you can have <input disabled /><input disabled="" /><input disabled="disabled" /> all conceptually mean the same thing. We probably have to be careful that we can generate the HTML that people expect even if it does't seem to matter because maybe wherever they're sending it needs exactly one version. So they can't just set title to "" they need the attribute gone.

[ianjosephwilson]

Here is one of those weird cases that seems wrong but I can't imagine someone would want.

html(t'<p qty="1" {dict(qty="2")} qty="3" {dict(qty=4)} qty={None} qty="5"></p>')

This brings up a point I already forgot about. The attributes are parsed upfront and clobbered before the spread attributes can be interpolated. I think the last value is retained but the first set call is used for ordering. The attributes should probably be stored in kv tuples anyways. Ie. t'<p class="red" class={class}></p>' I think would never be possible to merge because the dynamic class clobbers over the static class before attribute resolution.

My older idea was to model a set of sequences after templates like this with the same n seqs and n-1 interpolates and then just iterate and process into the final final attrs dictionary:

@dataclass
class AttributesTemplate:
    seqs: tuple[tuple[str, str|None], ...]
    interpolations: tuple[Interpolation, ...]

[pauleveritt]

Hey @ianjosephwilson nice to see someone from Pyramid/wired hanging out over here. 😀

Some (late) comments:

Dave does a good job of framing this as function invocation vs. dict semantics.

You're right to call out the mission: simple and fast, or pluggable and featureful? Confession: I'm a chaos monkey that generates too much of the latter.

Regarding performance, there probably are a number of techniques we can do longer-term. Obviously the JSX world has multiple generations of lessons on this, moving static stuff out, etc. Dave and I talked about ideas to move some of the work from rendering to parsing.

Your first post quietly brings in a case we are uncertain of: should interpolations need a Node or is a Template enough?

first = html(t"<div>Hello</div>")
second = t"<div>Hello</div>"

result = html(t"<main>{first} and {second}")

It sucks repeating html() all the time. But it seems to be the best way to tell tools the language of the t-string, so it's at minimum a best-practice.

[davepeck]

PS @ianjosephwilson and @pauleveritt sorry for my very sluggish reply here. Partly, I've been turning over this conversation in my head this week, and partly I'm just super heads-down on other stuff. I'll be back! :-)

[ianjosephwilson]

@davepeck No problem. I have been enjoying trying out t-strings probably a little too much. I tried out a different parsing approach I'll try to start another discussion about.

After trying out even more approaches I think you're right for now about sticking with the clobbering from left to right strategy. There are some confusing edge cases but I think it is the easiest to implement and to explain (after those are resolved).

@pauleveritt The few the proud. Hopefully this whole htmx revival + t-strings will bring back more python web dev or at least keep it alive.

Maybe typing is for another discussion but I couldn't find a lot of information about adding metadata to templates. It is kind of a bummer for the library too. If we could mark the templates then we could easily scan for them and process them beforehand. (my original discussion topic) Have you considered using something like Annotated[Template, TDom], ie. TDom = Annotated[Template, TDOM] where TDOM is a marker class and TDom is an alias people could use? I was using some mutant version of that with wired and it was working pretty well with venusian. Ie.

from tdom.types import TDom

def get_product_image_t(product_image_do: ProductImageDO) -> TDom:
    """ Create a product image html template fragment consumable by the tdom library. """
    return t'<div><img alt={truncate(product_image_do.description, 100)}></div>'

[davepeck]

After trying out even more approaches I think you're right for now about sticking with the clobbering from left to right strategy. There are some confusing edge cases but I think it is the easiest to implement and to explain (after those are resolved).

Ah, good to hear it. Even after mulling it over, this is still where my instincts lie too.

Maybe typing is for another discussion but I couldn't find a lot of information about adding metadata to templates

We punted on this when we wrote PEP 750 -- I think it's going to take the community a bit of time to decide how all this works. But yes, something like Annotated[Template, "html"] or (if you believe there will be different HTML + t-strings libraries in the future, potentially with different rules for where and what types of interpolations are accepted), Annotated[Template, "html-tdom"] makes the most sense to me. @koxudaxi (one of the PEP 750 authors) also built the prototype t-linter tool that accepts these types of annotations.

[koxudaxi]

@ianjosephwilson
Not directly about tdom, but regarding IDE support for t-strings.
Since t-strings require writing literals, I believe IDE and linter support is essential.
What t-linter does is provide syntax highlighting for sub-languages like SQL and HTML within t-strings, which is then delivered to IDEs via LSP. If we could support custom sub-languages through plugins, it would also reduce confusion for tdom users.
This is a proof of concept, and the discussion around how to define types and dialects has stalled. Most likely, once a killer app emerges (which might be tdom, or might be something else) and we understand users' actual pain points, we'll be able to establish more rigorous rules. https://discuss.python.org/t/language-hints-for-pep-750-template-strings/94311

[pauleveritt]

I'm very eager to return to this. I'm spending a lot of time writing tdom now (in fact, on a potential killer app) and I value PyCharm knowing the HTML part, but unofficially.

[ianjosephwilson]

I guess the final outcomes of attribute resolution (the clobbering left to right and special cases) as I see them would be this:

resolved value	html	component	notes
None	omitted	None	this allows for clearing previously set attributes
BARE_ATTR	bare attribute	True	Store the None parsed by HTMLParser as a special marker to prevent polluting the attributes, I kind of was thinking just internally storing it as True but maybe we don't want to lose that information? It is also just so easy for the meanings to pollute each other.
True	bare attribute	True	--
False	omitted	False	--
object	str(object)	object	--
str	str	str	--

[davepeck]

This is good, and feels correct to me. Still need to consider your notes on the BARE_ATTR case... it does feel like what HTMLParser treats as None is more logically treated as True when passed to a component. (Related: I'm wondering if there's a way to really clarify this stuff in the processor, and possibly in the node hierarchy too — when do you have str | None for values vs. when do you have component-friendly values?)

[ianjosephwilson]

Thanks for merging those changes. There are a few more attribute use-cases I think that still need a bit more smoothing out.

One immediate issue is that any sort of merging is going to mean that we probably need to store the attributes as a key/value sequence instead of a dictionary so that a static class is not clobbered with a dynamic class because of a common case (I think) like this:

# We have a default margin that is always set and then we conditionally apply one background color or the other.
background_color = {'bg-near-white': item.active, 'bg-gray': not item.active}
t'''<div class="ma2" class="{background_color}">{item.title}</div>'''

This is related to the other discussion about having a structured post-parsing structure:

parsed attributes with placeholders -- (parser pipeline) --> parsed attributes with placeholders removed -- (processor pipeline) --> attributes resolved (and interpolated)

I think keeping all the information together before resolution kind of makes sense anyways because then things could be changed later and we don't have to redo everything, ie. no data was lost when parsing from the t-string into the templated "structure". Specifically something like this:

From HTMLParser: list[tuple[str, str|None]] (dupes in order of appearance)
After parsing but before processing: list[tuple[str|None, str|Template|None]] (dupes in order of appearance, placeholders removed)
After processing (as html): dict[str,str|None] (no dupes)

So I kind of think this needs to be resolved before we can start on figuring out how to merge at least the class attribute. Specially what is the best format to use to store the k/v pairs with duplicates retained in a templated "structure"?

[ianjosephwilson]

We arrived at a solid stopping point with #89 , maybe need a few tests but I'm closing this discussion.