Easier way to specify AABBAABB patterns

[kalekundert]

I find that it's pretty common to have a range of (say) rows where one parameter varies on every row, one varies every other row, one varies every 4 rows, etc. Right now there's not really a good way to specify this. You can use patterns for the first case and ranges for the last case, but for the intermediate cases you have to just list every well with commas.

I'd like a more semantic way to express this, but it also needs to be something intuitive (because I don't want the format to turn cryptic). Right now, I'm not really sure what that would be. Some ideas:

• [row.A-H.xxoo]: Here x means that a well is in use, and o means that it's not. I could use other characters as well. In order to distinguish between regular values and patterns like this, I'd have to require values to be scalar. It's nice that I could specify a range once, and then a bunch of different masks within it, e.g.:

  [row.A-H]
  xo.attr = '1'
  ox.attr = '2'
  xxoo.attr = '3'
  ooxx.attr = '4'
  xxxxoooo.attr = '5'
  ooooxxxx.attr = '6'
  

  You could imagine extending this syntax to multiple dimensions, e.g. [well.A1-B12.xo-ox] specifies a pattern similar to [irow]. I might even allow [row.A-B.xo-ox] as an exact equivalent to [irow]. This is a bit interesting because it specifies a 2D pattern for rows, which up until now have been intrinsically 1D. But the semantics are pretty clear.

  I'm a bit worried that this syntax is getting a bit cryptic, and that it would be hard for someone who knew nothing about the format to look at and grok. I might have to do some research and see what people think.

If I do add some sort of syntax like this, I might want to remove the original pattern syntax and the irow/icol groups. That would of course be a massive backwards-incompatible change, and would require jumping to v4.

[kalekundert]

I think xx__xx__ or oo__oo__ (inverting the meaning of o) might be better than xxooxxoo. Certainly, the underscores make it easier to see the pattern at a glance. I like that the o character kinda looks like a well. Not sure what other characters might be good.

Here's the same example from above, just to see how it looks:

[row.A-H]
x_.attr = '1'
_x.attr = '2'
xx__.attr = '3'
__xx.attr = '4'
xxxx____.attr = '5'
____xxxx.attr = '6'

[row.A-H]
o_.attr = '1'
_o.attr = '2'
oo__.attr = '3'
__oo.attr = '4'
oooo____.attr = '5'
____oooo.attr = '6'

[kalekundert]

One issue with this syntax is that it's not really possible to unambiguously distinguish patterns from regular attributes. For example, is this a pattern that matches every well, or is x an attribute with the value {'y': 1}?

row.A-H.x.y = 1

Some thoughts on how to disambiguate this:

• Forbid attribute names that consist entirely of x and _. This feels fragile, though.
• Require values to be scalar (e.g. {'y': 1} would not be a legal value).
  • This would allow the depth of the data structure to distinguish between values and patterns.
  • It feels like an unnatural restriction, though, and most of the time there would be no reason for it.
  • That said, putting lists/dicts in dataframes in something of an anti-pattern.
• Use a different syntax for for introducing ranges that will have patterns, e.g. A--H, 'A~H', pattern.A-H, etc.
  • The first two seem too similar to A-H; I think it would be confusing to have things that look so similar but mean different things. That said, you wouldn't ever need to specify the same range in two different ways in the same file: if you have A--H already, you can do A--H.x instead of A-H.
  • The pattern option is clear, but it feels backwards to me. The pattern is a refinement on the range; it should be mentioned after the range.

[FitzsimmonsCM]

• In terms of the syntax for AABBAABB patterns, comment 1 (xxooxxoo) is more intuitive to me than xx__xx__. However, I would suggest more intuitive pairs (a/b) or (x/y) rather than x/o.
• For the syntax in comment 3, pattern.A-H is the most intuitive, and would be easier for non-programmers or beginner programs to understand