single segments of ruby text shouldn't be broken up

[rokke-git]

if you hover over ruby text, it continues reading from ruby text along the line to match words, which is good. however, it shouldn't match half of a ruby string (just like it wouldn't match half a character).

---

[birtles]

That's an interesting issue. I think matching on part ruby is still helpful however. If you had something like:

公益財団法人日本卓球協会[こうえきざいだいほうじんにほんたっきゅうきょうかい]

marked up as a single ruby, I think it's useful to be able to look up the parts?

[rokke-git]

personally, I disagree. first of all, I think people often break it up more so that the furigana lines up closer to the kanji it's denoting (eg 公益(こうえき)財団法人(ざいだいほうじん)日本卓球協会(にほんたっきゅうきょうかい) is a little easier to read than 公益財団法人日本卓球協会(こうえきざいだいほうじんにほんたっきゅうきょうかい)). but the main reason I disagree is twofold:

1. greedy matching does not work well with long strings of hiragana. japanese has lots of homophones, especially when you have no tone info and can't hear word breaks. when you see the kanji, you get to replace that missing info with the meaning of the word; and your greedy algorithm is able to restrict matches to words with the same meaning. it's not really that great for semantically segmenting text
2. (this is the bigger one) you can look up the parts in the ruby base. I'm not saying to treat <rb> strings as one unit, just <rt> ones. In the base, your tool has the kanji it needs to return the correct results; but it looses that context when you're only looking at the hiragana.

there are some arguments I could see for being able to parse half a <rt>, like maybe in a fantasy setting there's a spell 水球(ウォーターボール), but I still think it's better to treat it as one unit. maybe if it contains punctuation, you could break it up (like 水球(ウォーター・ボール)), but I think in the majority of cases it's better to treat it as one unit so that you're not returning a bunch of half-words.

(edit: actually your dictionary already defines ウォーターボール, that's even more of an argument for not breaking ruby text up)

[birtles]

Ok, I think you've convinced me. The next step is just to work out how invasive implementing that behavior would be. The ruby traversal code (including the source context handling) is a bit gnarly so hopefully this doesn't make it worse—perhaps it will make it simpler!

[birtles]

I had a look at implementing this but it turns out to be quite involved. Normally the result of scanning text is a string we can pass to the dictionary to look up but now we need to also pass along "(dis)allowed split points" or otherwise embed metadata preventing the string being concatenated mid transcription. Still do-able but a bit more involved than I first thought.

[rokke-git]

hmm, if you're just passing a giant string to the dictionary, then how are you de-inflecting? I would have thought by the time it's getting sent to a dictionary some kind of pipeline to find the word boundaries for de-inflection would have been passed through

[birtles]

By dictionary I mean "backend". On the backend we progressively shorten the string and try potential deinflections at each point. So basically we just need to pass some extra metadata to the backend so it knows where possible shortening boundaries are. We already have limitations on where things can be shortened so we don't split ようおん like しゃ for example.

[rokke-git]

I went and found some actual punny/chuuni spell names from the wild. the first one works currently and would still work if we only let <rt> break on center dots; but the second one doesn't work at all (and prob wouldn't work after this either, but at least it wouldn't be returning false results)

「秘技！　都市部の土飛沫《アーバン・スプラッシュ》！」

「約束された(エクス・)、狩り場の剣(カリバー)！」

[birtles]

Made up a very draft PR at #2795 outlining the basic structure.