Translation dashboard

[peterbe]

It's a bit of a beast and there are many things we can do but at this point, I'd rather not sink more time into it.
I also don't want to celebrate this as complete and perfect. Instead, I'd like to get this into main, ping a couple of savvy translators, and have them play with it.

Despite the roughness, this is 10x better than what translators have today, because, they have nothing of this sort.
The idea is that you can start with this to see which translated documents are in the most need to work. I.e. most differences and/or most out-of-date compared the en-US.

The next time to tackle would be to figure out which en-US documents have not been translated yet. That might be a separate table and once that comes in perhaps this current table has to become a "sub-page". I.e. you click "French" and then pick "State of existing translations" or "Documents in need of a first translation".

One thing at a time. But let's get the ball rolling without allowing ourselves to over-engineer this.

[peterbe]

Note-to-self; The getTranslationDifferences does a bunch of caching. It might not be necessary at all. The /_translations?locale=ko now only takes about 1.5s and it's probably not worth the hassle to try to cache the gathering of individual KS macros for en-US files.

[escattone]

Wow, this is a really nice step forward. I haven't looked at this in detail, but had some early feedback from my local test run.

[peterbe]

@Gregoor I don't know if you've had a chance to look yet but I hastily added a feature now so you can filter by differences. For example >=5 only includes documents that have 5 or more total KS differences.

[peterbe]

@Gregoor Do you mind if I merge it? I don't want to suck you into a lengthy review for something that's not nearly as important as Markdown. But it would be nice to have a sanity check and a skim-through of some of the code.
The biggest risk right now is that we land something imperfect that (translators) users won't accept that it's not perfect. But I hope to "launch" it to our super-user translators and have them play with it. There's no need for a big and loud launch.

[Gregoor]

Quick glance review, but I can't say that it was much more than a skim.

If it serves the translators purposes, I'm all for it, though I find it hard to gauge if the complexity we are taking on here is worth it. I.e. there is some custom search DSL parsing going on here, is that something we can maybe outsource, i.e. are there libraries we could rely on to accomplish this, that would also serve the translators needs?
Just one of the things I was wondering, but yeah like I said, I don't think this can be counted as a proper view yet.

[peterbe]

i.e. are there libraries we could rely on to accomplish this, that would also serve the translators needs?

This question is on my mind.
Perhaps KS macros is ultra-unique to us and there's not a lot of hope of finding a generic solution.
But surely, there much exists something that compares an original vs. translations where it can compute a difference in structure. I know we haven't found one yet and nobody's found a decent alternatives to even consider.
For example, if {{KsMacro('...')}} is so unique to us, perhaps if you change it (regex replace) to <MACROHERE> just before you send it into the lib, perhaps that could fit.

We did go into a deep deep tangent on comparing structure with cheerio. I.e. you turn the document into a "vector".
For example you [h2, p, p, h3, p, p, p, pre, h3, h2, pre, p, p, h3, p] and do that for both docs. Then you could maybe compare to such vectors and conclude that the structure is different. But the deeper we dug the harder we found it to be. So many nuances and "false positives". I don't think it's entirely over. You could perhaps go back to basics and just compare headings. I.e. [h2, h3, h3, h2, h3]. But that has problems too.
Let's not entirely give up! ...but I remember deciding that it got too complex for momentum so I scaled it back to be exclusively about KS macros. Just to get something where you can get a feel for how out-of-date a translation is.
Honestly, if translations were a higher priority and we had more time for this, I'd continue to dig.

Having said that, as mentioned in Slack/Zoom, I think we should extend the generic getTranslationDifferences to also spot differences in tags because they have an important meaning for sidebars. But then I'm thinking, perhaps the better solution is to NOT force translators to keep them in sync at all. The list of tags for a given French document could be implied instead of explicit. I.e. the translation document's front-matter's tags could simply be it's en-US parent's tags. Automatically. A challenge with that one, until June 30th we still have some translations that are "orphans" that we still build. So then you'd need to account for that.

So many things we can do. I'd rather merge this and incrementally move forward. Perhaps, from an overall translation quality point of view, the bigger fish to fry is to extend the dashboard to point out which en-US documents have 0 translations in the current chosen locale. That's a feature I'd like to look into next.

[peterbe]

I had it in my head that maybe we should say something on the dashboard itself that reminds users that this dashboard is a bit "beta". It's semi-officially supported because it's merge into main. But something to tell the story that it ain't perfect and not to expect it to be.
But I got writer's block trying to come up with something so eventually I just gave up.
Unless you think it's a good idea to just put "Translation dashboard (beta)" in the title.