Add multi-language support to Windows installer

[Dimi20cen]

Summary

This PR restores multi-language support to the Windows installer, addressing a regression from our legacy version. It implements a modern, industry-standard translation workflow using .po files and ISO locale codes.

All user-facing strings have been externalized from kolibri.iss into language files. Additionally, all installer related files (the main script, translations, and dependencies) have been moved into a new installer/ directory for better organization.

Changes

• Externalized Strings & ISO Standard: User-facing text has been moved from kolibri.iss into separate .isl files. The system now uses ISO locale codes (e.g., es_ES, de_DE) instead of English language names.
• Centralized Language Definitions: Added definitions.py to map ISO codes to their corresponding Inno Setup language names and Microsoft Language IDs. This automates the configuration of [LangOptions] and removes the need for manual ID lookup.
• Conversion Scripts:
  • isl_to_po.py: Converts Inno Setup files to .po. When scaffolding a new language, it automatically detects and merges standard Inno Setup translations (e.g., "Next >", "Cancel") so translators only need to focus on Kolibri-specific strings.
  • po_to_isl.py: Compiles translated .po files back into .isl format for the installer, automatically injecting the correct Language IDs from the definitions file.
• Automated Build Process: The build-installer-windows command automatically compiles all .po files found in installer/translations/locale/ into the required .isl format before building the executable.
• New Makefile Targets:
  • new-language: Scaffolds a new language (e.g., make new-language LANG=es_ES), creating a .po file pre-filled with standard translations.
  • translations-export-source: Updates the source messages.po from the master en.isl.
  • translations-compile: Converts all localized .po files into .isl files ready for packaging.

References

Fixes #181

Translation Workflow

Prerequisites

1. Install dependencies again, (make dependencies).
2. Run

make get-whl whl="https://github.com/learningequality/kolibri/releases/download/v0.18.1/kolibri-0.18.1-py2.py3-none-any.whl"

3. Run make pyinstaller to build the application.
4. Have Inno Setup installed.
5. Have polib installed, pip install polib.

1. Adding a New Language (Developer Task)

This process scaffolds the files for a new language using its ISO locale code.

1. Register the Language:
   Open installer/translations/definitions.py and add the new ISO code to the LANG_DEFINITIONS dictionary. You must map it to the corresponding Inno Setup language name and Microsoft Language ID.

     "es_ES": {"inno_name": "Spanish", "id": "$040A"},

   This command runs the create_new_language.py script, which creates installer/translations/Spanish.isl. The file will be pre-filled with standard Inno Setup translations if available, otherwise it will be a copy of the English template with empty values for translation.

2. Scaffold the PO File:
   Run the make target with the ISO code.

   make new-language LANG=es_ES

   • This checks your local Inno Setup installation for a standard translation file (e.g., Spanish.isl).
   • It creates installer/translations/locale/es_ES/messages.po.
   • Note: If a standard Inno file was found, standard strings (like "Next >" or "Cancel") will be pre-filled in the PO file, so translators only need to focus on Kolibri-specific strings.

3. Enable the Language in the Installer Script:
   Open installer/kolibri.iss and add a new entry in the [Languages] section. Point to the .isl file (note: this file doesn't exist yet; it will be generated during the build).

   [Languages]
   Name: "en"; MessagesFile: "translations\en.isl"
   Name: "de_DE"; MessagesFile: "translations\locale\de_DE\de_DE.isl"
   ; Add the new line for Spanish
   Name: "es_ES"; MessagesFile: "translations\locale\es_ES\es_ES.isl"
   

4. Commit Changes: Commit definitions.py, kolibri.iss, and the new messages.po file.

2. Syncing with Crowdin (Developer Task)

1. Update Source Strings:
   If en.isl has changed, regenerate the source PO file:

   make translations-export-source

   Upload installer/translations/locale/en/messages.po to Crowdin as the Source File.

2. Upload Existing Translations:
   If you just scaffolded a new language (Step 1) and it contains pre-filled standard translations, upload installer/translations/locale/es_ES/messages.po to Crowdin as Existing Translations. This ensures translators don't re-do work Inno Setup has already done.

3. Translating (Translator Task)

Translators work on Crowdin to translate the custom Kolibri strings (and any standard strings that were missing).

4. Integrating Translations (Developer Task)

1. Download Translations:
   Download the completed .po file for the language from Crowdin.

2. Update Repository:
   Overwrite the local file at installer/translations/locale/es_ES/messages.po with the version from Crowdin. Commit the change.

5. Building the Installer (Developer Task)

The build process automatically converts the PO files into the ISL format required by Inno Setup.

1. Run the Build:

   make build-installer-windows

2. What Happens Behind the Scenes:

   • The translations-compile target runs automatically.
   • It iterates through every folder in installer/translations/locale/.
   • It converts messages.po into a compiled .isl file (e.g., es_ES.isl).
   • It automatically injects the LanguageID and LanguageName from definitions.py into the .isl file.

3. Verify:
   Run the generated installer in dist-installer/. You should see the language selection dialog, and selecting the new language should switch the UI text accordingly.

[rtibbles]

I think this has most of the right pieces in place! Just a couple of clarifications that came to mind - and I'm not entirely sure we need all of the code here with the right workflow in place.

[rtibbles]

I don't think we need this script. The workflow we would most likely use is to take the English isl file, convert it to a po file then upload it to Crowdin (the translation platform that we use).

We would then download the translated po files for each language, and convert those to isl files for each language. I think your two conversion scripts should handle all that is needed in that case?

[Dimi20cen]

take the English isl file, convert it to a po file then upload it to Crowdin

If we don't use create_new_language.py, then the translator would have to re-translate even the standard strings that InnoSetup already provides (for languages like spanish).

Instead if we use create_new_language.py, then only the custom strings would need to be translated.

Do I miss something?

[rtibbles]

Crowdin lets you upload existing translations for a source as well - so if we have the original inno translations, we can convert them to a po file for upload.

My basic thought is that I'd like us to specify which languages we want to support, and then conditionally upload already existing translations for those languages for which translations already exist. It is also possible that I'm just misreading the workflow you're intending.

[rtibbles]

Might be easier to use the locale code as the filename en.isl rather than the language name? Especially as we have the language name encoded in here anyway!

[Dimi20cen]

Currently, create_new_language.py uses the language name (e.g. Spanish) to find the packaged translation in C:\Program Files (x86)\Inno Setup 6\Languages.

We could modify the create_new_language.py script to take a locale code as input, and then use an internal dictionary to look up the Inno Setup Filename it needs to find.

Something like that

LOCALE_TO_INNO_NAME = {
    "de": "German",
    "es": "Spanish",
    "fr": "French",
    "it": "Italian",
    # ...
}

[rtibbles]

Yeah, that works for me - we end up doing this kind of mapping quite a lot for translations, because every translation system uses something different. We use the locale (e.g. es_ES) canonically, because it is the most specific, whereas just saying "Spanish" is very ambiguous - is that Spanish from Spain, Spanish from Argentina, Spanish from Latin America?

So, essentially for each locale code that we want to support, we would also make a note of which Inno Setup language name we would pull translations from.

[rtibbles]

Where do we find out these values?

[Dimi20cen]

They can be found here: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c

I was thinking of adding this to the readme, when this PR is complete

[rtibbles]

That would be great - I suppose we could even add this information to the locale lookup object that we're defining above:

LOCALE_INFO = {
    "de": {
        "inno_setup_name": "German",
        "ms_languageid": "0x0007",
    },
    # ...
}

If we wanted to get really fancy, we could fetch the page and parse the HTML table - but that seems very unneccessary!

[rtibbles]

What does languagecodepage mean?

[Dimi20cen]

It specifies the Windows ANSI code page that the language file text is encoded in. It tells Inno Setup how to correctly read and display the localized strings if the file isn’t in Unicode (UTF-8).

However how that the files are UTF-8, its not needed.
We can either remove it when creating a new language or maybe we can leave it as is

[rtibbles]

I think if we can be sure to encode in UTF-8 that seems like the best solution!

[rtibbles]

I think the main distinction to keep in mind is that we treat English as the "source", and all other languages as the "translations".

So, the workflow is, we upload every message we need in English in a po file - but then for some languages we will essentially have 'pre-filled' translations based on what InnoSetup already provides (either from its built in translations or community supported translations).

[rtibbles]

This is looking pretty good to me! I'll do a quick test locally to make sure I can follow the workflow.

[rtibbles]

Naturally, when I said I would do this... I forgot the fact that I don't have a Windows dev machine, so there are significant parts of the workflow that I can't actually do!

@Dimi20cen as a final step here, could you add all the supported languages for Kolibri - you can see the full list of codes here: https://github.com/learningequality/kolibri/blob/develop/kolibri/utils/i18n.py#L47

If you need any more information about the languages, then this JSON file should have it: https://github.com/learningequality/kolibri/blob/develop/kolibri/locale/language_info.json

Feel free to just focus on the intl-codes for now - I can add the extra mapping for going into the crowdin language code space!

[rtibbles]

I think this is looking good, I'll have to run this on my windows machine to be absolutely sure, but I think it might be simpler just to merge this and iterate from there!