Skip to content

Refactor CLDR data import #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mlocati merged 1 commit into from
Mar 18, 2025
Merged

Refactor CLDR data import #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 67 additions & 12 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
name: Tests

env:
BUILD_CLDR_VERSION: "47"

on:
push:
branches:
- main
tags-ignore:
- "**"
pull_request:
branches:
- main
Expand All @@ -13,18 +18,21 @@ jobs:
name: Check PHP coding style
runs-on: ubuntu-latest
steps:
- name: Setup PHP
-
name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: "8.1"
extensions: mbstring, zip
tools: composer:2, php-cs-fixer:3
coverage: none
- name: Checkout
uses: actions/checkout@v2
- name: Check PHP coding style
run: |
php-cs-fixer fix --path-mode=intersection --config=./.php-cs-fixer.dist.php --dry-run --using-cache=no --diff --show-progress=dots --verbose --no-interaction --ansi .
-
name: Checkout
uses: actions/checkout@v4
-
name: Check PHP coding style
run: php-cs-fixer fix --config=./.php-cs-fixer.dist.php --dry-run --using-cache=no --diff --show-progress=dots --verbose --no-interaction --ansi

phpunit:
name: Run PHPUnit tests
needs: php-coding-style
Expand All @@ -45,24 +53,71 @@ jobs:
- "8.0"
- "8.1"
- "8.2"
- "8.3"
- "8.4"
include:
- os: windows-latest
php-version: "5.6"
- os: windows-latest
php-version: "7.4"
- os: windows-latest
php-version: "8.1"
php-version: "8.4"
runs-on: ${{ matrix.os }}
steps:
- name: Setup PHP
-
name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php-version }}
tools: composer:v2
coverage: none
- name: Checkout
uses: actions/checkout@v2
- name: Install Composer dependencies
-
name: Checkout
uses: actions/checkout@v4
-
name: Install Composer dependencies
run: composer update --no-progress --no-suggest --optimize-autoloader --ansi --no-interaction
- name: Run PHPUnit
-
name: Build CLDR data
run: php ./bin/import-cldr-data ${{ env.BUILD_CLDR_VERSION }}
-
name: Run PHPUnit
run: composer --no-interaction run-script test

commit:
name: Commit CLDR data
runs-on: ubuntu-latest
needs: phpunit
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
-
name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.4
tools: none
coverage: none
-
name: Checkout
uses: actions/checkout@v4
-
name: Build CLDR data
run: php ./bin/import-cldr-data ${{ env.BUILD_CLDR_VERSION }}
-
name: Check changes
id: check-changes
run: |
git add --all src/cldr-data
if git diff-index --name-status --exit-code HEAD src/cldr-data; then
echo 'No changes detected.'
else
echo 'Changes detected.'
echo 'commit=yes' >> $GITHUB_OUTPUT
fi
- name: Commit changes
if: steps.check-changes.outputs.commit == 'yes'
run: |
git config --local user.name 'GitHub Actions'
git config --local user.email noreply@github.com
git commit -m 'Update CLDR data'
git push
3 changes: 1 addition & 2 deletions .php-cs-fixer.dist.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
])
->append([
__DIR__ . '/bin/export-plural-rules',
__DIR__ . '/bin/import-cldr-data',
])
->in(__DIR__)
)
Expand Down Expand Up @@ -245,8 +246,6 @@
'phpdoc_no_access' => true,
// No alias PHPDoc tags should be used.
'phpdoc_no_alias_tag' => true,
// `@return void` and `@return null` annotations should be omitted from PHPDoc.
'phpdoc_no_empty_return' => true,
// `@package` and `@subpackage` annotations should be omitted from PHPDoc.
'phpdoc_no_package' => true,
// Classy that does not inherit must not have `@inheritdoc` tags.
Expand Down
58 changes: 1 addition & 57 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,63 +172,7 @@ the resulting formulas will be in this format:

### Generating the CLDR data
This repository uses the CLDR data, including American English (`en_US`) json files.
In order to generate this data, you can use Docker.
Start a new Docker container by running

```sh
docker run --rm -it -v path/to/src/cldr-data:/output alpine:3.13 sh
```

Then run the following script, setting the values of the variables accordingly to your needs:

```sh
# The value of the CLDR version (eg 39, 38.1, ...)
CLDR_VERSION=39
# Your GitHub username (required since CLDR 38) - see http://cldr.unicode.org/development/maven#TOC-Introduction
GITHUB_USERNAME=
# Your GitHub personal access token (required since CLDR 38) - see http://cldr.unicode.org/development/maven#TOC-Introduction
GITHUB_TOKEN=

if ! test -d /output; then
echo 'Missing output directory' >&2
return 1
fi
apk -U upgrade
apk add --no-cache git git-lfs openjdk8 apache-ant maven
CLDR_MAJORVERSION="$(printf '%s' "$CLDR_VERSION" | sed -E 's/^([0-9]+).*/\1/')"
SOURCE_DIR="$(mktemp -d)"
DESTINATION_DIR="$(mktemp -d)"
git clone --single-branch --depth=1 "--branch=release-$(printf '%s' "$CLDR_VERSION" | tr '.' '-')" https://github.com/unicode-org/cldr.git "$SOURCE_DIR"
if test $CLDR_MAJORVERSION -lt 38; then
git -C "$SOURCE_DIR" lfs pull --include tools/java || true
ant -f "$SOURCE_DIR/tools/java/build.xml" jar
JARFILE="$SOURCE_DIR/tools/java/cldr.jar"
DESTINATION_DIR_LOCALE="$DESTINATION_DIR/en_US"
DESTINATION_FILE_PLURALS="$DESTINATION_DIR/supplemental/plurals.json"
else
if test -z "${GITHUB_USERNAME:-}"; then
echo 'GITHUB_USERNAME is missing' >&2
return 1
fi
if test -z "${GITHUB_TOKEN:-}"; then
echo 'GITHUB_TOKEN is missing' >&2
return 1
fi
printf '<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"><servers><server><id>githubicu</id><username>%s</username><password>%s</password></server></servers></settings>' "$GITHUB_USERNAME" "$GITHUB_TOKEN" > "$SOURCE_DIR/mvn-settings.xml"
mvn --settings "$SOURCE_DIR/mvn-settings.xml" package -DskipTests=true --file "$SOURCE_DIR/tools/cldr-code/pom.xml"
JARFILE="$SOURCE_DIR//tools/cldr-code/target/cldr-code.jar"
DESTINATION_DIR_LOCALE="$DESTINATION_DIR"
DESTINATION_FILE_PLURALS="$DESTINATION_DIR/supplemental/plurals/plurals.json"
fi
java -Duser.language=en -Duser.country=US "-DCLDR_DIR=$SOURCE_DIR" "-DCLDR_GEN_DIR=$DESTINATION_DIR_LOCALE" -jar "$JARFILE" ldml2json -t main -r true -s contributed -m en_US
java -Duser.language=en -Duser.country=US "-DCLDR_DIR=$SOURCE_DIR" "-DCLDR_GEN_DIR=$DESTINATION_DIR/supplemental" -jar "$JARFILE" ldml2json -s contributed -o true -t supplemental
mkdir -p /output/main/en-US
cp $DESTINATION_DIR/en_US/languages.json /output/main/en-US/
cp $DESTINATION_DIR/en_US/scripts.json /output/main/en-US/
cp $DESTINATION_DIR/en_US/territories.json /output/main/en-US/
mkdir -p /output/supplemental
cp "$DESTINATION_FILE_PLURALS" /output/supplemental/
```
In order to generate this data, you can use the `bin/import-cldr-data` CLI command.


## Support this project
Expand Down
Loading