Update Norway counties to match GeoIP subdivision codes

sources/no/GEOIP_INVESTIGATION.md

@@ -0,0 +1,112 @@
+# Norway Counties: GeoIP Subdivision Code Investigation
+
+Related: https://github.com/elastic/ems-file-service/issues/565
+
+## Summary
+
+MaxMind's GeoLite2-City database uses a **hybrid of pre-2020 and 2020 ISO 3166-2 codes** for Norway.
+It does **not** use the 2024 codes from Wikidata, nor the 2020 codes for counties that were split back.
+
+## MaxMind Release Notes (March 2024)
+
+From the [March 2024 GeoNames diff report](https://dev.maxmind.com/geoip/release-notes/2024/#geonames-monthly-diff-report-march-2024):
+
+> In 2020, Norway merged a number of their subdivisions, but this proved unpopular and they have now
+> unmerged these subdivisions. We have used the ISO subdivision codes for these subdivisions from
+> before the 2020 change.
+
+The diff CSV confirms the transitions:
+
+```
+curl -sL "https://dev.maxmind.com/csv/GeoNames-Monthly-Diff-Report-March-2024.csv" | grep ",NO," \
+  | awk -F',' '{print $10 " -> " $11}' | sort | uniq -c | sort -rn
+```
+
+```
+    131 30 -> 2    # Viken → Akershus
+     54 54 -> 19   # Troms og Finnmark → Troms
+     50 30 -> 6    # Viken → Buskerud
+     45 30 -> 1    # Viken → Østfold
+     44 38 -> 7    # Vestfold og Telemark → Vestfold
+     33 38 -> 8    # Vestfold og Telemark → Telemark
+     28 54 -> 20   # Troms og Finnmark → Finnmark
+```
+
+Counties that stayed merged (NO-34, NO-42, NO-46, NO-50) are absent from the diff — they kept their 2020 codes.
+No changes in the [2025](https://dev.maxmind.com/geoip/release-notes/2025/) or [2026](https://dev.maxmind.com/geoip/release-notes/2026/) release notes.
+
+## Elasticsearch GeoIP Validation
+
+Scanned 2,560 Norwegian IPs via the Elasticsearch GeoIP ingest processor (`_simulate` API) using
+GeoLite2-City.mmdb (downloaded via Elastic's GeoIP downloader on ES 9.3.0).
+
+### Setup
+
+```bash
+# Enable eager download to fetch GeoLite2-City.mmdb
+curl -s -u "$AUTH" -X PUT "$ES/_cluster/settings" \
+  -H 'Content-Type: application/json' -d '{
+  "transient": { "ingest.geoip.downloader.eager.download": true }
+}'
+
+# Create test pipeline
+curl -s -u "$AUTH" -X PUT "$ES/_ingest/pipeline/norway-geoip-test" \
+  -H 'Content-Type: application/json' -d '{
+  "processors": [{
+    "geoip": {
+      "field": "ip",
+      "properties": ["country_iso_code", "region_iso_code", "region_name", "city_name"]
+    }
+  }]
+}'
+```
+
+### Scan
+
+IPs sampled across: Telenor (77.16.0.0/13, 84.208.0.0/12), Altibox (80.91.0.0/16),
+NextGenTel (85.164.0.0/15), Telia (88.89-95.0.0), UNINETT (128.39, 129.241, 152.94, 158.36.0.0/14),
+and various 193.x/195.x ranges. Processed in batches of 100 via `_simulate`.
+
+```python
+# Core loop (abbreviated)
+for batch in batched(ips, 100):
+    docs = [{"_source": {"ip": ip}} for ip in batch]
+    result = curl POST "/_ingest/pipeline/norway-geoip-test/_simulate" {"docs": docs}
+    for doc in result["docs"]:
+        geoip = doc["doc"]["_source"].get("geoip", {})
+        # collect unique region_iso_code values where country_iso_code == "NO"
+```
+
+### Results: 15 unique region codes
+
+| Code | Name | Type | Example cities |
+|------|------|------|----------------|
+| `NO-01` | Østfold | Pre-2020 (split back) | Fredrikstad, Sarpsborg, Halden, Moss |
+| `NO-02` | Akershus | Pre-2020 (split back) | Lillestrøm, Skjetten, Asker |
+| `NO-03` | Oslo | Unchanged | Oslo |
+| `NO-06` | Buskerud | Pre-2020 (split back) | Drammen, Kongsberg, Hokksund |
+| `NO-07` | Vestfold | Pre-2020 (split back) | Sandefjord, Tønsberg, Horten |
+| `NO-08` | Telemark | Pre-2020 (split back) | Skien, Porsgrunn, Seljord |
+| `NO-11` | Rogaland | Unchanged | Stavanger, Haugesund, Sandnes |
+| `NO-15` | Møre og Romsdal | Unchanged | Molde, Ålesund |
+| `NO-18` | Nordland | Unchanged | Bodø, Narvik, Leknes |
+| `NO-19` | Troms | Pre-2020 (split back) | Tromsø |
+| `NO-20` | Finnmark | Pre-2020 (split back) | Veines |
+| `NO-34` | Innlandet | 2020 (stayed merged) | Hamar, Kongsvinger, Løten |
+| `NO-42` | Agder | 2020 (stayed merged) | Kristiansand, Mandal, Grimstad |
+| `NO-46` | Vestland | 2020 (stayed merged) | Bergen, Paradis, Manger |
+| `NO-50` | Trøndelag | 2020 (stayed merged) | Trondheim, Orkanger, Rørvik |
+
+Codes **not** present: NO-04, NO-05, NO-09, NO-10, NO-12, NO-14 (pre-2020 for merged counties),
+NO-30, NO-38, NO-54 (2020 split-back), NO-31–33, NO-39–40, NO-55–56 (2024 Wikidata).
+
+## Required EMS Changes
+
+| Current EMS code | GeoIP status | Action |
+|---|---|---|
+| NO-04 Hedmark + NO-05 Oppland | ❌ Gone | Merge geometry → **NO-34 Innlandet** |
+| NO-09 Aust-Agder + NO-10 Vest-Agder | ❌ Gone | Merge geometry → **NO-42 Agder** |
+| NO-12 Hordaland + NO-14 Sogn og Fjordane | ❌ Gone | Merge geometry → **NO-46 Vestland** |
+| All other codes (01–03, 06–08, 11, 15, 18–20, 50) | ✅ Match | Keep as-is |
+
+Six pre-2020 codes → four 2020 merged codes. Feature count: 19 → 16 (+ Svalbard, Jan Mayen = 18 total).

sources/no/Makefile

@@ -0,0 +1,40 @@
+DATA_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/../../data)
+V1 := $(DATA_DIR)/norway_counties_v1.geo.json
+V2 := $(DATA_DIR)/norway_counties_v2.geo.json
+.DEFAULT_GOAL := all
+
+.PHONY: all clean
+
+all: $(V2)
+
+clean:
+	rm -f $(V2) /tmp/no_innlandet.geo.json /tmp/no_agder.geo.json /tmp/no_vestland.geo.json
+
+# NO-04 (Hedmark) + NO-05 (Oppland) → NO-34 (Innlandet)
+/tmp/no_innlandet.geo.json: $(V1)
+	mapshaper $< \
+	  -filter '"NO-04,NO-05".split(",").indexOf(iso_3166_2) > -1' \
+	  -dissolve calc="FID='Q56404886', iso_3166_2='NO-34', label_en='Innlandet', label_nn='Innlandet', label_nb='Innlandet'" \
+	  -o format=geojson $@
+
+# NO-09 (Aust-Agder) + NO-10 (Vest-Agder) → NO-42 (Agder)
+/tmp/no_agder.geo.json: $(V1)
+	mapshaper $< \
+	  -filter '"NO-09,NO-10".split(",").indexOf(iso_3166_2) > -1' \
+	  -dissolve calc="FID='Q2729021', iso_3166_2='NO-42', label_en='Agder', label_nn='Agder', label_nb='Agder'" \
+	  -o format=geojson $@
+
+# NO-12 (Hordaland) + NO-14 (Sogn og Fjordane) → NO-46 (Vestland)
+/tmp/no_vestland.geo.json: $(V1)
+	mapshaper $< \
+	  -filter '"NO-12,NO-14".split(",").indexOf(iso_3166_2) > -1' \
+	  -dissolve calc="FID='Q56407177', iso_3166_2='NO-46', label_en='Vestland', label_nn='Vestland', label_nb='Vestland'" \
+	  -o format=geojson $@
+
+# Combine all v1 features with the 3 new merged features
+$(V2): $(V1) /tmp/no_innlandet.geo.json /tmp/no_agder.geo.json /tmp/no_vestland.geo.json
+	mapshaper -i combine-files $^ \
+	  -merge-layers force \
+	  -sort this.properties.iso_3166_2 \
+	  -o precision=0.000001 format=geojson prettify $@
+	node ../../scripts/clean-geom.js -v $@

sources/no/README.md

@@ -0,0 +1,41 @@
+# Norway Counties
+
+Related issue: https://github.com/elastic/ems-file-service/issues/565
+
+## Dataset versions
+
+### v1 (`norway_counties_v1.geo.json`)
+
+Original dataset with 20 features using pre-2020 ISO 3166-2 codes (NO-01 through NO-20, plus NO-50 Troendelag). Served to Kibana/Elasticsearch versions 1 through 7.
+
+### v2 (`norway_counties_v2.geo.json`)
+
+Updated dataset with 23 features, served to Kibana/Elasticsearch versions 8+. Contains all 20 original pre-2020 features plus 3 new features created by dissolving county pairs that were merged in the 2020 Norwegian administrative reform:
+
+| New code | New name | Merged from |
+|----------|----------|-------------|
+| NO-34 | Innlandet | NO-04 (Hedmark) + NO-05 (Oppland) |
+| NO-42 | Agder | NO-09 (Aust-Agder) + NO-10 (Vest-Agder) |
+| NO-46 | Vestland | NO-12 (Hordaland) + NO-14 (Sogn og Fjordane) |
+
+The original features are kept alongside the merged ones so that users with any version of the MaxMind GeoIP database (pre-2020 or post-March 2024) can find matching geometries. This means there are intentionally overlapping geometries in the dataset.
+
+## Background
+
+In March 2024, MaxMind updated their GeoLite2-City database to reflect Norway's 2024 county splits. However, rather than using the new 2024 ISO codes (NO-31, NO-32, etc.), they reverted to the pre-2020 codes for the split counties while keeping the 2020 codes for counties that stayed merged. See [`GEOIP_INVESTIGATION.md`](GEOIP_INVESTIGATION.md) for the full analysis.
+
+## Building v2
+
+Run `make` in this directory. Requires [`mapshaper`](https://github.com/mbloch/mapshaper) on PATH.
+
+The Makefile takes v1 as input and:
+
+1. Filters and dissolves each pair of pre-2020 counties into a single merged feature with the 2020 ISO code and labels
+2. Combines all original v1 features with the 3 new merged features
+3. Sorts by ISO code and runs `clean-geom.js` to fix any geometry issues
+
+To rebuild from scratch:
+
+```
+make clean && make
+```

sources/no/counties.hjson

@@ -1,5 +1,5 @@
 {
-    versions: '>=1'
+    versions: '1 - 7'
     type: http
     production: true
     countryCode: NO

sources/no/counties_v2.hjson

@@ -0,0 +1,116 @@
+{
+    versions: '>=8'
+    type: http
+    production: true
+    countryCode: NO
+    note: Counties of Norway (v2: includes both pre-2020 and 2020 merged county codes for GeoIP compatibility)
+    wikidata: Q192299
+    data: https://sophox.org/regions/geojson.json
+    attribution: [
+      {
+        label: {
+          en: OpenStreetMap contributors
+        }
+        url: {
+          en: https://www.openstreetmap.org/copyright
+        }
+      }
+      {
+        label: {
+          en: Elastic Maps Service
+        }
+        url: {
+          en: https://www.elastic.co/elastic-maps-service
+        }
+      }
+    ]
+    fieldMapping: [
+        {
+            type: id
+            name: iso_3166_2
+            desc: ISO-3166-2 identifier
+        }
+        {
+            type: property
+            name: label_en
+            desc: County name (English)
+        }
+        {
+            type: property
+            name: label_nn
+            desc: County name (Nynorsk)
+        }
+        {
+            type: property
+            name: label_nb
+            desc: County name (Bokmål)
+        }
+    ]
+    name: norway_counties
+    legacyIds: [
+      Norway Counties
+    ]
+    humanReadableName: {
+      ar: مقاطعة في النرويج
+      ast: Provincies de Noruega
+      bg: Административно деление на Норвегия
+      bs: Općine u Norveškoj
+      ca: Comtat de Noruega
+      cs: norský kraj
+      da: Norges fylker
+      de: norwegischen Provinzen
+      en: Norway Counties
+      eo: provinco de Norvegio
+      es: provincia de Noruega
+      eu: Norvegiaren banaketa administratiboa
+      fi: Norjan läänit
+      fo: Fylki Noregs
+      fr: fylker de Norvège
+      fy: Provinsjes fan Noarwegen
+      he: מחוזות נורווגיה
+      hr: Okruzi Norveške
+      hu: Norvégia megyéi
+      is: Fylki Noregs
+      it: contee della Norvegia
+      ja: ノルウェーの県
+      ka: ნორვეგიის საგრაფოები
+      ko: 노르웨이의 주
+      li: perviencies in Nórwaeg
+      lt: Norvegijos administracinio suskirstymo vienetai
+      lv: Norvēģijas filke
+      mk: округ во Норвешка
+      ms: Daerah di Norway
+      nb: norsk fylke
+      nl: provincie van Noorwegen
+      nn: Fylkeskommune
+      pl: Okręgi Norwegii
+      pnb: ناروے دے ضلعے
+      ps: د ناروې فیلکی
+      pt: condado da Noruega
+      ro: Judeţele Norvegiei
+      ru: губерния Норвегии
+      sco: Coonties o Norawa
+      se: Norgga fylkkat
+      sl: Okrožja Norveške
+      sq: Qarqet e Norvegjisë
+      sr: округ Норвешке
+      sr-ec: Административна подела Норвешке
+      sr-el: Administrativna podela Norveške
+      sv: Norges fylken
+      sw: majimbo ya Norwei
+      tr: Norveç eyaleti
+      uk: Норвегія: фюльке
+      ur: ناروے کی کاؤنٹیاں
+      vec: Contee de la Norvesa
+      zh: 縣 (挪威)
+    }
+    emsFormats: [{
+        type: geojson
+        file: norway_counties_v2.geo.json
+        default: true
+    }]
+    ttl: 108000
+    weight: 0
+    createdAt: "2018-08-29T17:27:28.246260"
+    id: 5646874153320448
+}

sources/world/Makefile

@@ -14,7 +14,9 @@ clean: ## Remove output files
    /tmp/wikidata.csv /tmp/area.csv /tmp/population.csv \
    /tmp/missing-area.csv /tmp/missing-population.csv \
    /tmp/ne_10m_admin_1_states_provinces.topo.json \
-   /tmp/additional-regions.geo.json
+   /tmp/additional-regions.geo.json \
+   /tmp/no_world_innlandet.geo.json /tmp/no_world_agder.geo.json \
+   /tmp/no_world_vestland.geo.json /tmp/no_merged_counties.geo.json
 
 /tmp/ne_10m_admin_1_states_provinces.topo.json: ## Uncompress the NE regions dataset
 	gunzip --keep ./ne_10m_admin_1_states_provinces.topo.json.gz
@@ -24,16 +26,43 @@ clean: ## Remove output files
 	mapshaper -i /tmp/ne_10m_admin_1_states_provinces.topo.json \
   -o format=geojson /tmp/ne_10m_admin_1_states_provinces.geo.json
 
+# Norway 2020 merged counties: dissolve pre-2020 county pairs from the NE dataset
+# NO-04 (Hedmark) + NO-05 (Oppland) → NO-34 (Innlandet)
+/tmp/no_world_innlandet.geo.json: /tmp/ne_10m_admin_1_states_provinces.geo.json
+	mapshaper $< \
+  -filter '"NO-04,NO-05".split(",").indexOf(iso_3166_2) > -1' \
+  -dissolve calc="iso_3166_2='NO-34', name='Innlandet', iso_a2='NO', adm0_a3='NOR', admin='Norway'" \
+  -o format=geojson $@
+
+# NO-09 (Aust-Agder) + NO-10 (Vest-Agder) → NO-42 (Agder)
+/tmp/no_world_agder.geo.json: /tmp/ne_10m_admin_1_states_provinces.geo.json
+	mapshaper $< \
+  -filter '"NO-09,NO-10".split(",").indexOf(iso_3166_2) > -1' \
+  -dissolve calc="iso_3166_2='NO-42', name='Agder', iso_a2='NO', adm0_a3='NOR', admin='Norway'" \
+  -o format=geojson $@
+
+# NO-12 (Hordaland) + NO-14 (Sogn og Fjordane) → NO-46 (Vestland)
+/tmp/no_world_vestland.geo.json: /tmp/ne_10m_admin_1_states_provinces.geo.json
+	mapshaper $< \
+  -filter '"NO-12,NO-14".split(",").indexOf(iso_3166_2) > -1' \
+  -dissolve calc="iso_3166_2='NO-46', name='Vestland', iso_a2='NO', adm0_a3='NOR', admin='Norway'" \
+  -o format=geojson $@
+
+/tmp/no_merged_counties.geo.json: /tmp/no_world_innlandet.geo.json /tmp/no_world_agder.geo.json /tmp/no_world_vestland.geo.json
+	mapshaper -i combine-files $^ \
+  -merge-layers force \
+  -o format=geojson $@
+
 /tmp/additional-regions.geo.json: /tmp/ne_10m_admin_1_states_provinces.geo.json ./duplicated-regions.csv
 	cd ../../scripts && \
   node duplicate-regions.js -v \
     ../sources/world/duplicated-regions.csv \
     /tmp/ne_10m_admin_1_states_provinces.geo.json \
     /tmp/additional-regions.geo.json
 
-../../data/admin_regions_lvl2_v2.topo.json: /tmp/ne_10m_admin_1_states_provinces.topo.json /tmp/additional-regions.geo.json ## Regions TopoJSON
+../../data/admin_regions_lvl2_v2.topo.json: /tmp/ne_10m_admin_1_states_provinces.topo.json /tmp/additional-regions.geo.json /tmp/no_merged_counties.geo.json ## Regions TopoJSON
 	mapshaper -i   combine-files \
-  /tmp/ne_10m_admin_1_states_provinces.topo.json /tmp/additional-regions.geo.json \
+  /tmp/ne_10m_admin_1_states_provinces.topo.json /tmp/additional-regions.geo.json /tmp/no_merged_counties.geo.json \
   -merge-layers force \
   -dissolve "iso_3166_2" copy-fields="iso_3166_2,name,iso_a2,adm0_a3,admin" \
   -filter "this.properties.iso_3166_2 !== ''" \