chore: release 0.3.1 (CLI updater, docs, refreshed catalogs)

Author: esteininger

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 All notable changes to this project will be documented here.
 
+## [0.3.1] - 2025-09-10
+- CLI: add `update-catalogs` subcommand (fetch exact or latest 2.x/3.x)
+- Script: `scripts/update_catalogs.py` robust TSV header detection
+- README: document catalog updater (script and CLI)
+- Data: refreshed catalogs (3.1 and 2.2)
+
 ## [0.3.0] - 2025-09-09
 - Vectors catalogs and output wiring
 - Configurable cattax flag

README.md

@@ -42,7 +42,11 @@ Outputs are **IAB‑3.0–compatible IDs** for OpenRTB/VAST, with optional **vec
 Use the bundled fetcher to sync to the latest Content Taxonomy files from the official IAB GitHub repository. It will locate the latest 2.x and 3.x datasets and normalize them into this tool’s schemas.
 
 ```bash
+# via Python script (direct)
 python scripts/update_catalogs.py
+
+# or via CLI command
+mixpeek-iab-mapper update-catalogs --exact3 "3.1" --exact2 "2.2"
 # Optional: use a GitHub token to raise rate limits
 # export GITHUB_TOKEN=ghp_...
 ```
@@ -312,13 +316,13 @@ Each value maps to a **stable IAB 3.0 ID** that is appended to the `cat` array.
 
 ## 📎 Official IAB References
 
-- Content Taxonomy 3.0 Implementation Guide (PDF): `https://iabtechlab.com/wp-content/uploads/2021/09/Implementation-Guide-Content-Taxonomy-3-0-pc-Sept2021.pdf`
-- IAB Tech Lab Content Taxonomy page: `https://iabtechlab.com/standards/content-taxonomy/`
+- Content Taxonomy 3.0 Implementation Guide (PDF): https://iabtechlab.com/wp-content/uploads/2021/09/Implementation-Guide-Content-Taxonomy-3-0-pc-Sept2021.pdf
+- IAB Tech Lab Content Taxonomy page: https://iabtechlab.com/standards/content-taxonomy/
 - Implementation guidance (historic mappings and migration notes):
-  - `https://github.com/InteractiveAdvertisingBureau/Taxonomies/blob/develop/implementation.md#content-21-to-ad-product-20-taxonomy-mapping-implementation-guidance`
-  - `https://github.com/InteractiveAdvertisingBureau/Taxonomies/blob/develop/Taxonomy%20Mappings/Ad%20Product%202.0%20to%20Content%202.1.tsv`
-  - `https://github.com/katieshell/Taxonomies/blob/main/implementation.md#implementation-guidance-for-content-1--content-2-mapping`
-  - `https://github.com/katieshell/Taxonomies/blob/main/implementation.md#migrating-from-content-taxonomy-10`
+  - https://github.com/InteractiveAdvertisingBureau/Taxonomies/blob/develop/implementation.md#content-21-to-ad-product-20-taxonomy-mapping-implementation-guidance
+  - https://github.com/InteractiveAdvertisingBureau/Taxonomies/blob/develop/Taxonomy%20Mappings/Ad%20Product%202.0%20to%20Content%202.1.tsv
+  - https://github.com/katieshell/Taxonomies/blob/main/implementation.md#implementation-guidance-for-content-1--content-2-mapping
+  - https://github.com/katieshell/Taxonomies/blob/main/implementation.md#migrating-from-content-taxonomy-10
 
 ---
 
@@ -364,7 +368,7 @@ mixpeek-iab-mapper sample_2x_codes.csv -o mapped.json --use-embeddings --drop-sc
 ---
 
 ## 📜 License
-MIT. See `LICENSE`.
+MIT. See [LICENSE](LICENSE).
 
 Include IAB attribution in your deployed UI/footer:
 > “IAB is a registered trademark of the Interactive Advertising Bureau. This tool is an independent utility built by Mixpeek for interoperability with IAB Content Taxonomy standards.”

iab_mapper.egg-info/PKG-INFO

@@ -2,8 +2,25 @@ Metadata-Version: 2.4
 Name: iab-mapper
 Version: 0.3.0
 Summary: Local IAB Content Taxonomy 2.x -> 3.0 mapper with vectors, SCD, OpenRTB/VAST exporters.
+Author: Mixpeek
+License: MIT
+Project-URL: Homepage, https://github.com/mixpeek/iab-mapper
+Project-URL: Repository, https://github.com/mixpeek/iab-mapper
+Project-URL: Issues, https://github.com/mixpeek/iab-mapper/issues
+Keywords: iab,taxonomy,content,openrtb,ctv,classification
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
+License-File: LICENSE
 Requires-Dist: pandas>=2.1
 Requires-Dist: rapidfuzz>=3.0
 Requires-Dist: typer>=0.12
@@ -15,13 +32,66 @@ Requires-Dist: scikit-learn>=1.4
 Requires-Dist: requests>=2.31
 Provides-Extra: emb
 Requires-Dist: sentence-transformers>=3.0; extra == "emb"
+Dynamic: license-file
+
+<p align="center">
+  <img src="assets/header.png" alt="IAB Taxonomy Mapper" width="900" />
+</p>
 
 # IAB Content Taxonomy Mapper (Local CLI)
 
-Map **IAB Content Taxonomy 2.x** labels/codes to **IAB 3.0** locally with a deterministic→fuzzy→(optional) local-embeddings pipeline.
-Outputs are **IAB-3.0–compatible IDs** suitable for OpenRTB/VAST, with optional **vector attributes** (Channel, Type, Format, Language, Source, Environment) and **SCD** awareness.
+<p align="center">
+  <a href="https://pypi.org/project/iab-mapper/"><img alt="PyPI" src="https://img.shields.io/pypi/v/iab-mapper.svg"></a>
+  <a href="https://github.com/mixpeek/iab-mapper/actions"><img alt="CI" src="https://github.com/mixpeek/iab-mapper/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="https://github.com/mixpeek/iab-mapper/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
+</p>
+
+Map **IAB Content Taxonomy 2.x** labels/codes to **IAB 3.0** locally with a deterministic → fuzzy → (optional) semantic pipeline.
+Outputs are **IAB‑3.0–compatible IDs** for OpenRTB/VAST, with optional **vector attributes** (Channel, Type, Format, Language, Source, Environment) and **SCD** awareness.
+
+> Local-first by default. No external APIs are required; LLM re‑rank is optional.
+
+---
+
+## 📚 Table of Contents
+
+- [✨ Features](#-features)
+- [Why migrate to IAB 3.0?](#-why-migrate-to-iab-30)
+- [How it works](#-how-it-works)
+- [🔧 Install](#-install)
+- [🚀 Quick Start](#-quick-start)
+- [🐍 Python API](#-python-api-alternative-to-cli)
+- [📥 Input Formats](#-input-formats)
+- [📤 Output Formats](#-output-formats)
+- [⚙️ Useful Flags](#️-useful-flags)
+- [🧩 Vectors](#-vectors-orthogonal-attributes)
+- [✅ IAB 3.0 Conformance Notes](#-iab-30-conformance-notes)
+- [📎 Official IAB References](#-official-iab-references)
+- [🧯 Troubleshooting](#-troubleshooting)
+- [📦 Example Commands](#-example-commands)
+- [📜 License](#-license)
+
+---
+
+### Update catalogs (fetch latest from IAB)
+
+Use the bundled fetcher to sync to the latest Content Taxonomy files from the official IAB GitHub repository. It will locate the latest 2.x and 3.x datasets and normalize them into this tool’s schemas.
+
+```bash
+# via Python script (direct)
+python scripts/update_catalogs.py
+
+# or via CLI command
+mixpeek-iab-mapper update-catalogs --exact3 "3.1" --exact2 "2.2"
+# Optional: use a GitHub token to raise rate limits
+# export GITHUB_TOKEN=ghp_...
+```
+
+Outputs:
+- `iab_mapper/data/iab_2x.json` → `[{"code","label"}]`
+- `iab_mapper/data/iab_3x.json` → `[{"id","label","path":[],"scd":bool}]`
 
-> No external APIs. Runs fully local. LLMs are **not required**. You can enable local embeddings for tougher matches.
+Replace or extend `synonyms_*.json` and `vectors_*.json` as needed for your org.
 
 ---
 
@@ -35,8 +105,34 @@ Outputs are **IAB-3.0–compatible IDs** suitable for OpenRTB/VAST, with optiona
 
 ---
 
+## 🔎 Why migrate to IAB 3.0?
+
+- 3.0 introduces clearer separation of primary topic “aboutness” vs. orthogonal vectors (e.g., news vs. opinion, formats, channels).
+- Better support for CTV/video, podcasts, games, and app stores.
+- Non‑backwards compatible in areas like News/Opinion and entertainment genres; careful migration is required.
+
+This tool makes migration practical: it emits valid 3.0 IDs and helps curate edge cases with overrides, synonyms, thresholds, and audit outputs.
+
+---
+
+## 🧠 How it works
+
+1) Normalize text and apply alias/exact matches via synonyms.
+2) Fuzzy retrieval (rapidfuzz | TF‑IDF | BM25) with configurable thresholds.
+3) Optional semantic augmentation with local embeddings (Sentence‑Transformers or TF‑IDF KNN).
+4) Optional local LLM re‑ranking (Ollama) for ordering only.
+5) Assemble outputs: topic IDs + vector IDs → OpenRTB `content.cat` with configurable `cattax`.
+6) SCD flags are surfaced and can be excluded with `--drop-scd`.
+
+---
+
 ## 🔧 Install
 
+### From PyPI (recommended)
+```bash
+pip install iab-mapper
+```
+
 ### 1) Clone / unpack
 ```bash
 unzip iab-mapper.zip && cd iab-mapper
@@ -102,6 +198,74 @@ The output contains for each input row:
 
 ---
 
+## 🐍 Python API (alternative to CLI)
+
+Install:
+```bash
+pip install iab-mapper
+```
+
+Basic usage:
+```python
+from pathlib import Path
+from iab_mapper.pipeline import Mapper, MapConfig
+import iab_mapper as pkg
+
+# Use packaged stub catalogs or point data_dir to your own
+data_dir = Path(pkg.__file__).parent / "data"
+
+cfg = MapConfig(
+    fuzzy_method="bm25",   # rapidfuzz|tfidf|bm25
+    fuzzy_cut=0.92,
+    use_embeddings=False,   # set True and choose emb_model to enable
+    max_topics=3,
+    drop_scd=False,
+    cattax="2",            # OpenRTB content.cattax enum
+    overrides_path=None     # path to JSON overrides if desired
+)
+
+mapper = Mapper(cfg, str(data_dir))
+
+# Single record with optional vectors
+rec = {
+    "code": "2-12",
+    "label": "Food & Drink",
+    "channel": "editorial",
+    "type": "article",
+    "format": "video",
+    "language": "en",
+    "source": "professional",
+    "environment": "ctv",
+}
+
+out = mapper.map_record(rec)
+print(out["out_ids"])         # topic + vector IDs
+print(out["openrtb"])         # {"content": {"cat": [...], "cattax": "2"}}
+print(out["vast_contentcat"]) # "id1","id2",...
+
+# Or just map topics
+topics = mapper.map_topics("Cooking how-to")
+
+# Batch over a list of dicts
+rows = [rec, {"label": "Sports"}]
+mapped = [mapper.map_record(r) for r in rows]
+```
+
+Enable local embeddings (optional):
+```python
+cfg = MapConfig(fuzzy_method="rapidfuzz", use_embeddings=True, emb_model="tfidf", emb_cut=0.8)
+mapper = Mapper(cfg, str(data_dir))
+out = mapper.map_record({"label": "Cooking how-to"})
+```
+
+Use overrides (force mapping before matching):
+```python
+cfg = MapConfig(overrides_path="overrides.json")  # [{"code":"1-4","label":null,"ids":["2-3-18"]}]
+mapper = Mapper(cfg, str(data_dir))
+```
+
+---
+
 ## 📥 Input Formats
 
 ### CSV
@@ -186,6 +350,18 @@ Each value maps to a **stable IAB 3.0 ID** that is appended to the `cat` array.
 
 ---
 
+## 📎 Official IAB References
+
+- Content Taxonomy 3.0 Implementation Guide (PDF): https://iabtechlab.com/wp-content/uploads/2021/09/Implementation-Guide-Content-Taxonomy-3-0-pc-Sept2021.pdf
+- IAB Tech Lab Content Taxonomy page: https://iabtechlab.com/standards/content-taxonomy/
+- Implementation guidance (historic mappings and migration notes):
+  - https://github.com/InteractiveAdvertisingBureau/Taxonomies/blob/develop/implementation.md#content-21-to-ad-product-20-taxonomy-mapping-implementation-guidance
+  - https://github.com/InteractiveAdvertisingBureau/Taxonomies/blob/develop/Taxonomy%20Mappings/Ad%20Product%202.0%20to%20Content%202.1.tsv
+  - https://github.com/katieshell/Taxonomies/blob/main/implementation.md#implementation-guidance-for-content-1--content-2-mapping
+  - https://github.com/katieshell/Taxonomies/blob/main/implementation.md#migrating-from-content-taxonomy-10
+
+---
+
 ## 🔬 Evaluation (recommended)
 Create a small gold set for your domain and run periodic checks:
 ```bash
@@ -228,5 +404,7 @@ mixpeek-iab-mapper sample_2x_codes.csv -o mapped.json --use-embeddings --drop-sc
 ---
 
 ## 📜 License
-TBD by Mixpeek. Include IAB attribution in your deployed UI/footer:
+MIT. See [LICENSE](LICENSE).
+
+Include IAB attribution in your deployed UI/footer:
 > “IAB is a registered trademark of the Interactive Advertising Bureau. This tool is an independent utility built by Mixpeek for interoperability with IAB Content Taxonomy standards.”

iab_mapper.egg-info/SOURCES.txt

@@ -1,3 +1,4 @@
+LICENSE
 README.md
 pyproject.toml
 iab_mapper/__init__.py
@@ -8,6 +9,7 @@ iab_mapper/llm.py
 iab_mapper/matching.py
 iab_mapper/normalize.py
 iab_mapper/pipeline.py
+iab_mapper/updater.py
 iab_mapper.egg-info/PKG-INFO
 iab_mapper.egg-info/SOURCES.txt
 iab_mapper.egg-info/dependency_links.txt

iab_mapper/cli.py

@@ -3,6 +3,7 @@
 from rich.console import Console
 from rich.table import Table
 from iab_mapper.pipeline import Mapper, MapConfig
+from iab_mapper.updater import update_catalogs
 
 app = typer.Typer(add_completion=False)
 con = Console()
@@ -78,3 +79,17 @@ def run(
         Path(unmapped_out).write_text(json.dumps(unmapped, ensure_ascii=False), encoding="utf-8")
         con.print(f"Unmapped → {unmapped_out} ({len(unmapped)})")
 if __name__=="__main__": app()
+
+
+@app.command(name="update-catalogs")
+def update_catalogs_cmd(
+    data_dir: Path = typer.Option(Path(__file__).parent / "data", help="Data dir to write catalogs"),
+    major3: int = typer.Option(3, help="Major version to pick for Content Taxonomy 3.x (e.g., 3)"),
+    major2: int = typer.Option(2, help="Major version to pick for Content Taxonomy 2.x (e.g., 2)"),
+    exact3: str = typer.Option(None, help="Exact filename substring for 3.x (e.g., '3.1')"),
+    exact2: str = typer.Option(None, help="Exact filename substring for 2.x (e.g., '2.2')"),
+    token: str = typer.Option(None, help="GitHub token (overrides env GITHUB_TOKEN)"),
+):
+    """Fetch latest IAB catalogs from IAB GitHub and normalize into JSON."""
+    update_catalogs(str(data_dir), major3=major3, major2=major2, exact3=exact3, exact2=exact2, token=token)
+    con.print(f"Updated catalogs in {data_dir}")