Changes in KuraTHor to prepare for the DiVA migration

[awandahl]

In the new DiVA system regexps are used to verify the content in some fields. Here are the expressions that most likely will be used:

DOI: ^(10[.])[0-9]{4}[\S]+$
ISBN: ^((?:[\dX]{13})|(?:[\d-X]{17})|(?:[\dX]{10})|(?:[\d-X]{13}))$
ISSN: ^(\d{4})-(\d{3}[0-9X])$
WOS: (^\d+$)
SCOPUS: ^(2-s2.0-)(\d+)$
PMID: ^(\d{1,8})(|.\d{1,3})$

It would be useful to check the present DiVA content satisfying at least these expressions. If we have other and more strict rules we can add them on top of the above. We know that there are a number of old WOS id that begin with a letter (usually "A" followed by numbers), example https://www.webofscience.com/wos/woscc/full-record/WOS:A1986A067700004 We will probably have to write our own regexp for that.

This could be used: ^A\d{4}[A-Za-z]{1,2}\d+$

Another thing that eventually will be needed is to extend the time frame to 1066-2026. As all publications in DiVA will be migrated, we have to clean them all. I guess that could be implemented immediately, and that we make sure that the default sorting is "new-to-old".

[awandahl]

DOI: ^(10[.])[0-9]{4}[\S]+$

• Must start with 10. (the ^ means "start here").
• Then has exactly 4 digits (e.g., 1234).
• Then has one or more non-whitespace characters (\S+ means any character that isn’t a space, tab, or newline).
• The entire string must match this pattern (the $ at the end ensures it ends there).
• Example: 10.1234/example123.[^2]

ISBN: ^((?:[\dX]{13})|(?:[\d-X]{17})|(?:[\dX]{10})|(?:[\d-X]{13}))$

• Must match one of these four formats:
  • Exactly 13 digits or X (for ISBN-13, like 9780123456789).
  • Up to 17 characters including digits, hyphens, and X (common ISBN-10 or ISBN-13 with hyphens, like 978-0-12-345678-9).
  • Exactly 10 digits or X (for older ISBN-10, like 012345678X).
  • Up to 13 characters with digits, hyphens, and X (another ISBN-13 format with hyphens).
• The entire string must be one of these types.[^3]

ISSN: ^(\d{4})-(\d{3}[0-9X])$

• Starts with exactly 4 digits (e.g., 1234).
• Then a hyphen -.
• Then exactly 3 digits followed by one digit or X (e.g., 009X).
• Total format: 1234-009X — used for journals and periodicals.[^3]

WOS: (^\d+$)

• Must contain only digits (0–9) from start to end.
• No letters, spaces, or symbols allowed.
• Example: 000991713400001 — this matches the current Web of Science UT (Unique Article Identifier) format.[^1]

SCOPUS: ^(2-s2.0-)(\d+)$

• Must start with the exact prefix 2-s2.0-.
• Then one or more digits (e.g., 1234567890).
• Example: 2-s2.0-1234567890 — this is the standard Scopus ID format.[^4]

PMID: ^(\d{1,8})(|.\d{1,3})$

• Starts with 1 to 8 digits (e.g., a PubMed ID like 12345678).
• Optionally followed by a dot and 1 to 3 more digits (e.g., .123 for versioned records).
• So it matches both 12345678 and 12345678.123.[^5]

[mskyttner]

Draft (oai.db) script:

│ n_pid │    issue     │
│ int64 │   varchar    │
├───────┼──────────────┤
│  2675 │ ISBN issue   │
│   303 │ ISSN issue   │
│     3 │ DOI issue    │
│     2 │ PMID issue   │
│     1 │ Scopus issue │

load webbed;

copy (
    with m as (
        from mods
        select
            mods: xml(mods),
    ), 

    ids as (

        from m
        select
            * exclude (mods),
            recordIdentifier: xml_extract_text(mods, '//*[local-name()=''recordIdentifier'']'),
            pid: replace(recordIdentifier, 'diva2:', ''),
            issn: nullif(xml_extract_text(mods, '//*[local-name()=''relatedItem'']//*[local-name()=''identifier''][@type=''issn'']'),''),
            isbn: nullif(xml_extract_text(mods, '//*[local-name()=''identifier''][@type=''isbn'']'), ''),
            doi: nullif(xml_extract_text(mods, '//*[local-name()=''identifier''][@type=''doi'']'),''),
            wos: nullif(xml_extract_text(mods, '//*[local-name()=''identifier''][@type=''wos'']'),''),
            scopus: nullif(xml_extract_text(mods, '//*[local-name()=''identifier''][@type=''scopus'']'),''),
            pmid: nullif(xml_extract_text(mods, '//*[local-name()=''identifier''][@type=''pmid'']'),''),
    ),

    blaha as (
        from ids
        select 
            ids.pid,
            ids.* exclude (pid, recordIdentifier),
            ok_doi: regexp_matches(doi, '^(10[.])[0-9]{4,}/[\S]+$'),
            ok_isbn: regexp_matches(isbn, '^((?:[\dX]{13})|(?:[\d-X]{17})|(?:[\dX]{10})|(?:[\d-X]{13}))$'),
            ok_issn: regexp_matches(issn, '^(\d{4})-(\d{3}[0-9X])$'),
            ok_wos: regexp_matches(wos, '(^\d+$)'),
            ok_scopus: regexp_matches(scopus, '^(2-s2.0-)(\d+)$'),
            ok_pmid: regexp_matches(pmid, '^(\d{1,8})(|.\d{1,3})$'),
            ok_all: ok_doi and ok_isbn and ok_issn and ok_wos and ok_scopus and ok_pmid,
            ok_any: ok_doi or ok_isbn or ok_issn or ok_wos or ok_scopus or ok_pmid,
    ) 
    
    from blaha
    where ok_all is false

) to 
'checks.parquet';

from 'checks.parquet';

copy (
    with issues as (

        from 'checks.parquet'
        select pid, issue: 'DOI issue', value: doi
        where ok_doi is false

        union

        from 'checks.parquet'
        select pid, issue: 'ISBN issue', value: isbn
        where ok_isbn is false

        union

        from 'checks.parquet'
        select pid, issue: 'ISSN issue', value: issn
        where ok_issn is false

        union

        from 'checks.parquet'
        select pmid, issue: 'PMID issue', value: pmid
        where ok_pmid is false

        union

        from 'checks.parquet'
        select scopus, issue: 'Scopus issue', value: scopus
        where ok_scopus is false

        union

        from 'checks.parquet'
        select wos, issue: 'WoS issue', value: wos
        where ok_wos is false

    )

    from issues
    order by pid desc
)
to 'issues.parquet'
;

from 'issues.parquet'
select 
    n_pid: count(distinct pid),
    issue
group by issue
order by n_pid desc
;

[biblionaut]

Another thing that eventually will be needed is to extend the time frame to 1066-2026. As all publications in DiVA will be migrated, we have to clean them all. I guess that could be implemented immediately, and that we make sure that the default sorting is "new-to-old".

Sorting and limiting records shown based on publishing year will miss the small subset of records that don't have that data. This would primarily be manuscripts and articles registered under other statuses than "published" and "aheadofprint".

While these publications should lack the identifiers listed, we might want to think of some way of listing them so they won't be forgotten/always excluded from the quality assurance process.