Memory leak in phonemize.py

[Dave1475]

Memory leak in phonemize.py

phonemizer = BACKENDS[backend]
Spawns a new instance each time it's ran, while the old one persists in memory

One solution is to cache it, so it's only instantiated once:

Global cache for phonemizer backends

_PHONEMIZER_CACHE = {}

Create a cache key from the configuration

cache_key = (
    backend,
    language,
    str(punctuation_marks),
    preserve_punctuation,
    with_stress,
    tie,
    language_switch,
    words_mismatch
)

# Check if we have a cached backend
if cache_key not in _PHONEMIZER_CACHE:
    # initialize the phonemization backend
    if backend == 'espeak':
        _PHONEMIZER_CACHE[cache_key] = BACKENDS[backend](
            language,
            punctuation_marks=punctuation_marks,
            preserve_punctuation=preserve_punctuation,
            with_stress=with_stress,
            tie=tie,
            language_switch=language_switch,
            words_mismatch=words_mismatch,
            logger=logger)
    elif backend == 'espeak-mbrola':
        _PHONEMIZER_CACHE[cache_key] = BACKENDS[backend](
            language,
            logger=logger)
    else:  # festival or segments
        _PHONEMIZER_CACHE[cache_key] = BACKENDS[backend](
            language,
            punctuation_marks=punctuation_marks,
            preserve_punctuation=preserve_punctuation,
            logger=logger)

phonemizer = _PHONEMIZER_CACHE[cache_key]

[gerelef]

Bump.

Not only that, whenever a call is made to phonemizer.phonemize a new instance of libespeak-ng is copied to /tmp, and after a few thousand calls this completely fills up my disk, and this results in phonemizer being unusable on any respectable dataset for my usecase (TTS).

[npuichigo]

@gerelef Do you know how to prevent new instance of libespeak-ng be copied to /tmp?

[gerelef]

@gerelef Do you know how to prevent new instance of libespeak-ng be copied to /tmp?

Unfortunately, I don't. However, if you merge my pull request, it'll only instanciate espeak-ng once to /tmp; I only implemented a cache to speed up consecutive operations.

[npuichigo]

I find use njobs on the instance still copy the espeak-ng each time phonemize is called